Grammar Extended

Grammar that tokenizes lines of text.

This class should not be instantiated directly but instead obtained from a GrammarRegistry by calling GrammarRegistry::loadGrammar.

Event Subscription

::onDidUpdate(callback)

Invoke the given callback when this grammar is updated due to a grammar it depends on being added or removed from the registry.

Argument Description

callback

Function to call when this grammar is updated.

Return values

Returns a Disposable on which .dispose() can be called to unsubscribe.

Tokenizing

::tokenizeLines(text)

Tokenize all lines in the given text.

Argument Description

text

A String containing one or more lines.

Return values

Returns an Array of token arrays for each line tokenized.

::tokenizeLine(line, ruleStack, firstLine)

Tokenize the line of text.

Argument Description

line

A String of text to tokenize.

ruleStack

An optional Array of rules previously returned from this method. This should be null when tokenizing the first line in the file.

firstLine

A optional Boolean denoting whether this is the first line in the file which defaults to false. This should be true when tokenizing the first line in the file.

Return values

Returns an Object containing the following properties:

  • line The String of text that was tokenized.
  • tags An Array of integer scope ids and strings. Positive ids indicate the beginning of a scope, and negative tags indicate the end. To resolve ids to scope names, call GrammarRegistry::scopeForId with the absolute value of the id.
  • tokens This is a dynamic property. Invoking it will incur additional overhead, but will automatically translate the tags into token objects with value and scopes properties.
  • ruleStack An Array of rules representing the tokenized state at the end of the line. These should be passed back into this method when tokenizing the next line in the file.