App Engine Python SDK  v1.6.9 rev.445
The Python runtime is available as an experimental Preview feature.
Public Member Functions | List of all members
google.appengine._internal.antlr3.recognizers.TokenSource Class Reference

if self._state.failed: sys.stdout.write(" failed=%s" % self._state.failed) More...

Inheritance diagram for google.appengine._internal.antlr3.recognizers.TokenSource:
google.appengine._internal.antlr3.recognizers.Lexer google.appengine.api.search.ExpressionLexer.ExpressionLexer google.appengine.api.search.QueryLexer.QueryLexer google.appengine.api.search.expression_parser.ExpressionLexerWithErrors google.appengine.api.search.query_parser.QueryLexerWithErrors

Public Member Functions

def nextToken
 
def __iter__
 
def next
 

Detailed Description

if self._state.failed: sys.stdout.write(" failed=%s" % self._state.failed)

@brief Abstract baseclass for token producers.

A source of tokens must provide a sequence of tokens via nextToken()
and also must reveal it's source of characters; CommonToken's text is
computed from a CharStream; it only store indices into the char stream.

Errors from the lexer are never passed to the parser.  Either you want
to keep going or you do not upon token recognition error.  If you do not
want to continue lexing then you do not want to continue parsing.  Just
throw an exception not under RecognitionException and Java will naturally
toss you all the way out of the recognizers.  If you want to continue
lexing then you should not throw an exception to the parser--it has already
requested a token.  Keep lexing until you get a valid one.  Just report
errors and keep going, looking for a valid token.

Member Function Documentation

def google.appengine._internal.antlr3.recognizers.TokenSource.__iter__ (   self)
The TokenSource is an interator.

The iteration will not include the final EOF token, see also the note
for the next() method.
def google.appengine._internal.antlr3.recognizers.TokenSource.next (   self)
Return next token or raise StopIteration.

Note that this will raise StopIteration when hitting the EOF token,
so EOF will not be part of the iteration.
def google.appengine._internal.antlr3.recognizers.TokenSource.nextToken (   self)
Return a Token object from your input stream (usually a CharStream).

Do not fail/return upon lexing error; keep chewing on the characters
until you get a good one; errors are not passed through to the parser.

The documentation for this class was generated from the following file: