App Engine Python SDK  v1.6.9 rev.445
The Python runtime is available as an experimental Preview feature.
Public Member Functions | Public Attributes | List of all members
google.appengine._internal.antlr3.streams.CommonTokenStream Class Reference

Token streams TokenStream +- CommonTokenStream

More...

Inheritance diagram for google.appengine._internal.antlr3.streams.CommonTokenStream:
google.appengine._internal.antlr3.streams.TokenStream google.appengine._internal.antlr3.streams.IntStream google.appengine._internal.antlr3.streams.TokenRewriteStream

Public Member Functions

def __init__
 
def setTokenSource
 
def reset
 
def fillBuffer
 
def consume
 
def skipOffTokenChannels
 
def skipOffTokenChannelsReverse
 
def setTokenTypeChannel
 
def discardTokenType
 
def getTokens
 
def LT
 
def LB
 
def get
 
def LA
 
def mark
 
def release
 
def size
 
def index
 
def rewind
 
def seek
 
def getTokenSource
 
def getSourceName
 
def toString
 
- Public Member Functions inherited from google.appengine._internal.antlr3.streams.TokenStream
def LT
 
def get
 
def getTokenSource
 
def toString
 
- Public Member Functions inherited from google.appengine._internal.antlr3.streams.IntStream
def consume
 
def LA
 
def mark
 
def index
 
def rewind
 
def release
 
def seek
 
def size
 
def getSourceName
 

Public Attributes

 tokenSource
 
 tokens
 
 p
 
 channel
 
 lastMarker
 

Detailed Description

Token streams TokenStream +- CommonTokenStream

@brief The most common stream of tokens

The most common stream of tokens is one where every token is buffered up
and tokens are prefiltered for a certain channel (the parser will only
see these tokens and cannot change the filter channel number during the
parse).

Constructor & Destructor Documentation

def google.appengine._internal.antlr3.streams.CommonTokenStream.__init__ (   self,
  tokenSource = None,
  channel = DEFAULT_CHANNEL 
)
@param tokenSource A TokenSource instance (usually a Lexer) to pull
    the tokens from.

@param channel Skip tokens on any channel but this one; this is how we
    skip whitespace...

Member Function Documentation

def google.appengine._internal.antlr3.streams.CommonTokenStream.consume (   self)
Move the input pointer to the next incoming token.  The stream
must become active with LT(1) available.  consume() simply
moves the input pointer so that LT(1) points at the next
input symbol. Consume at least one token.

Walk past any token not on the channel the parser is listening to.
def google.appengine._internal.antlr3.streams.CommonTokenStream.fillBuffer (   self)
Load all tokens from the token source and put in tokens.
    This is done upon first LT request because you might want to
set some token type / channel overrides before filling buffer.
def google.appengine._internal.antlr3.streams.CommonTokenStream.get (   self,
  i 
)
Return absolute token i; ignore which channel the tokens are on;
that is, count all tokens not just on-channel tokens.
def google.appengine._internal.antlr3.streams.CommonTokenStream.getTokens (   self,
  start = None,
  stop = None,
  types = None 
)
Given a start and stop index, return a list of all tokens in
the token type set.  Return None if no tokens were found.  This
method looks at both on and off channel tokens.
def google.appengine._internal.antlr3.streams.CommonTokenStream.LB (   self,
  k 
)
Look backwards k tokens on-channel tokens
def google.appengine._internal.antlr3.streams.CommonTokenStream.LT (   self,
  k 
)
Get the ith token from the current position 1..n where k=1 is the
first symbol of lookahead.
def google.appengine._internal.antlr3.streams.CommonTokenStream.setTokenSource (   self,
  tokenSource 
)
Reset this token stream by setting its token source.
def google.appengine._internal.antlr3.streams.CommonTokenStream.setTokenTypeChannel (   self,
  ttype,
  channel 
)
A simple filter mechanism whereby you can tell this token stream
to force all tokens of type ttype to be on channel.  For example,
when interpreting, we cannot exec actions so we need to tell
the stream to force all WS and NEWLINE to be a different, ignored
channel.
def google.appengine._internal.antlr3.streams.CommonTokenStream.skipOffTokenChannels (   self,
  i 
)
Given a starting index, return the index of the first on-channel
token.

The documentation for this class was generated from the following file: