App Engine Python SDK  v1.6.9 rev.445
The Python runtime is available as an experimental Preview feature.
Public Member Functions | Public Attributes | Static Public Attributes | List of all members
google.appengine.ext.mapreduce.file_format_parser._Tokenizer Class Reference
Inheritance diagram for google.appengine.ext.mapreduce.file_format_parser._Tokenizer:

Public Member Functions

def __init__
 
def peek
 
def next
 
def consume
 
def consume_if
 
def remainder
 

Public Attributes

 index
 

Static Public Attributes

string SPECIAL_CHARS = '[]()=,'
 
string ESCAPE_CHAR = '\\'
 

Detailed Description

Tokenizes a user supplied format string.

A token is either a special character or a group of characters between
two special characters or the beginning or the end of format string.
Escape character can be used to escape special characters and itself.

Constructor & Destructor Documentation

def google.appengine.ext.mapreduce.file_format_parser._Tokenizer.__init__ (   self,
  format_string 
)
Initialize.

Args:
  format_string: user supplied format string for MapReduce InputReader.

Member Function Documentation

def google.appengine.ext.mapreduce.file_format_parser._Tokenizer.consume (   self,
  expected_token 
)
Consumes the next token which must match expectation.

Args:
  expected_token: the expected value of the next token.

Raises:
  ValueError: raised when the next token doesn't match expected_token.
def google.appengine.ext.mapreduce.file_format_parser._Tokenizer.consume_if (   self,
  token 
)
Consumes the next token when it matches expectation.

Args:
  token: the expected next token.

Returns:
  True when next token matches the argument and is consumed.
  False otherwise.
def google.appengine.ext.mapreduce.file_format_parser._Tokenizer.next (   self,
  advance = True 
)
Returns the next token with surrounding white spaces stripped.

Args:
  advance: boolean. True if underlying buffer should be advanced.

Returns:
  the next token with surrounding whitespaces stripped.
def google.appengine.ext.mapreduce.file_format_parser._Tokenizer.peek (   self)
Returns the next token with surrounding white spaces stripped.

This method does not advance underlying buffer.

Returns:
  the next token with surrounding whitespaces stripped.
def google.appengine.ext.mapreduce.file_format_parser._Tokenizer.remainder (   self)
Returns the number of bytes left to be processed.

The documentation for this class was generated from the following file: