App Engine Python SDK  v1.6.9 rev.445
The Python runtime is available as an experimental Preview feature.
Public Member Functions | Static Public Attributes | List of all members
google.appengine.ext.mapreduce.input_readers.BlobstoreZipLineInputReader Class Reference
Inheritance diagram for google.appengine.ext.mapreduce.input_readers.BlobstoreZipLineInputReader:
google.appengine.ext.mapreduce.input_readers.InputReader google.appengine.ext.mapreduce.json_util.JsonMixin

Public Member Functions

def __init__
 
def validate
 
def split_input
 
def next
 
def to_json
 
def from_json
 
def __str__
 
- Public Member Functions inherited from google.appengine.ext.mapreduce.input_readers.InputReader
def __iter__
 
def next
 
def from_json
 
def to_json
 
def split_input
 
def validate
 
- Public Member Functions inherited from google.appengine.ext.mapreduce.json_util.JsonMixin
def to_json_str
 
def from_json_str
 

Static Public Attributes

string BLOB_KEYS_PARAM = "blob_keys"
 
string BLOB_KEY_PARAM = "blob_key"
 
string START_FILE_INDEX_PARAM = "start_file_index"
 
string END_FILE_INDEX_PARAM = "end_file_index"
 
string OFFSET_PARAM = "offset"
 
- Static Public Attributes inherited from google.appengine.ext.mapreduce.input_readers.InputReader
 expand_parameters = False
 
string NAMESPACE_PARAM = "namespace"
 
string NAMESPACES_PARAM = "namespaces"
 

Detailed Description

Input reader for newline delimited files in zip archives from Blobstore.

This has the same external interface as the BlobstoreLineInputReader, in that
it takes a list of blobs as its input and yields lines to the reader.
However the blobs themselves are expected to be zip archives of line delimited
files instead of the files themselves.

This is useful as many line delimited files gain greatly from compression.

Constructor & Destructor Documentation

def google.appengine.ext.mapreduce.input_readers.BlobstoreZipLineInputReader.__init__ (   self,
  blob_key,
  start_file_index,
  end_file_index,
  offset,
  _reader = blobstore.BlobReader 
)
Initializes this instance with the given blob key and file range.

This BlobstoreZipLineInputReader will read from the file with index
start_file_index up to but not including the file with index end_file_index.
It will return lines starting at offset within file[start_file_index]

Args:
  blob_key: the BlobKey that this input reader is processing.
  start_file_index: the index of the first file to read within the zip.
  end_file_index: the index of the first file that will not be read.
  offset: the byte offset within blob_key.zip[start_file_index] to start
reading. The reader will continue to the end of the file.
  _reader: a callable that returns a file-like object for reading blobs.
  Used for dependency injection.

Member Function Documentation

def google.appengine.ext.mapreduce.input_readers.BlobstoreZipLineInputReader.__str__ (   self)
Returns the string representation of this reader.

Returns:
  string blobkey:[start file num, end file num]:current offset.
def google.appengine.ext.mapreduce.input_readers.BlobstoreZipLineInputReader.from_json (   cls,
  json,
  _reader = blobstore.BlobReader 
)
Creates an instance of the InputReader for the given input shard state.

Args:
  json: The InputReader state as a dict-like object.
  _reader: For dependency injection.

Returns:
  An instance of the InputReader configured using the values of json.
def google.appengine.ext.mapreduce.input_readers.BlobstoreZipLineInputReader.next (   self)
Returns the next line from this input reader as (lineinfo, line) tuple.

Returns:
  The next input from this input reader, in the form of a 2-tuple.
  The first element of the tuple describes the source, it is itself
a tuple (blobkey, filenumber, byteoffset).
  The second element of the tuple is the line found at that offset.
def google.appengine.ext.mapreduce.input_readers.BlobstoreZipLineInputReader.split_input (   cls,
  mapper_spec,
  _reader = blobstore.BlobReader 
)
Returns a list of input readers for the input spec.

Args:
  mapper_spec: The MapperSpec for this InputReader. Must contain
  'blob_keys' parameter with one or more blob keys.
  _reader: a callable that returns a file-like object for reading blobs.
  Used for dependency injection.

Returns:
  A list of InputReaders spanning the subfiles within the blobs.
  There will be at least one reader per blob, but it will otherwise
  attempt to keep the expanded size even.
def google.appengine.ext.mapreduce.input_readers.BlobstoreZipLineInputReader.to_json (   self)
Returns an input shard state for the remaining inputs.

Returns:
  A json-izable version of the remaining InputReader.
def google.appengine.ext.mapreduce.input_readers.BlobstoreZipLineInputReader.validate (   cls,
  mapper_spec 
)
Validates mapper spec and all mapper parameters.

Args:
  mapper_spec: The MapperSpec for this InputReader.

Raises:
  BadReaderParamsError: required parameters are missing or invalid.

The documentation for this class was generated from the following file: