App Engine Python SDK  v1.6.9 rev.445
The Python runtime is available as an experimental Preview feature.
Public Member Functions | Static Public Attributes | List of all members
google.appengine.ext.mapreduce.shuffler._MergingReader Class Reference
Inheritance diagram for google.appengine.ext.mapreduce.shuffler._MergingReader:
google.appengine.ext.mapreduce.input_readers.InputReader google.appengine.ext.mapreduce.json_util.JsonMixin

Public Member Functions

def __init__
 
def __iter__
 
def from_json
 
def to_json
 
def split_input
 
def validate
 
- Public Member Functions inherited from google.appengine.ext.mapreduce.input_readers.InputReader
def __iter__
 
def next
 
def from_json
 
def to_json
 
def split_input
 
def validate
 
- Public Member Functions inherited from google.appengine.ext.mapreduce.json_util.JsonMixin
def to_json_str
 
def from_json_str
 

Static Public Attributes

 expand_parameters = True
 
string FILES_PARAM = "files"
 
string MAX_VALUES_COUNT_PARAM = "max_values_count"
 
string MAX_VALUES_SIZE_PARAM = "max_values_size"
 
- Static Public Attributes inherited from google.appengine.ext.mapreduce.input_readers.InputReader
 expand_parameters = False
 
string NAMESPACE_PARAM = "namespace"
 
string NAMESPACES_PARAM = "namespaces"
 

Detailed Description

Reader which merge-reads multiple sorted KeyValue files.

Reads list of lists of filenames. Each filename list constitutes one shard
and is merged together.

Yields (key, values) tuple. If none of the max_values_count and
max_values_size parameters are not specified, then there will be a single key.
Otherwise multiple (key, values) pairs for the same key will be created,
according to restrictions.

Constructor & Destructor Documentation

def google.appengine.ext.mapreduce.shuffler._MergingReader.__init__ (   self,
  offsets,
  max_values_count,
  max_values_size 
)
Constructor.

Args:
  offsets: offsets for each input file to start from as list of ints.
  max_values_count: maximum number of values to yield for a single value at
a time. Ignored if -1.
  max_values_size: maximum total size of yielded values.  Ignored if -1

Member Function Documentation

def google.appengine.ext.mapreduce.shuffler._MergingReader.__iter__ (   self)
Iterate over records in input files.

self._offsets is always correctly updated so that stopping iterations
doesn't skip records and doesn't read the same record twice.
def google.appengine.ext.mapreduce.shuffler._MergingReader.from_json (   cls,
  json 
)
Restore reader from json state.
def google.appengine.ext.mapreduce.shuffler._MergingReader.split_input (   cls,
  mapper_spec 
)
Split input into multiple shards.
def google.appengine.ext.mapreduce.shuffler._MergingReader.to_json (   self)
Serialize reader state to json.
def google.appengine.ext.mapreduce.shuffler._MergingReader.validate (   cls,
  mapper_spec 
)
Validate reader parameters in mapper_spec.

The documentation for this class was generated from the following file: