App Engine Python SDK  v1.6.9 rev.445
The Python runtime is available as an experimental Preview feature.
Public Member Functions | Static Public Attributes | List of all members
google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader Class Reference
Inheritance diagram for google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader:
google.appengine.ext.mapreduce.api.map_job.shard_life_cycle._ShardLifeCycle google.appengine.ext.mapreduce.json_util.JsonMixin google.appengine.ext.mapreduce.api.map_job.abstract_datastore_input_reader.AbstractDatastoreInputReader

Public Member Functions

def __init__
 
def __iter__
 
def next
 
def from_json
 
def to_json
 
def split_input
 
def validate
 
def begin_slice
 
def end_slice
 
def params_to_json
 
def params_from_json
 
- Public Member Functions inherited from google.appengine.ext.mapreduce.api.map_job.shard_life_cycle._ShardLifeCycle
def begin_shard
 
def end_shard
 
def begin_slice
 
def end_slice
 
- Public Member Functions inherited from google.appengine.ext.mapreduce.json_util.JsonMixin
def to_json_str
 
def from_json_str
 

Static Public Attributes

string COUNTER_IO_READ_BYTE = "io-read-byte"
 
string COUNTER_IO_READ_MSEC = "io-read-msec"
 

Detailed Description

Abstract base class for input readers.

InputReader's lifecycle:
1. validate() is called to validate JobConfig.
2. split_input is called to split inputs based on map_job.JobConfig.
   The class method creates a set of InputReader instances.
3. beging_shard/end_shard/begin_slice/end_slice are called at the time
   implied by the names.
4. next() is called by each shard on each instance. The output of next()
   is fed into JobConfig.mapper instance.
5. to_json()/from_json() are used to persist reader's state across multiple
   slices.

Member Function Documentation

def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.begin_slice (   self,
  slice_ctx 
)
Keeps an internal reference to slice_ctx.

Args:
  slice_ctx: SliceContext singleton instance for this slice.
def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.end_slice (   self,
  slice_ctx 
)
Drops the internal reference to slice_ctx.

Args:
  slice_ctx: SliceContext singleton instance for this slice.
def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.from_json (   cls,
  state 
)
Creates an instance of the InputReader for the given state.

Args:
  state: The InputReader state as returned by to_json.

Returns:
  An instance of the InputReader that can resume iteration.
def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.next (   self)
Returns the next input from this input reader.

Returns:
  The next input read by this input reader. The return value is
  fed into mapper.

Raises:
  StopIteration when no more item is left.
def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.params_from_json (   cls,
  json_params 
)
Reverse function of params_to_json.
def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.params_to_json (   cls,
  params 
)
Translates JobConfig.input_reader_params to json serializable format.

For most reader, this may be an identity transformation.

Args:
  params: JobConfig.input_reader_params.

Returns:
  The json serializable format of params.
def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.split_input (   cls,
  job_config 
)
Returns an iterator of input readers.

This method returns a container of input readers,
one for each shard. The container must have __iter__ defined.
http://docs.python.org/2/reference/datamodel.html#object.__iter__

This method should try to split inputs among readers evenly.

Args:
  job_config: an instance of map_job.JobConfig.

Returns:
  An iterator of input readers.
def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.to_json (   self)
Returns input reader state for the remaining inputs.

Returns:
  A json-serializable state for the InputReader.
def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.validate (   cls,
  job_config 
)
Validates relevant parameters.

This method can validate fields which it deems relevant.

Args:
  job_config: an instance of map_job.JobConfig.

Raises:
  errors.BadReaderParamsError: required parameters are missing or invalid.

The documentation for this class was generated from the following file: