![]() |
App Engine Python SDK
v1.6.9 rev.445
The Python runtime is available as an experimental Preview feature.
|
Public Member Functions | |
def | __init__ |
def | __iter__ |
def | next |
def | from_json |
def | to_json |
def | split_input |
def | validate |
def | begin_slice |
def | end_slice |
def | params_to_json |
def | params_from_json |
![]() | |
def | begin_shard |
def | end_shard |
def | begin_slice |
def | end_slice |
![]() | |
def | to_json_str |
def | from_json_str |
Static Public Attributes | |
string | COUNTER_IO_READ_BYTE = "io-read-byte" |
string | COUNTER_IO_READ_MSEC = "io-read-msec" |
Abstract base class for input readers. InputReader's lifecycle: 1. validate() is called to validate JobConfig. 2. split_input is called to split inputs based on map_job.JobConfig. The class method creates a set of InputReader instances. 3. beging_shard/end_shard/begin_slice/end_slice are called at the time implied by the names. 4. next() is called by each shard on each instance. The output of next() is fed into JobConfig.mapper instance. 5. to_json()/from_json() are used to persist reader's state across multiple slices.
def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.begin_slice | ( | self, | |
slice_ctx | |||
) |
Keeps an internal reference to slice_ctx. Args: slice_ctx: SliceContext singleton instance for this slice.
def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.end_slice | ( | self, | |
slice_ctx | |||
) |
Drops the internal reference to slice_ctx. Args: slice_ctx: SliceContext singleton instance for this slice.
def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.from_json | ( | cls, | |
state | |||
) |
Creates an instance of the InputReader for the given state. Args: state: The InputReader state as returned by to_json. Returns: An instance of the InputReader that can resume iteration.
def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.next | ( | self | ) |
Returns the next input from this input reader. Returns: The next input read by this input reader. The return value is fed into mapper. Raises: StopIteration when no more item is left.
def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.params_from_json | ( | cls, | |
json_params | |||
) |
Reverse function of params_to_json.
def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.params_to_json | ( | cls, | |
params | |||
) |
Translates JobConfig.input_reader_params to json serializable format. For most reader, this may be an identity transformation. Args: params: JobConfig.input_reader_params. Returns: The json serializable format of params.
def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.split_input | ( | cls, | |
job_config | |||
) |
Returns an iterator of input readers. This method returns a container of input readers, one for each shard. The container must have __iter__ defined. http://docs.python.org/2/reference/datamodel.html#object.__iter__ This method should try to split inputs among readers evenly. Args: job_config: an instance of map_job.JobConfig. Returns: An iterator of input readers.
def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.to_json | ( | self | ) |
Returns input reader state for the remaining inputs. Returns: A json-serializable state for the InputReader.
def google.appengine.ext.mapreduce.api.map_job.input_reader.InputReader.validate | ( | cls, | |
job_config | |||
) |
Validates relevant parameters. This method can validate fields which it deems relevant. Args: job_config: an instance of map_job.JobConfig. Raises: errors.BadReaderParamsError: required parameters are missing or invalid.