![]() |
App Engine Python SDK
v1.6.9 rev.445
The Python runtime is available as an experimental Preview feature.
|
Public Member Functions | |
def | __init__ |
def | validate |
def | from_json |
def | to_json |
def | create |
def | write |
def | commit_output |
def | begin_slice |
def | end_slice |
![]() | |
def | begin_shard |
def | end_shard |
def | begin_slice |
def | end_slice |
![]() | |
def | to_json_str |
def | from_json_str |
Abstract base class for output writers. OutputWriter's lifecycle: 0) validate() is called to validate JobConfig. 1) create() is called, which should create a new instance of output writer for the given shard 2) beging_shard/end_shard/begin_slice/end_slice are called at the time implied by the names. 3) from_json()/to_json() are used to persist writer's state across multiple slices. 4) write() method is called with data yielded by JobConfig.mapper.
def google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter.begin_slice | ( | self, | |
slice_ctx | |||
) |
Keeps an internal reference to slice_ctx. Args: slice_ctx: SliceContext singleton instance for this slice.
def google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter.commit_output | ( | cls, | |
shard_ctx, | |||
iterator | |||
) |
Saves output references when a shard finishes. Inside end_shard(), an output writer can optionally use this method to persist some references to the outputs from this shard (e.g a list of filenames) Args: shard_ctx: map_job.ShardContext for this shard. iterator: an iterator that yields json serializable references to the outputs from this shard. Contents from the iterator can be accessible later via map_job.Job.get_outputs.
def google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter.create | ( | cls, | |
shard_ctx | |||
) |
Create new writer for a shard. Args: shard_ctx: map_job.ShardContext for this shard.
def google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter.end_slice | ( | self, | |
slice_ctx | |||
) |
Drops the internal reference to slice_ctx. Args: slice_ctx: SliceContext singleton instance for this slice.
def google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter.from_json | ( | cls, | |
state | |||
) |
Creates an instance of the OutputWriter for the given json state. No RPC should take place in this method. Use start_slice/end_slice instead. Args: state: The output writer state as returned by to_json. Returns: An instance of the OutputWriter that can resume writing.
def google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter.to_json | ( | self | ) |
Returns writer state. No RPC should take place in this method. Use start_slice/end_slice instead. Returns: A json-serializable state for the OutputWriter instance.
def google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter.validate | ( | cls, | |
job_config | |||
) |
Validates relevant parameters. This method can validate fields which it deems relevant. Args: job_config: an instance of map_job.JobConfig. Raises: errors.BadWriterParamsError: required parameters are missing or invalid.
def google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter.write | ( | self, | |
data | |||
) |
Write data. Args: data: actual data yielded from handler. User is responsible to match the type expected by this writer to the type yielded by mapper.