App Engine Python SDK  v1.6.9 rev.445
The Python runtime is available as an experimental Preview feature.
Public Member Functions | List of all members
google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter Class Reference
Inheritance diagram for google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter:
google.appengine.ext.mapreduce.api.map_job.shard_life_cycle._ShardLifeCycle google.appengine.ext.mapreduce.json_util.JsonMixin

Public Member Functions

def __init__
 
def validate
 
def from_json
 
def to_json
 
def create
 
def write
 
def commit_output
 
def begin_slice
 
def end_slice
 
- Public Member Functions inherited from google.appengine.ext.mapreduce.api.map_job.shard_life_cycle._ShardLifeCycle
def begin_shard
 
def end_shard
 
def begin_slice
 
def end_slice
 
- Public Member Functions inherited from google.appengine.ext.mapreduce.json_util.JsonMixin
def to_json_str
 
def from_json_str
 

Detailed Description

Abstract base class for output writers.

OutputWriter's lifecycle:
  0) validate() is called to validate JobConfig.
  1) create() is called, which should create a new instance of output
     writer for the given shard
  2) beging_shard/end_shard/begin_slice/end_slice are called at the time
     implied by the names.
  3) from_json()/to_json() are used to persist writer's state across
     multiple slices.
  4) write() method is called with data yielded by JobConfig.mapper.

Member Function Documentation

def google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter.begin_slice (   self,
  slice_ctx 
)
Keeps an internal reference to slice_ctx.

Args:
  slice_ctx: SliceContext singleton instance for this slice.
def google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter.commit_output (   cls,
  shard_ctx,
  iterator 
)
Saves output references when a shard finishes.

Inside end_shard(), an output writer can optionally use this method
to persist some references to the outputs from this shard
(e.g a list of filenames)

Args:
  shard_ctx: map_job.ShardContext for this shard.
  iterator: an iterator that yields json serializable
references to the outputs from this shard.
Contents from the iterator can be accessible later via
map_job.Job.get_outputs.
def google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter.create (   cls,
  shard_ctx 
)
Create new writer for a shard.

Args:
  shard_ctx: map_job.ShardContext for this shard.
def google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter.end_slice (   self,
  slice_ctx 
)
Drops the internal reference to slice_ctx.

Args:
  slice_ctx: SliceContext singleton instance for this slice.
def google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter.from_json (   cls,
  state 
)
Creates an instance of the OutputWriter for the given json state.

No RPC should take place in this method. Use start_slice/end_slice instead.

Args:
  state: The output writer state as returned by to_json.

Returns:
  An instance of the OutputWriter that can resume writing.
def google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter.to_json (   self)
Returns writer state.

No RPC should take place in this method. Use start_slice/end_slice instead.

Returns:
  A json-serializable state for the OutputWriter instance.
def google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter.validate (   cls,
  job_config 
)
Validates relevant parameters.

This method can validate fields which it deems relevant.

Args:
  job_config: an instance of map_job.JobConfig.

Raises:
  errors.BadWriterParamsError: required parameters are missing or invalid.
def google.appengine.ext.mapreduce.api.map_job.output_writer.OutputWriter.write (   self,
  data 
)
Write data.

Args:
  data: actual data yielded from handler. User is responsible to match the
type expected by this writer to the type yielded by mapper.

The documentation for this class was generated from the following file: