App Engine Python SDK  v1.6.9 rev.445
The Python runtime is available as an experimental Preview feature.
Public Member Functions | List of all members
google.appengine.ext.mapreduce.shuffler._HashingBlobstoreOutputWriter Class Reference
Inheritance diagram for google.appengine.ext.mapreduce.shuffler._HashingBlobstoreOutputWriter:
google.appengine.ext.mapreduce.output_writers.BlobstoreOutputWriterBase google.appengine.ext.mapreduce.output_writers.FileOutputWriterBase google.appengine.ext.mapreduce.output_writers.OutputWriter google.appengine.ext.mapreduce.json_util.JsonMixin

Public Member Functions

def __init__
 
def validate
 
def init_job
 
def finalize_job
 
def from_json
 
def to_json
 
def create
 
def get_filenames
 
def finalize
 
def write
 
- Public Member Functions inherited from google.appengine.ext.mapreduce.output_writers.FileOutputWriterBase
def __init__
 
def validate
 
def init_job
 
def finalize_job
 
def from_json
 
def to_json
 
def create
 
def finalize
 
def get_filenames
 
- Public Member Functions inherited from google.appengine.ext.mapreduce.output_writers.OutputWriter
def validate
 
def init_job
 
def finalize_job
 
def from_json
 
def to_json
 
def create
 
def write
 
def finalize
 
def get_filenames
 
- Public Member Functions inherited from google.appengine.ext.mapreduce.json_util.JsonMixin
def to_json_str
 
def from_json_str
 

Additional Inherited Members

- Static Public Attributes inherited from google.appengine.ext.mapreduce.output_writers.FileOutputWriterBase
string OUTPUT_SHARDING_PARAM = "output_sharding"
 
string OUTPUT_SHARDING_NONE = "none"
 
string OUTPUT_SHARDING_INPUT_SHARDS = "input"
 
string OUTPUT_FILESYSTEM_PARAM = "filesystem"
 
string GS_BUCKET_NAME_PARAM = "gs_bucket_name"
 
string GS_ACL_PARAM = "gs_acl"
 

Detailed Description

An OutputWriter which outputs data into blobstore in key-value format.

The output is tailored towards shuffler needs. It shards key/values using
key hash modulo number of output files.

Constructor & Destructor Documentation

def google.appengine.ext.mapreduce.shuffler._HashingBlobstoreOutputWriter.__init__ (   self,
  filenames 
)
Constructor.

Args:
  filenames: list of filenames that this writer outputs to.

Member Function Documentation

def google.appengine.ext.mapreduce.shuffler._HashingBlobstoreOutputWriter.create (   cls,
  mr_spec,
  shard_number,
  shard_attempt,
  _writer_state = None 
)
Inherit docs.
def google.appengine.ext.mapreduce.shuffler._HashingBlobstoreOutputWriter.finalize_job (   cls,
  mapreduce_state 
)
Finalize job-level writer state.

Args:
  mapreduce_state: an instance of model.MapreduceState describing current
job. State can be modified during finalization.
def google.appengine.ext.mapreduce.shuffler._HashingBlobstoreOutputWriter.from_json (   cls,
  json 
)
Creates an instance of the OutputWriter for the given json state.

Args:
  json: The OutputWriter state as a dict-like object.

Returns:
  An instance of the OutputWriter configured using the values of json.
def google.appengine.ext.mapreduce.shuffler._HashingBlobstoreOutputWriter.get_filenames (   cls,
  mapreduce_state 
)
See parent class.
def google.appengine.ext.mapreduce.shuffler._HashingBlobstoreOutputWriter.init_job (   cls,
  mapreduce_state 
)
Initialize job-level writer state.

Args:
  mapreduce_state: an instance of model.MapreduceState describing current
  job. State can be modified during initialization.
def google.appengine.ext.mapreduce.shuffler._HashingBlobstoreOutputWriter.to_json (   self)
Returns writer state to serialize in json.

Returns:
  A json-izable version of the OutputWriter state.
def google.appengine.ext.mapreduce.shuffler._HashingBlobstoreOutputWriter.validate (   cls,
  mapper_spec 
)
Validates mapper specification.

Args:
  mapper_spec: an instance of model.MapperSpec to validate.
def google.appengine.ext.mapreduce.shuffler._HashingBlobstoreOutputWriter.write (   self,
  data 
)
Write data.

Args:
  data: actual data yielded from handler. Type is writer-specific.

The documentation for this class was generated from the following file: