App Engine Python SDK  v1.6.9 rev.445
The Python runtime is available as an experimental Preview feature.
Public Member Functions | Public Attributes | Static Public Attributes | Properties | List of all members
google.appengine.ext.mapreduce.model.ShardState Class Reference
Inheritance diagram for google.appengine.ext.mapreduce.model.ShardState:
google.appengine.ext.db.Model

Public Member Functions

def __str__
 
def reset_for_retry
 
def advance_for_next_slice
 
def set_for_failure
 
def set_for_abort
 
def set_for_success
 
def copy_from
 
def __eq__
 
def get_shard_number
 
def get_shard_id
 
def kind
 
def shard_id_from_number
 
def get_key_by_shard_id
 
def get_by_shard_id
 
def find_by_mapreduce_state
 
def find_all_by_mapreduce_state
 
def calculate_keys_by_mapreduce_state
 
def create_new
 
- Public Member Functions inherited from google.appengine.ext.db.Model
def __new__
 
def __init__
 
def key
 
def put
 
def delete
 
def is_saved
 
def has_key
 
def dynamic_properties
 
def instance_properties
 
def parent
 
def parent_key
 
def to_xml
 
def get
 
def get_by_key_name
 
def get_by_id
 
def get_or_insert
 
def all
 
def gql
 
def from_entity
 
def kind
 
def entity_type
 
def properties
 
def fields
 

Public Attributes

 last_work_item
 
 active
 
 result_status
 
 counters_map
 
 slice_id
 
 slice_start_time
 
 slice_request_id
 
 slice_retries
 
 acquired_once
 

Static Public Attributes

string RESULT_SUCCESS = "success"
 
string RESULT_FAILED = "failed"
 
string RESULT_ABORTED = "aborted"
 
tuple mapreduce_id = db.StringProperty(required=True)
 
tuple active = db.BooleanProperty(default=True, indexed=False)
 
tuple counters_map
 
tuple result_status = db.StringProperty(choices=_RESULTS, indexed=False)
 
tuple retries = db.IntegerProperty(default=0, indexed=False)
 
tuple writer_state = json_util.JsonProperty(dict, indexed=False)
 
tuple slice_id = db.IntegerProperty(default=0, indexed=False)
 
tuple slice_start_time = db.DateTimeProperty(indexed=False)
 
tuple slice_request_id = db.ByteStringProperty(indexed=False)
 
tuple slice_retries = db.IntegerProperty(default=0, indexed=False)
 
tuple acquired_once = db.BooleanProperty(default=False, indexed=False)
 
tuple update_time = db.DateTimeProperty(auto_now=True, indexed=False)
 
tuple shard_description = db.TextProperty(default="")
 
tuple last_work_item = db.TextProperty(default="")
 
- Static Public Attributes inherited from google.appengine.ext.db.Model
 save = put
 

Properties

 shard_number = property(get_shard_number)
 
 shard_id = property(get_shard_id)
 

Detailed Description

Single shard execution state.

The shard state is stored in the datastore and is later aggregated by
controller task. ShardState key_name is equal to shard_id.

Shard state contains critical state to ensure the correctness of
shard execution. It is the single source of truth about a shard's
progress. For example:
1. A slice is allowed to run only if its payload matches shard state's
   expectation.
2. A slice is considered running only if it has acquired the shard's lock.
3. A slice is considered done only if it has successfully committed shard
   state to db.

Properties about the shard:
  active: if we have this shard still running as boolean.
  counters_map: shard's counters map as CountersMap. All counters yielded
    within mapreduce are stored here.
  mapreduce_id: unique id of the mapreduce.
  shard_id: unique id of this shard as string.
  shard_number: ordered number for this shard.
  retries: the number of times this shard has been retried.
  result_status: If not None, the final status of this shard.
  update_time: The last time this shard state was updated.
  shard_description: A string description of the work this shard will do.
  last_work_item: A string description of the last work item processed.
  writer_state: writer state for this shard. The shard's output writer
    instance can save in-memory output references to this field in its
    "finalize" method.

 Properties about slice management:
  slice_id: slice id of current executing slice. A slice's task
    will not run unless its slice_id matches this. Initial
    value is 0. By the end of slice execution, this number is
    incremented by 1.
  slice_start_time: a slice updates this to now at the beginning of
    execution. If the transaction succeeds, the current task holds
    a lease of slice duration + some grace period. During this time, no
    other task with the same slice_id will execute. Upon slice failure,
    the task should try to unset this value to allow retries to carry on
    ASAP.
  slice_request_id: the request id that holds/held the lease. When lease has
    expired, new request needs to verify that said request has indeed
    ended according to logs API. Do this only when lease has expired
    because logs API is expensive. This field should always be set/unset
    with slice_start_time. It is possible Logs API doesn't log a request
    at all or doesn't log the end of a request. So a new request can
    proceed after a long conservative timeout.
  slice_retries: the number of times a slice has been retried due to
    processing data when lock is held. Taskqueue/datastore errors
    related to slice/shard management are not counted. This count is
    only a lower bound and is used to determined when to fail a slice
    completely.
  acquired_once: whether the lock for this slice has been acquired at
    least once. When this is True, duplicates in outputs are possible.

Member Function Documentation

def google.appengine.ext.mapreduce.model.ShardState.advance_for_next_slice (   self,
  recovery_slice = False 
)
Advance self for next slice.

Args:
  recovery_slice: True if this slice is running recovery logic.
See handlers.MapperWorkerCallbackHandler._attempt_slice_recovery
for more info.
def google.appengine.ext.mapreduce.model.ShardState.calculate_keys_by_mapreduce_state (   cls,
  mapreduce_state 
)
Calculate all shard states keys for given mapreduce.

Args:
  mapreduce_state: MapreduceState instance

Returns:
  A list of keys for shard states, sorted by shard id.
  The corresponding shard states may not exist.
def google.appengine.ext.mapreduce.model.ShardState.copy_from (   self,
  other_state 
)
Copy data from another shard state entity to self.
def google.appengine.ext.mapreduce.model.ShardState.create_new (   cls,
  mapreduce_id,
  shard_number 
)
Create new shard state.

Args:
  mapreduce_id: unique mapreduce id as string.
  shard_number: shard number for which to create shard state.

Returns:
  new instance of ShardState ready to put into datastore.
def google.appengine.ext.mapreduce.model.ShardState.find_all_by_mapreduce_state (   cls,
  mapreduce_state 
)
Find all shard states for given mapreduce.

Args:
  mapreduce_state: MapreduceState instance

Yields:
  shard states sorted by shard id.
def google.appengine.ext.mapreduce.model.ShardState.find_by_mapreduce_state (   cls,
  mapreduce_state 
)
Find all shard states for given mapreduce.

Deprecated. Use find_all_by_mapreduce_state.
This will be removed after 1.8.9 release.

Args:
  mapreduce_state: MapreduceState instance

Returns:
  A list of ShardStates.
def google.appengine.ext.mapreduce.model.ShardState.get_by_shard_id (   cls,
  shard_id 
)
Get shard state from datastore by shard_id.

Args:
  shard_id: shard id as string.

Returns:
  ShardState for given shard id or None if it's not found.
def google.appengine.ext.mapreduce.model.ShardState.get_key_by_shard_id (   cls,
  shard_id 
)
Retrieves the Key for this ShardState.

Args:
  shard_id: The shard ID to fetch.

Returns:
  The Datatore key to use to retrieve this ShardState.
def google.appengine.ext.mapreduce.model.ShardState.get_shard_id (   self)
Returns the shard ID.
def google.appengine.ext.mapreduce.model.ShardState.get_shard_number (   self)
Gets the shard number from the key name.
def google.appengine.ext.mapreduce.model.ShardState.kind (   cls)
Returns entity kind.
def google.appengine.ext.mapreduce.model.ShardState.reset_for_retry (   self)
Reset self for shard retry.
def google.appengine.ext.mapreduce.model.ShardState.shard_id_from_number (   cls,
  mapreduce_id,
  shard_number 
)
Get shard id by mapreduce id and shard number.

Args:
  mapreduce_id: mapreduce id as string.
  shard_number: shard number to compute id for as int.

Returns:
  shard id as string.

Member Data Documentation

tuple google.appengine.ext.mapreduce.model.ShardState.counters_map
static
Initial value:
1 = json_util.JsonProperty(
2  CountersMap, default=CountersMap(), indexed=False)

The documentation for this class was generated from the following file: