![]() |
App Engine Python SDK
v1.6.9 rev.445
The Python runtime is available as an experimental Preview feature.
|
Public Member Functions | |
def | __str__ |
def | reset_for_retry |
def | advance_for_next_slice |
def | set_for_failure |
def | set_for_abort |
def | set_for_success |
def | copy_from |
def | __eq__ |
def | get_shard_number |
def | get_shard_id |
def | kind |
def | shard_id_from_number |
def | get_key_by_shard_id |
def | get_by_shard_id |
def | find_by_mapreduce_state |
def | find_all_by_mapreduce_state |
def | calculate_keys_by_mapreduce_state |
def | create_new |
![]() | |
def | __new__ |
def | __init__ |
def | key |
def | put |
def | delete |
def | is_saved |
def | has_key |
def | dynamic_properties |
def | instance_properties |
def | parent |
def | parent_key |
def | to_xml |
def | get |
def | get_by_key_name |
def | get_by_id |
def | get_or_insert |
def | all |
def | gql |
def | from_entity |
def | kind |
def | entity_type |
def | properties |
def | fields |
Public Attributes | |
last_work_item | |
active | |
result_status | |
counters_map | |
slice_id | |
slice_start_time | |
slice_request_id | |
slice_retries | |
acquired_once | |
Static Public Attributes | |
string | RESULT_SUCCESS = "success" |
string | RESULT_FAILED = "failed" |
string | RESULT_ABORTED = "aborted" |
tuple | mapreduce_id = db.StringProperty(required=True) |
tuple | active = db.BooleanProperty(default=True, indexed=False) |
tuple | counters_map |
tuple | result_status = db.StringProperty(choices=_RESULTS, indexed=False) |
tuple | retries = db.IntegerProperty(default=0, indexed=False) |
tuple | writer_state = json_util.JsonProperty(dict, indexed=False) |
tuple | slice_id = db.IntegerProperty(default=0, indexed=False) |
tuple | slice_start_time = db.DateTimeProperty(indexed=False) |
tuple | slice_request_id = db.ByteStringProperty(indexed=False) |
tuple | slice_retries = db.IntegerProperty(default=0, indexed=False) |
tuple | acquired_once = db.BooleanProperty(default=False, indexed=False) |
tuple | update_time = db.DateTimeProperty(auto_now=True, indexed=False) |
tuple | shard_description = db.TextProperty(default="") |
tuple | last_work_item = db.TextProperty(default="") |
![]() | |
save = put | |
Properties | |
shard_number = property(get_shard_number) | |
shard_id = property(get_shard_id) | |
Single shard execution state. The shard state is stored in the datastore and is later aggregated by controller task. ShardState key_name is equal to shard_id. Shard state contains critical state to ensure the correctness of shard execution. It is the single source of truth about a shard's progress. For example: 1. A slice is allowed to run only if its payload matches shard state's expectation. 2. A slice is considered running only if it has acquired the shard's lock. 3. A slice is considered done only if it has successfully committed shard state to db. Properties about the shard: active: if we have this shard still running as boolean. counters_map: shard's counters map as CountersMap. All counters yielded within mapreduce are stored here. mapreduce_id: unique id of the mapreduce. shard_id: unique id of this shard as string. shard_number: ordered number for this shard. retries: the number of times this shard has been retried. result_status: If not None, the final status of this shard. update_time: The last time this shard state was updated. shard_description: A string description of the work this shard will do. last_work_item: A string description of the last work item processed. writer_state: writer state for this shard. The shard's output writer instance can save in-memory output references to this field in its "finalize" method. Properties about slice management: slice_id: slice id of current executing slice. A slice's task will not run unless its slice_id matches this. Initial value is 0. By the end of slice execution, this number is incremented by 1. slice_start_time: a slice updates this to now at the beginning of execution. If the transaction succeeds, the current task holds a lease of slice duration + some grace period. During this time, no other task with the same slice_id will execute. Upon slice failure, the task should try to unset this value to allow retries to carry on ASAP. slice_request_id: the request id that holds/held the lease. When lease has expired, new request needs to verify that said request has indeed ended according to logs API. Do this only when lease has expired because logs API is expensive. This field should always be set/unset with slice_start_time. It is possible Logs API doesn't log a request at all or doesn't log the end of a request. So a new request can proceed after a long conservative timeout. slice_retries: the number of times a slice has been retried due to processing data when lock is held. Taskqueue/datastore errors related to slice/shard management are not counted. This count is only a lower bound and is used to determined when to fail a slice completely. acquired_once: whether the lock for this slice has been acquired at least once. When this is True, duplicates in outputs are possible.
def google.appengine.ext.mapreduce.model.ShardState.advance_for_next_slice | ( | self, | |
recovery_slice = False |
|||
) |
Advance self for next slice. Args: recovery_slice: True if this slice is running recovery logic. See handlers.MapperWorkerCallbackHandler._attempt_slice_recovery for more info.
def google.appengine.ext.mapreduce.model.ShardState.calculate_keys_by_mapreduce_state | ( | cls, | |
mapreduce_state | |||
) |
Calculate all shard states keys for given mapreduce. Args: mapreduce_state: MapreduceState instance Returns: A list of keys for shard states, sorted by shard id. The corresponding shard states may not exist.
def google.appengine.ext.mapreduce.model.ShardState.copy_from | ( | self, | |
other_state | |||
) |
Copy data from another shard state entity to self.
def google.appengine.ext.mapreduce.model.ShardState.create_new | ( | cls, | |
mapreduce_id, | |||
shard_number | |||
) |
Create new shard state. Args: mapreduce_id: unique mapreduce id as string. shard_number: shard number for which to create shard state. Returns: new instance of ShardState ready to put into datastore.
def google.appengine.ext.mapreduce.model.ShardState.find_all_by_mapreduce_state | ( | cls, | |
mapreduce_state | |||
) |
Find all shard states for given mapreduce. Args: mapreduce_state: MapreduceState instance Yields: shard states sorted by shard id.
def google.appengine.ext.mapreduce.model.ShardState.find_by_mapreduce_state | ( | cls, | |
mapreduce_state | |||
) |
Find all shard states for given mapreduce. Deprecated. Use find_all_by_mapreduce_state. This will be removed after 1.8.9 release. Args: mapreduce_state: MapreduceState instance Returns: A list of ShardStates.
def google.appengine.ext.mapreduce.model.ShardState.get_by_shard_id | ( | cls, | |
shard_id | |||
) |
Get shard state from datastore by shard_id. Args: shard_id: shard id as string. Returns: ShardState for given shard id or None if it's not found.
def google.appengine.ext.mapreduce.model.ShardState.get_key_by_shard_id | ( | cls, | |
shard_id | |||
) |
Retrieves the Key for this ShardState. Args: shard_id: The shard ID to fetch. Returns: The Datatore key to use to retrieve this ShardState.
def google.appengine.ext.mapreduce.model.ShardState.get_shard_id | ( | self | ) |
Returns the shard ID.
def google.appengine.ext.mapreduce.model.ShardState.get_shard_number | ( | self | ) |
Gets the shard number from the key name.
def google.appengine.ext.mapreduce.model.ShardState.kind | ( | cls | ) |
Returns entity kind.
def google.appengine.ext.mapreduce.model.ShardState.reset_for_retry | ( | self | ) |
Reset self for shard retry.
def google.appengine.ext.mapreduce.model.ShardState.shard_id_from_number | ( | cls, | |
mapreduce_id, | |||
shard_number | |||
) |
Get shard id by mapreduce id and shard number. Args: mapreduce_id: mapreduce id as string. shard_number: shard number to compute id for as int. Returns: shard id as string.
|
static |