App Engine Python SDK  v1.6.9 rev.445
The Python runtime is available as an experimental Preview feature.
Public Member Functions | Public Attributes | List of all members
google.appengine.tools.bulkloader._ProgressDatabase Class Reference
Inheritance diagram for google.appengine.tools.bulkloader._ProgressDatabase:
google.appengine.tools.bulkloader._Database google.appengine.tools.bulkloader.ExportProgressDatabase

Public Member Functions

def __init__
 
def UseProgressData
 
def StoreKeys
 
def UpdateState
 
def DeleteKey
 
def GetProgressStatusGenerator
 
- Public Member Functions inherited from google.appengine.tools.bulkloader._Database
def __init__
 
def ThreadComplete
 

Public Attributes

 prior_key_end
 
 py_type
 
- Public Attributes inherited from google.appengine.tools.bulkloader._Database
 db_filename
 
 primary_conn
 
 primary_thread
 
 secondary_conn
 
 secondary_thread
 
 operation_count
 
 commit_periodicity
 
 existing_table
 
 insert_cursor
 
 update_cursor
 

Additional Inherited Members

- Static Public Attributes inherited from google.appengine.tools.bulkloader._Database
string SIGNATURE_TABLE_NAME = 'bulkloader_database_signature'
 

Detailed Description

Persistently record all progress information during an upload.

This class wraps a very simple SQLite database which records each of
the relevant details from a chunk of work. If the loader is
resumed, then data is replayed out of the database.

Constructor & Destructor Documentation

def google.appengine.tools.bulkloader._ProgressDatabase.__init__ (   self,
  db_filename,
  sql_type,
  py_type,
  signature,
  commit_periodicity = 100 
)
Initialize the ProgressDatabase instance.

Args:
  db_filename: The name of the SQLite database to use.
  sql_type: A string of the SQL type to use for entity keys.
  py_type: The python type of entity keys.
  signature: A string identifying the important invocation options,
used to make sure we are not using an old database.
  commit_periodicity: How many operations to perform between commits.

Member Function Documentation

def google.appengine.tools.bulkloader._ProgressDatabase.DeleteKey (   self,
  progress_key 
)
Delete the entities with the given key from the result database.
def google.appengine.tools.bulkloader._ProgressDatabase.GetProgressStatusGenerator (   self)
Get a generator which yields progress information.

The returned generator will yield a series of 5-tuples that specify
progress information about a prior run of the uploader. The 5-tuples
have the following values:

  progress_key: The unique key to later update this record with new
            progress information.
  state: The last state saved for this progress record.
  kind: The datastore kind of the items for uploading.
  key_start: The starting key of the items for uploading (inclusive).
  key_end: The ending key of the items for uploading (inclusive).

After all incompletely-transferred records are provided, then one
more 5-tuple will be generated:

  None
  DATA_CONSUMED_TO_HERE: A unique string value indicating this record
                     is being provided.
  None
  None
  key_end: An integer value specifying the last data source key that
       was handled by the previous run of the uploader.

The caller should begin uploading records which occur after key_end.

Yields:
  Five-tuples of (progress_key, state, kind, key_start, key_end)
def google.appengine.tools.bulkloader._ProgressDatabase.StoreKeys (   self,
  kind,
  key_start,
  key_end 
)
Record a new progress record, returning a key for later updates.

The specified progress information will be persisted into the database.
A unique key will be returned that identifies this progress state. The
key is later used to (quickly) update this record.

For the progress resumption to proceed properly, calls to StoreKeys
MUST specify monotonically increasing key ranges. This will result in
a database whereby the ID, KEY_START, and KEY_END rows are all
increasing (rather than having ranges out of order).

NOTE: the above precondition is NOT tested by this method (since it
would imply an additional table read or two on each invocation).

Args:
  kind: The kind for the WorkItem
  key_start: The starting key of the WorkItem (inclusive)
  key_end: The end key of the WorkItem (inclusive)

Returns:
  A string to later be used as a unique key to update this state.
def google.appengine.tools.bulkloader._ProgressDatabase.UpdateState (   self,
  key,
  new_state 
)
Update a specified progress record with new information.

Args:
  key: The key for this progress record, returned from StoreKeys
  new_state: The new state to associate with this progress record.
def google.appengine.tools.bulkloader._ProgressDatabase.UseProgressData (   self)
Returns True if the database has progress information.

Note there are two basic cases for progress information:
1) All saved records indicate a successful upload. In this case, we
   need to skip everything transmitted so far and then send the rest.
2) Some records for incomplete transfer are present. These need to be
   sent again, and then we resume sending after all the successful
   data.

Returns:
  True: if the database has progress information.

Raises:
  ResumeError: if there is an error retrieving rows from the database.

The documentation for this class was generated from the following file: