App Engine Python SDK  v1.6.9 rev.445
The Python runtime is available as an experimental Preview feature.
Public Member Functions | Public Attributes | Static Public Attributes | List of all members
google.appengine.ext.remote_api.throttle.Throttle Class Reference
Inheritance diagram for google.appengine.ext.remote_api.throttle.Throttle:

Public Member Functions

def __init__
 
def AddThrottle
 
def AddThrottles
 
def Register
 
def VerifyThrottleName
 
def AddTransfer
 
def Sleep
 
def TotalTransferred
 

Public Attributes

 get_time
 
 thread_sleep
 
 start_time
 
 transferred
 
 prior_block
 
 totals
 
 throttles
 
 last_rotate
 
 rotate_mutex
 

Static Public Attributes

int ROTATE_PERIOD = 600
 

Detailed Description

A base class for upload rate throttling.

Transferring large number of entities, too quickly, could trigger
quota limits and cause the transfer process to halt.  In order to
stay within the application's quota, we throttle the data transfer
to a specified limit (across all transfer threads).

This class tracks a moving average of some aspect of the transfer
rate (bandwidth, records per second, http connections per
second). It keeps two windows of counts of bytes transferred, on a
per-thread basis. One block is the "current" block, and the other is
the "prior" block. It will rotate the counts from current to prior
when ROTATE_PERIOD has passed.  Thus, the current block will
represent from 0 seconds to ROTATE_PERIOD seconds of activity
(determined by: time.time() - self.last_rotate).  The prior block
will always represent a full ROTATE_PERIOD.

Sleeping is performed just before a transfer of another block, and is
based on the counts transferred *before* the next transfer. It really
does not matter how much will be transferred, but only that for all the
data transferred SO FAR that we have interspersed enough pauses to
ensure the aggregate transfer rate is within the specified limit.

These counts are maintained on a per-thread basis, so we do not require
any interlocks around incrementing the counts. There IS an interlock on
the rotation of the counts because we do not want multiple threads to
multiply-rotate the counts.

There are various race conditions in the computation and collection
of these counts. We do not require precise values, but simply to
keep the overall transfer within the bandwidth limits. If a given
pause is a little short, or a little long, then the aggregate delays
will be correct.

Member Function Documentation

def google.appengine.ext.remote_api.throttle.Throttle.AddTransfer (   self,
  throttle_name,
  token_count 
)
Add a count to the amount this thread has transferred.

Each time a thread transfers some data, it should call this method to
note the amount sent. The counts may be rotated if sufficient time
has passed since the last rotation.

Args:
  throttle_name: The name of the throttle to add to.
  token_count: The number to add to the throttle counter.
def google.appengine.ext.remote_api.throttle.Throttle.Register (   self,
  thread 
)
Register this thread with the throttler.
def google.appengine.ext.remote_api.throttle.Throttle.Sleep (   self,
  throttle_name = None 
)
Possibly sleep in order to limit the transfer rate.

Note that we sleep based on *prior* transfers rather than what we
may be about to transfer. The next transfer could put us under/over
and that will be rectified *after* that transfer. Net result is that
the average transfer rate will remain within bounds. Spiky behavior
or uneven rates among the threads could possibly bring the transfer
rate above the requested limit for short durations.

Args:
  throttle_name: The name of the throttle to sleep on.  If None or
omitted, then sleep on all throttles.
def google.appengine.ext.remote_api.throttle.Throttle.TotalTransferred (   self,
  throttle_name 
)
Return the total transferred, and over what period.

Args:
  throttle_name: The name of the throttle to total.

Returns:
  A tuple of the total count and running time for the given throttle name.

The documentation for this class was generated from the following file: