![]() |
App Engine Python SDK
v1.6.9 rev.445
The Python runtime is available as an experimental Preview feature.
|
Public Member Functions | |
def | __init__ |
def | AddThrottle |
def | AddThrottles |
def | Register |
def | VerifyThrottleName |
def | AddTransfer |
def | Sleep |
def | TotalTransferred |
Public Attributes | |
get_time | |
thread_sleep | |
start_time | |
transferred | |
prior_block | |
totals | |
throttles | |
last_rotate | |
rotate_mutex | |
Static Public Attributes | |
int | ROTATE_PERIOD = 600 |
A base class for upload rate throttling. Transferring large number of entities, too quickly, could trigger quota limits and cause the transfer process to halt. In order to stay within the application's quota, we throttle the data transfer to a specified limit (across all transfer threads). This class tracks a moving average of some aspect of the transfer rate (bandwidth, records per second, http connections per second). It keeps two windows of counts of bytes transferred, on a per-thread basis. One block is the "current" block, and the other is the "prior" block. It will rotate the counts from current to prior when ROTATE_PERIOD has passed. Thus, the current block will represent from 0 seconds to ROTATE_PERIOD seconds of activity (determined by: time.time() - self.last_rotate). The prior block will always represent a full ROTATE_PERIOD. Sleeping is performed just before a transfer of another block, and is based on the counts transferred *before* the next transfer. It really does not matter how much will be transferred, but only that for all the data transferred SO FAR that we have interspersed enough pauses to ensure the aggregate transfer rate is within the specified limit. These counts are maintained on a per-thread basis, so we do not require any interlocks around incrementing the counts. There IS an interlock on the rotation of the counts because we do not want multiple threads to multiply-rotate the counts. There are various race conditions in the computation and collection of these counts. We do not require precise values, but simply to keep the overall transfer within the bandwidth limits. If a given pause is a little short, or a little long, then the aggregate delays will be correct.
def google.appengine.ext.remote_api.throttle.Throttle.AddTransfer | ( | self, | |
throttle_name, | |||
token_count | |||
) |
Add a count to the amount this thread has transferred. Each time a thread transfers some data, it should call this method to note the amount sent. The counts may be rotated if sufficient time has passed since the last rotation. Args: throttle_name: The name of the throttle to add to. token_count: The number to add to the throttle counter.
def google.appengine.ext.remote_api.throttle.Throttle.Register | ( | self, | |
thread | |||
) |
Register this thread with the throttler.
def google.appengine.ext.remote_api.throttle.Throttle.Sleep | ( | self, | |
throttle_name = None |
|||
) |
Possibly sleep in order to limit the transfer rate. Note that we sleep based on *prior* transfers rather than what we may be about to transfer. The next transfer could put us under/over and that will be rectified *after* that transfer. Net result is that the average transfer rate will remain within bounds. Spiky behavior or uneven rates among the threads could possibly bring the transfer rate above the requested limit for short durations. Args: throttle_name: The name of the throttle to sleep on. If None or omitted, then sleep on all throttles.
def google.appengine.ext.remote_api.throttle.Throttle.TotalTransferred | ( | self, | |
throttle_name | |||
) |
Return the total transferred, and over what period. Args: throttle_name: The name of the throttle to total. Returns: A tuple of the total count and running time for the given throttle name.