airflow.gcp.hooks.cloud_storage_transfer_service
¶
This module contains a Google Storage Transfer Service Hook.
Module Contents¶
-
class
airflow.gcp.hooks.cloud_storage_transfer_service.
GcpTransferJobsStatus
[source]¶ Class with GCP Transfer jobs statuses.
-
class
airflow.gcp.hooks.cloud_storage_transfer_service.
GcpTransferOperationStatus
[source]¶ Class with GCP Transfer operations statuses.
-
airflow.gcp.hooks.cloud_storage_transfer_service.
ALREADY_EXISTING_IN_SINK
= overwriteObjectsAlreadyExistingInSink[source]¶
-
airflow.gcp.hooks.cloud_storage_transfer_service.
TRANSFER_JOB_FIELD_MASK
= update_transfer_job_field_mask[source]¶
-
class
airflow.gcp.hooks.cloud_storage_transfer_service.
GCPTransferServiceHook
(api_version:str='v1', gcp_conn_id:str='google_cloud_default', delegate_to:str=None)[source]¶ Bases:
airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook
Hook for Google Storage Transfer Service.
All the methods in the hook where project_id is used must be called with keyword arguments rather than positional.
-
get_conn
(self)[source]¶ Retrieves connection to Google Storage Transfer service.
- Returns
Google Storage Transfer service object
- Return type
-
create_transfer_job
(self, body:Dict)[source]¶ Creates a transfer job that runs periodically.
- Parameters
body (dict) – (Required) A request body, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/patch#request-body
- Returns
transfer job. See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TransferJob
- Return type
-
get_transfer_job
(self, job_name:str, project_id:str=None)[source]¶ Gets the latest state of a long-running operation in Google Storage Transfer Service.
-
list_transfer_job
(self, request_filter:Dict=None, **kwargs)[source]¶ Lists long-running operations in Google Storage Transfer Service that match the specified filter.
- Parameters
request_filter (dict) – (Required) A request filter, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter
- Returns
List of Transfer Jobs
- Return type
-
update_transfer_job
(self, job_name:str, body:Dict)[source]¶ Updates a transfer job that runs periodically.
- Parameters
job_name (str) – (Required) Name of the job to be updated
body (dict) – A request body, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/patch#request-body
- Returns
If successful, TransferJob.
- Return type
-
delete_transfer_job
(self, job_name:str, project_id:str=None)[source]¶ Deletes a transfer job. This is a soft delete. After a transfer job is deleted, the job and all the transfer executions are subject to garbage collection. Transfer jobs become eligible for garbage collection 30 days after soft delete.
-
cancel_transfer_operation
(self, operation_name:str)[source]¶ Cancels an transfer operation in Google Storage Transfer Service.
-
get_transfer_operation
(self, operation_name:str)[source]¶ Gets an transfer operation in Google Storage Transfer Service.
- Parameters
operation_name (str) – (Required) Name of the transfer operation.
- Returns
transfer operation See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/Operation
- Return type
-
list_transfer_operations
(self, request_filter:Dict=None, **kwargs)[source]¶ Gets an transfer operation in Google Storage Transfer Service.
- Parameters
request_filter (dict) –
(Required) A request filter, as described in https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/list#body.QUERY_PARAMETERS.filter With one additional improvement:
project_id is optional if you have a project id defined in the connection See: Google Cloud Platform Connection
- Returns
transfer operation
- Return type
-
pause_transfer_operation
(self, operation_name:str)[source]¶ Pauses an transfer operation in Google Storage Transfer Service.
-
resume_transfer_operation
(self, operation_name:str)[source]¶ Resumes an transfer operation in Google Storage Transfer Service.
-
wait_for_transfer_job
(self, job:Dict, expected_statuses:Tuple[str]=(GcpTransferOperationStatus.SUCCESS, ), timeout:Optional[Union[float, timedelta]]=None)[source]¶ Waits until the job reaches the expected state.
- Parameters
job (dict) – Transfer job See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs#TransferJob
expected_statuses (set[str]) – State that is expected See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferOperations#Status
timeout (Optional[Union[float, timedelta]]) – Time in which the operation must end in seconds. If not specified, defaults to 60 seconds.
- Return type
-
static
operations_contain_expected_statuses
(operations:List[Dict], expected_statuses:Union[Tuple[str], str])[source]¶ Checks whether the operation list has an operation with the expected status, then returns true If it encounters operations in FAILED or ABORTED state throw
airflow.exceptions.AirflowException
.- Parameters
operations (list[dict]) – (Required) List of transfer operations to check.
expected_statuses (set[str]) – (Required) status that is expected See: https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferOperations#Status
- Returns
If there is an operation with the expected state in the operation list, returns true,
- Raises
airflow.exceptions.AirflowException If it encounters operations with a state in the list,
- Return type
-