CODEWITHSUNDEEP
amazon-web-servicesamazon-s3google-bigquerycommand-line-interface
r123
r123

Reputation: 618

How to schedule BigQuery DataTransfer Service using bq command

I am trying to create a Data Transfer service using BigQuery. I used bq command to create the DTS,

  1. I am able to create DTS successfully
  2. I need to specify custom time for scheduling using the bq command

Is it possible to schedule custom time while creating the Data Transfer service. Refer sample bq command

bq mk --transfer_config \
--project_id='My project' \
--target_dataset='My Dataset' \
--display_name='test_bqdts' \
--params='{"data_path":<data_path>,
"destination_table_name_template":<destination_table_name>,
"file_format":<>,
"ignore_unknown_values":"true",
"access_key_id": "access_key_id",
"secret_access_key": "secret_access_key"
}' \
--data_source=data_source_id

NOTE: When you create an Amazon S3 transfer using the command-line tool, the transfer configuration is set up using the default value for Schedule (every 24 hours).

Upvotes: 2

Views: 3503

Answers (1)

rmesteves
rmesteves

Reputation: 4085

You can use the flag --schedule as you can see here

Option 2: Use the bq mk command.

Scheduled queries are a kind of transfer. To schedule a query, you can use the BigQuery Data Transfer Service CLI to make a transfer configuration.

Queries must be in StandardSQL dialect to be scheduled.

Enter the bq mk command and supply the transfer creation flag --transfer_config. The following flags are also required:

  • --data_source
  • --target_dataset (Optional for DDL/DML queries.)
  • --display_name
  • --params

Optional flags:

  • --project_id is your project ID. If --project_id isn't specified, the default project is used.

  • --schedule is how often you want the query to run. If --schedule isn't specified, the default is 'every 24 hours' based on creation time.

  • For DDL/DML queries, you can also supply the --location flag to specify a particular region for processing. If --location isn't specified, the global Google Cloud location is used.

  • --service_account_name is for authenticating your scheduled query with a service account instead of your individual user account. Note: Using service accounts with scheduled queries is in beta.


    bq mk \
    --transfer_config \
    --project_id=project_id \
    --target_dataset=dataset \
    --display_name=name \
    --params='parameters' \
    --data_source=data_source

If you want to set a 24 hours schedule, for example, you should use --schedule='every 24 hours' You can find the complete reference for the time syntax here

I hope it helps

Upvotes: 3

Related Questions