CODEWITHSUNDEEP
google-bigquerycommand-line-interface
mkantepe
mkantepe

Reputation: 55

BigQuery job filter by label

Is there any way to filter big query jobs with label(s)?

I created a job(query) with label task_id:my_task

bq query --use_legacy_sql=false --label "task_id:my_task"  --project my-project 'SELECT * FROM `dataset.mytable`'

Tried those to get all jobs with the label but none of them worked:

bq ls -j --filter 'configuration.labels(task_id):my_task'
bq ls -j --filter 'configuration.labels.task_id:my_task'
bq ls -j --filter  configuration.labels(task_id):my_task
bq ls -j --filter  labels.task_id:my_task

Upvotes: 1

Views: 1820

Answers (1)

Héctor Neri
Héctor Neri

Reputation: 1452

According to the documentation for "bq ls" [1], --filter lists datasets that match the filter expression. But for listing jobs, the documentation [2] mentions that the allowed flags are three: -j, -a and -n.

So, there isn't a way to filter the listing of jobs, at least through the Bigquery command tool. But as a workaround, you can use the following command to get all the Jobs that are labeled as "task_id:my_task"

for i in $(bq ls -j | awk 'NR>2 {print $1}'); do echo "$(bq show -j $i) $i" | awk '/task_id:my_task/ && /SUCCESS/ {print $(NF)}'; done

This command may take some time, though; so consider adding the -n flag like this:

for i in $(bq ls -j -n 10 | awk 'NR>2 {print $1}'); do echo "$(bq show -j $i) $i" | awk '/task_id:my_task/ && /SUCCESS/ {print $(NF)}'; done

You may also submit a feature request to the BigQuery's Issue Tracker [3].

[1] https://cloud.google.com/bigquery/docs/reference/bq-cli-reference#bq_ls

[2] https://cloud.google.com/bigquery/docs/managing-jobs#listing_jobs

[3] https://issuetracker.google.com/issues/new?component=187149&template=1100108

Upvotes: 2

Related Questions