CODEWITHSUNDEEP
linuxgoogle-bigquery
supreet padhi
supreet padhi

Reputation: 69

BigQuery: Does bq load command support loading from named pipe as a source?

I am trying to load data to Google bigquery using bq load from a named pipe.

Console Window1:

 $ mkfifo /usr/pipe1
 $ cat /dev1/item.dat > /usr/pipe1

Console Window2:

 $ bq load --source_format=CSV  projectid:dataset.itemtbl /usr/pipe1 field1:integer,field2:integer

Got the following error:

BigQuery error in load operation: Source path is not a file: /usr/pipe1

Upvotes: 6

Views: 1097

Answers (2)

Jorge Caceres
Jorge Caceres

Reputation: 1

The bq load command it doesn't support pipe files. this is the error when you change the code to bypass the pipe file validation.

== Error trace ==
Traceback (most recent call last):
  File "/usr/local/share/google/google-cloud-sdk/platform/bq/bq.py", line 1001, in RunSafely
    return_value = self.RunWithArgs(*args, **kwds)
  File "/usr/local/share/google/google-cloud-sdk/platform/bq/bq.py", line 1355, in RunWithArgs
    job = client.Load(table_reference, source, schema=schema, **opts)
  File "/usr/local/share/google/google-cloud-sdk/platform/bq/bigquery_client.py", line 3504, in Load
    upload_file=upload_file, **kwds)
  File "/usr/local/share/google/google-cloud-sdk/platform/bq/bigquery_client.py", line 2924, in ExecuteJob
    location=location)
  File "/usr/local/share/google/google-cloud-sdk/platform/bq/bigquery_client.py", line 2901, in RunJobSynchronously
    location=location)
  File "/usr/local/share/google/google-cloud-sdk/platform/bq/bigquery_client.py", line 2755, in StartJob
    resumable=resumable)
  File "/usr/local/share/google/google-cloud-sdk/platform/bq/third_party/oauth2client_4_0/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/share/google/google-cloud-sdk/platform/bq/third_party/googleapiclient/http.py", line 562, in __init__
    resumable=resumable)
  File "/usr/local/share/google/google-cloud-sdk/platform/bq/third_party/oauth2client_4_0/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/share/google/google-cloud-sdk/platform/bq/third_party/googleapiclient/http.py", line 439, in __init__
    self._fd.seek(0, os.SEEK_END)
IOError: [Errno 29] Illegal seek
========================================

Unexpected exception in load operation: You have encountered a bug in the
BigQuery CLI. Please file a bug report in our
public issue tracker:
https://issuetracker.google.com/issues/new?component=187149&template=0
Please include a brief description of the steps that led to this issue, as well
as any rows that can be made public from the following information:
Hora : 2018-10-05 10:04:02

Upvotes: -1

Michael Sheldon
Michael Sheldon

Reputation: 2057

The BigQuery client bq.py does not support named pipes. It explicitly requires files:

https://code.google.com/p/google-bigquery-tools/source/browse/bq/bigquery_client.py?r=30df4638ff2ddb01d3f495af5c131ed3c2cfbd04#617

Allowing named pipes is a good feature suggestion. You can request it here:

https://code.google.com/p/google-bigquery/issues/list

It looks like you could tweak your copy of bigquery_client.py pretty easily to make this work as well. Good luck!

Upvotes: 4

Related Questions