CODEWITHSUNDEEP
pythonpython-2.7google-cloud-dataflowapache-beamapache-beam-io
MRvaino
MRvaino

Reputation: 1

Module object has no attribute BigqueryV2 - Local Apache Beam

I am trying to run a pipeline locally (Sierra) with Apache Beam using beam provided I/O APIs for Google BigQuery.

I settled up my environment using Virtualenv as suggested by Beam Python quickstart and I can run the wordcount.py example. I can also run properly a custom pipeline with beam.Create and beam.ParDo.

But I cannot run a pipeline with BigQuery I/O. Any idea on what I am doing wrong?

The python script is the following.

import apache_beam as beam
from apache_beam.utils.pipeline_options import PipelineOptions
from apache_beam.io import WriteToText


class MyDoFn(beam.DoFn):
  def process(self, element):
    return element


def run():
  opts = {
    'project': 'gc-project-name'
  }
  p = beam.Pipeline(options=PipelineOptions(**opts))

  input_query = "SELECT name FROM `gc-project-name.dataset_name.table_name`"

  (p
   | beam.io.Read(beam.io.BigQuerySource(query=input_query))
   | beam.ParDo(MyDoFn())
   | beam.io.WriteToText('output.txt')
  )

  result = p.run()
  result.wait_until_finish()

if __name__ == '__main__':
  run()

When I run it I get the following error.

WARNING:root:Task failed: Traceback (most recent call last):
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 300, in __call__
result = evaluator.finish_bundle()
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 208, in finish_bundle
with self._source.reader() as reader:
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py", line 590, in __enter__
self.client = BigQueryWrapper(client=self.test_bigquery_client)
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py", line 682, in __init__
self.client = client or bigquery.BigqueryV2(
AttributeError: 'module' object has no attribute 'BigqueryV2'
Traceback (most recent call last):
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 300, in __call__
result = evaluator.finish_bundle()
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 208, in finish_bundle
with self._source.reader() as reader:
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py", line 590, in __enter__
self.client = BigQueryWrapper(client=self.test_bigquery_client)
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py", line 682, in __init__
self.client = client or bigquery.BigqueryV2(
AttributeError: 'module' object has no attribute 'BigqueryV2'
WARNING:root:A task failed with exception.
 'module' object has no attribute 'BigqueryV2'
Traceback (most recent call last):
  File "frombigquery.py", line 54, in <module>
run()
  File "frombigquery.py", line 51, in run
result.wait_until_finish()
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py", line 157, in wait_until_finish
self._executor.await_completion()
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 335, in await_completion
self._executor.await_completion()
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/runners/direct/executor.py", line 300, in __call__
result = evaluator.finish_bundle()
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/runners/direct/transform_evaluator.py", line 208, in finish_bundle
with self._source.reader() as reader:
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py", line 590, in __enter__
self.client = BigQueryWrapper(client=self.test_bigquery_client)
  File "/Users/localuser/Virtualenvs/abeam/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py", line 682, in __init__
self.client = client or bigquery.BigqueryV2(
AttributeError: 'module' object has no attribute 'BigqueryV2'

Upvotes: 0

Views: 1864

Answers (1)

chamikara
chamikara

Reputation: 2024

When installing Apache Beam Python SDK, you have to add an additional option to use Google Cloud Platform related dependencies.

pip install dist/apache-beam-*.tar.gz[gcp]

Upvotes: 2

Related Questions