CODEWITHSUNDEEP
google-cloud-platformapache-beamdataflow
Quack
Quack

Reputation: 740

How to solve BeamDeprecationWarning in Apache-Beam

First, Code for storing data in google cloud platform bigQuery tables after generating simple data. Imported Apache-Beam library and used it. Runner used Google Cloud Platform Dataflow.

Here code.

from apache_beam.options.pipeline_options import PipelineOptions
import apache_beam as beam

pipeline_options = PipelineOptions(
     project='project-id',
     runner='runner',
     temp_location='bucket-location'
)
def pardo_dofn_methods(test=None):
   import apache_beam as beam 
 class testFunction(beam.DoFn):
    def process(self, element):
        result = element.split(',')

        testing = {'test_column': result[0], 'test_column2': result[1], 'test_column3': result[2],
                       'test_column4': result[3]}
        return [testing]

    def finish(self):
        print('finish')

with beam.Pipeline(options=pipeline_options) as pipeline:
  results = (
      pipeline
      | 'Generating data' >> beam.Create([
          'test1,test2,test3,test4'
          'test5,test6,test7,test8'
       ])
      | beam.ParDo(testFunction())
      | beam.io.WriteToBigQuery(
            'project-id:bigQuery-dataset.table-name',
            schema='test_column:STRING, test_column2:STRING, test_column3:STRING, test_column4:STRING',
            create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
            write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE
      )
  )

pardo_dofn_methods()

It works well when run it. However, there are two warnings:

BeamDeprecationWarning: options is deprecated since First stable release. References to <pipeline>.options will not be supported
 experiments = p.options.view_as(DebugOptions).experiments or []

BeamDeprecationWarning: BigQuerySink is deprecated since 2.11.0. Use WriteToBigQuery instead.
kms_key=self.kms_key))

I don't know why there is a warning. Thank you.

Upvotes: 2

Views: 5566

Answers (1)

Ricco D
Ricco D

Reputation: 7287

The deprecation warning "BigQuerySink is deprecated since 2.11.0. Use WriteToBigQuery instead." is no longer displayed when I tested WriteToBigQuery() using apache-beam==2.31.0 and runners DataflowRunner and DirectRunner. Thus this should not be displayed when using version 2.31.0 onwards.

The deprecation warning "options is deprecated since First stable release. References to .options will not be supported" is still shown as long as options is defined (see github reference) to provide warning on users.

Upvotes: 1

Related Questions