goumar
goumar

Reputation: 1

Create a read session with BigQuery Emulator Testcontainer with Apache Beam

I am using the BigQuery emulator (https://github.com/goccy/bigquery-emulator) for my integration tests in local machine.

I have a problem on my job that cannot create a read session, the job stuck indefinitely.

versions: Apache Beam: 2.61 BigQuery emulator: 0.6.6

Here the codes that I am try to make it works

My Apache Beam BigQuery reader

        return BigQueryIO.read(SchemaAndRecord::getRecord)
                .fromQuery("SELECT * FROM `test-project.test-dataset.test-table`")
                .usingStandardSql()
                .withTemplateCompatibility()
                .withMethod(BigQueryIO.TypedRead.Method.DIRECT_READ)
                .withCoder(AvroCoder.of(schema));

My beam pipeline options

        System.setProperty(
                "beamTestPipelineOptions",
                """
                          [
                            "--bigQueryEndpoint=%s",
                            "--credentialFactoryClass=org.apache.beam.sdk.extensions.gcp.auth.NoopCredentialFactory",
                            "--project=%s"
                          ]
                        """
                        .formatted(container.getEmulatorHttpEndpoint(), container.getProjectId()));

My container declaration

    @Container
    private static final BigQueryEmulatorContainer container = new BigQueryEmulatorContainer("ghcr.io/goccy/bigquery-emulator:0.6.6");

The query job is completed is the last and here the last relevant logs

BigQuery job GenericData{classInfo=[jobId, location, projectId], {jobId=beam_bq_job_QUERY_testpipelineaz027760209071426734a5120_30b6b6143b7d4f3c9211dcfff9d9fc33_3daddee8e66f44efa001335da49f491e, projectId=test-project}} completed in state DONE
Query job beam_bq_job_QUERY_testpipelineaz027760209071426734a5120_30b6b6143b7d4f3c9211dcfff9d9fc33_3daddee8e66f44efa001335da49f491e completed

Upvotes: 0

Views: 25

Answers (0)

Related Questions