CODEWITHSUNDEEP
postgresqldata-streamairbyte
DimasJackson
DimasJackson

Reputation: 1

Airbyte Postgres Kafka connection failed to run schema discovery

I'm trying to create a connection in Airbyte 0.61.0 (deployed in Docker containers in a Linux Server) using PostgreSQL (conector version 3.4.10) as source and Kafka (conector version 0.1.10) as destination. The settings can not be completed since in step 3-Select Streams an error occurs: Discovering schema failed - Failed to run schema discovery.

I already checked the connections with Postgres and Kafka and everything is working, also the user grants on Postgres Schema is correctely set. The Postgres database that I'm trying to connect is just a test one with only one small table and only the default schemas, I passed 'public' schema in postgres conector configs.

Any information on how to fix this will be very helpfully!

Thanks in advance!

enter image description here

The log lines are:

"logLines": [
                "2024-06-04 19:02:00  > Docker volume job log path: /tmp/workspace/61b2f90b-9bb0-4a63-be1f-4b0b5a1c59aa/0/logs.log",
                "2024-06-04 19:02:00  > Executing worker wrapper. Airbyte version: 0.61.0",
                "2024-06-04 19:02:00  > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'",
                "2024-06-04 19:02:00  > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'",
                "2024-06-04 19:02:00  > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'",
                "2024-06-04 19:02:00  > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'",
                "2024-06-04 19:02:00  > Checking if airbyte/source-postgres:3.4.10 exists...",
                "2024-06-04 19:02:00  > airbyte/source-postgres:3.4.10 was found locally.",
                "2024-06-04 19:02:00  > Creating docker container = source-postgres-discover-61b2f90b-9bb0-4a63-be1f-4b0b5a1c59aa-0-gejuz with resources io.airbyte.config.ResourceRequirements@66677c2e[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null",
                "2024-06-04 19:02:00  > Preparing command: docker run --rm --init -i -w /data/61b2f90b-9bb0-4a63-be1f-4b0b5a1c59aa/0 --log-driver none --name source-postgres-discover-61b2f90b-9bb0-4a63-be1f-4b0b5a1c59aa-0-gejuz --network host -v airbyte_workspace:/data -v oss_local_root:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-postgres:3.4.10 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e WORKER_ENVIRONMENT=DOCKER -e AIRBYTE_ROLE=dev -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.61.0 -e WORKER_JOB_ID=61b2f90b-9bb0-4a63-be1f-4b0b5a1c59aa airbyte/source-postgres:3.4.10 discover --config source_config.json",
                "2024-06-04 19:02:00  > Reading messages from protocol version 0.2.0",
                "2024-06-04 19:02:01  > INFO main i.a.i.s.p.PostgresSource(main):667 starting source: class io.airbyte.integrations.source.postgres.PostgresSource",
                "2024-06-04 19:02:01  > INFO main i.a.c.i.b.IntegrationCliParser$Companion(parseOptions):144 integration args: {discover=null, config=source_config.json}",
                "2024-06-04 19:02:01  > INFO main i.a.c.i.b.IntegrationRunner(runInternal):124 Running integration: io.airbyte.cdk.integrations.base.ssh.SshWrappedSource",
                "2024-06-04 19:02:01  > INFO main i.a.c.i.b.IntegrationRunner(runInternal):125 Command: DISCOVER",
                "2024-06-04 19:02:01  > INFO main i.a.c.i.b.IntegrationRunner(runInternal):126 Integration config: IntegrationConfig{command=DISCOVER, configPath='source_config.json', catalogPath='null', statePath='null'}",
                "2024-06-04 19:02:01  > WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword groups - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword",
                "2024-06-04 19:02:01  > WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword",
                "2024-06-04 19:02:01  > WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword group - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword",
                "2024-06-04 19:02:01  > WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword",
                "2024-06-04 19:02:01  > WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword always_show - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword",
                "2024-06-04 19:02:01  > WARN main c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword display_type - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword",
                "2024-06-04 19:02:01  > INFO main i.a.c.i.b.s.SshTunnel$Companion(getInstance):423 Starting connection with method: NO_TUNNEL",
                "2024-06-04 19:02:01  > INFO main i.a.i.s.p.PostgresSource(toSslJdbcParamInternal):888 DISABLED toSslJdbcParam disable",
                "2024-06-04 19:02:01  > INFO main c.z.h.HikariDataSource(<init>):79 HikariPool-1 - Starting...",
                "2024-06-04 19:02:01  > INFO main c.z.h.HikariDataSource(<init>):81 HikariPool-1 - Start completed.",
                "2024-06-04 19:02:02  > INFO main i.a.i.s.p.PostgresUtils(isCdc):70 using CDC: false",
                "2024-06-04 19:02:02  > INFO main i.a.c.i.s.j.AbstractJdbcSource(discoverInternal):291 Internal schemas to exclude: [catalog_history, information_schema, pg_catalog, pg_internal]",
                "2024-06-04 19:02:02  > INFO main i.a.c.d.j.s.AdaptiveStreamingQueryConfig(initialize):24 Set initial fetch size: 10 rows",
                "2024-06-04 19:02:02  > INFO main i.a.c.i.s.j.AbstractJdbcSource(discoverPrimaryKeys):428 Discover primary keys for tables: []",
                "2024-06-04 19:02:02  > INFO main c.z.h.HikariDataSource(close):349 HikariPool-1 - Shutdown initiated...",
                "2024-06-04 19:02:02  > INFO main c.z.h.HikariDataSource(close):351 HikariPool-1 - Shutdown completed.",
                "2024-06-04 19:02:02  > INFO main i.a.i.s.p.PostgresSource(toSslJdbcParamInternal):888 DISABLED toSslJdbcParam disable",
                "2024-06-04 19:02:02  > INFO main c.z.h.HikariDataSource(<init>):79 HikariPool-2 - Starting...",
                "2024-06-04 19:02:02  > INFO main c.z.h.HikariDataSource(<init>):81 HikariPool-2 - Start completed.",
                "2024-06-04 19:02:02  > INFO main i.a.i.s.p.PostgresUtils(isCdc):70 using CDC: false",
                "2024-06-04 19:02:02  > INFO main i.a.i.s.p.PostgresUtils(isCdc):70 using CDC: false",
                "2024-06-04 19:02:02  > INFO main i.a.i.s.p.PostgresUtils(isXmin):190 using Xmin: false",
                "2024-06-04 19:02:02  > INFO main i.a.c.i.b.IntegrationRunner(runInternal):268 Completed integration: io.airbyte.cdk.integrations.base.ssh.SshWrappedSource",
                "2024-06-04 19:02:02  > INFO main i.a.i.s.p.PostgresSource(main):669 completed source: class io.airbyte.integrations.source.postgres.PostgresSource"
            ]

I tryed Detect Changes with Xmin System Column and Scan Changes with User Defined Cursor in postgres conector but nothing was changed. Also, connecting to other Postgres database or schemas results in the same error.

Upvotes: 0

Views: 157

Answers (1)

DimasJackson
DimasJackson

Reputation: 1

I found a solution: just use the previous version of Airbyte. To do that, go to the Airbyte cloned repository path and run:

git checkout -b 0.62.0

Then run the script

bash run-ab-platform -r

The -r parameter remove the current image and containers and pull again. Finally run the script again

bash run-ab-platform -b

If you want to see the logs, replace -b to -x to run in debug mode. After that everthing worked.

Upvotes: 0

Related Questions