CODEWITHSUNDEEP
databaseamazon-web-servicesopen-sourceaws-datasyncairbyte
Manjul Tripathi
Manjul Tripathi

Reputation: 1

Airbyte Missing Columns in tables

I am considering Airbyte as a replacement for singer-io because of its user-friendly UI and code reduction features. However, facing an issue where Airbyte is unable to detect certain columns in our source (Atlas hosted on AWS), which is a significant concern for me.

For instance, Consider a table users with columns user_id, name, location, phone_num. While the sync is performed, column location is missed, while it is present in mongo db. In Airbyte UI also column location is not being shown. I tried to figure out this issue via Octavia CLI, but could not find any way that columns can be defined. Just wanted to mention that this column has ~70% nan values. But going with the fact that even for 30% of data the column should not be missed.

I tried to re-sync data but no results. I tried to install Octavia CLI to try to define columns for the table, but no such option was available.

Upvotes: 0

Views: 556

Answers (1)

Andy Twiss
Andy Twiss

Reputation: 1

I've had issues with custom datatypes in Airbyte before where fields are silently left out of the schema discovery process.

Try sticking to the intrinsic types.

Upvotes: 0

Related Questions