CODEWITHSUNDEEP
google-cloud-data-fusion
Adolfo Garza
Adolfo Garza

Reputation: 3029

Cloud Data Fusion: Datastore Inconsistent Data Type Issues

I am working with a Google Datastore data source and within the kind (table) there's a field which holds a number which in some records show as an INTEGER type and in others as a FLOAT type. When running that source Data Fusion throws an error saying that the field is not of type LONG. I tried changing the type to DOUBLE and STRING and still fails.

ERROR: Failed to execute pipeline stage 'Datastore' with the error: Field 'amount' is not of expected type 'LONG'. Please review your pipeline configuration and check the system logs for more details.

Any ideas on how to cast the field as DOUBLE or just take the entire response as JSON/String and process it on a Javascript Transform step?

I am trying to avoid running an update in Datastore to update the data type to make them consistent.

Upvotes: 0

Views: 205

Answers (1)

Adolfo Garza
Adolfo Garza

Reputation: 3029

I figured it out. I had to select Union type and then add all possible data types that the field can hold.

Upvotes: 1

Related Questions