CODEWITHSUNDEEP
jsonpostgresqlgoogle-bigquery
Nicoowr
Nicoowr

Reputation: 809

Load JSON in BigQuery / JSON parsing error in row starting at position ... : Parser terminated before end of string

I'm trying to load a 350MB JSON file in BigQuery using Airflow GoogleCloudStorageToBigQueryOperator. The job always stop at some position N (N never changes), with this error: Error while reading data, error message: JSON parsing error in row starting at position 170468557: Parser terminated before end of string

I've searched for this line in the file, which goes like this:

{"active": true,
"currency": "USD",
"dangerous": "all",
"filing_reference": null,
"is_freight": false,
"max": NaN,
"min": 15.0,
"rate": 15.0,
"rate_unit": "teu",
"rates": [],
"rates_fixed": null,
"shipowner_id": "12",
"thresholds": [],
"transit_time": null,
"updated_at": 1566912641.0,
"validity_end": 1556582400.0,
"validity_start": 1554076800.0,
"via": "UNKNOWN"}

The BigQuery schema is generated from Postgres type values. This error is not clear at all, I'd appreciate some help!

Upvotes: 0

Views: 3361

Answers (2)

ARtem
ARtem

Reputation: 36

Try to change line separator in your input file. I had this problem when I had Windows 'CR' instead of Linux 'LF'.

Also I had similar problem when there was an extra line break.

Upvotes: 0

Antoine Sauvage
Antoine Sauvage

Reputation: 68

We got the same error a few days ago, it comes from the NaN in your "max"

This type of JSON can be parsed with python for example, but when it comes to BigQuery, it throws an arror

=> Be sure to replace the NaNs to "null", it should work

Hope it helps !

Upvotes: 2

Related Questions