CODEWITHSUNDEEP
google-bigquery
shantanuo
shantanuo

Reputation: 32346

new lines in big query strings

What exactly does "new-lines in strings" mean?

https://developers.google.com/bigquery/docs/quota-policy#import

How to check and remove it?

Import Jobs: Daily limit: 1,000 import jobs per table per day (including failures), 10,000 import jobs per project per day (including failures)
Maximum size per import job: 1TB uncompressed
Maximum number of files per import job: 500

Upvotes: 3

Views: 9384

Answers (1)

jspcal
jspcal

Reputation: 51914

newlines in strings refers to embedded newline characters within CSV field values. this makes it difficult to parallelize imports.

e.g.: 2 lines, the first has an embedded newline:

1,2,"this is my
string",4,5
1,2,"another string",4,5

The BigQuery importer has the "--noallow_quoted_newlines" option, which signifies the file doesn't contain any embedded newlines.

# --noallow_quoted_newlines:
#   Do not allow quoted newlines in the data.  This allows BigQuery to parallelize the load.

https://developers.google.com/bigquery/articles/ingestioncookbook#atomic

Individual JSON and CSV files without newlines in string fields may be as large as 100GB, and CSV files that contain newlines within strings must be 4GB or less.

https://developers.google.com/bigquery/articles/ingestioncookbook

Upvotes: 6

Related Questions