CODEWITHSUNDEEP
google-bigquery
YABADABADOU
YABADABADOU

Reputation: 1328

BigQuery - Warning Message: TableXXX is highly fragmented

This morning, I looked in the Query History of the BigQuery browser tool and I saw that got the following error from a query that I run everyday:

Errors:
Warning: Table XX:XX.XX@1412077252527-1412682052526 is highly fragmented. 
Query performance may be impacted.

I've been running this query for months without any errors. Any clue on what I can do about this error/warning? I thought that all the hardware and indexes stuff was supposed to be handled on Google side and that we didn't need to worry about it.

If this can help the Google-BigQuery team, my job id was job_iHCt38RHGhT-FV0HR1kNZYAI508 .

Upvotes: 1

Views: 248

Answers (1)

Jordan Tigani
Jordan Tigani

Reputation: 26637

You see this message when the number of underlying chunks of data backing your tables is large compared to the overall table size.

In general, BigQuery does manage the file sizes in order to maximize query performance by periodically rewriting your table or moving it around. However, when you use a time range decorator (which it looks like you're using), bigquery needs to use the original table representation, since you're asking for the table state as of a particular time.

The warning is there to tell you that your queries are not going to be functioning at optimal performance. One way to avoid it is to copy your table, which will compact the table representation. You also could just ignore the issue; it doesn't mean your query will fail, just that it will be slow. The other way to avoid the issue is to run fewer, but larger imports to the table.

Upvotes: 1

Related Questions