ali izadi
ali izadi

Reputation: 549

Preventing Duplicate Inserts in BigQuery Community Connector with Concurrent Requests in Looker Studio

I'm developing a Google BigQuery community connector for Looker Studio that retrieves data for a specified date range (start_date and end_date). The connector logic is set to:

1- Check if data for the date range exists in a BigQuery table. 2- If it doesn’t exist, insert the data. 3- If it exists, simply fetch the existing data.

Problem: When a Looker Studio dashboard with multiple charts loads, the connector gets triggered by several requests simultaneously. Since the data initially doesn’t exist, each request tries to insert the data, resulting in duplicate rows.

I attempted solutions like implementing user-specific locks and adding a small delay, but these don't work because the requests are sent at the same millisecond. As a result, all insert operations are initiated before any lock or delay can take effect, leading to multiple identical rows.

multiple concurrent requests sent from a Looker Studio dashboard1

Question: Is there a way to ensure that only one insert operation occurs per date range, even when multiple requests are triggered at the exact same time? Alternatively, can I detect an insert in progress and skip additional data retrieval attempts for that range?

Upvotes: 0

Views: 34

Answers (0)

Related Questions