CODEWITHSUNDEEP
google-bigquerydata-partitioning
DrPib81
DrPib81

Reputation: 63

Can I leverage BigQuery (BQ) partition via a join?

I am a Tableau designer, and we are building some views that get filtered by category a lot. Because of this, we tried to create a category_id that would serve as partition. The problem seems to be that if I filter data category only, the partition doesn't get used and the total table GB and cost gets hit.

Our team is trying to see if this could be minimized by using a nested query as follows:

SELECT *
FROM table a
INNER JOIN (

SELECT DISTINCT category_id, category
FROM table

) b
ON a.category_id = b.category_id

WHERE b.category = 'Category A'

The idea is that we could show the user b.category, they select it in Tableau and then the inner join would kick off the partition and limit the bytes returned. When I try this in the BQ interface, the estimated returned size comes back the same.

Upvotes: 0

Views: 1074

Answers (2)

Steve Martin
Steve Martin

Reputation: 109

When you say that your attempts to filter only by category, the partition isn't used, have you actually tested querying the table from the console to test whether the partition is being used or not. If it isn't then you need to look at the partition, but if it is, then you would need to take another look at your Tableau query.

VizQL (Viz query language) is Tableau's sql parser that converts your Tableau viz into SQL for execution, so whilst you cannot really modify the outgoing SQL, you can at least capture it and test which enables you to identify poor performing calculations and/or vizzes, as well as optimise the backend for the queries that Tableau will send. I've written an article about this here: https://datawonders.atlassian.net/wiki/spaces/TABLEAU/pages/1290600449/Let+s+Talk+Errors+Tuning+6+minute+read

The thing about Tableau is that it treats the source as a derived table, with all filters being placed at the upper-level of the query immediately before the stream, so your query:

Select *
From table a
Join (
    Select Distinct Category_ID, Category
    From table
)b On a.category_id = b.category_id

Where b.category = 'Category A'

Will actually look like this (assuming you just select everything):

Select a1.*
From (
    Select *
    From table a
    Join (
        Select Distinct Category_ID, Category
        From table
    )b On a.category_id = b.category_id
)a1

Where a1.category = <your selected category>

So you can see from here that being two-levels deep, your Category table just won't be hit, instead everything shall be read into the spool, the join taking place in tempdb, and only the complete set is filtered immediately before streaming to Tableau. Bad, underperforming sql it most certainly is. And this is where the relational method of v2020.2 comes into play, as this has been designed to treat each table as a separate exclusive entity, joins are only made at execution time, so you could build a view that uses data from table a where you are using table b to provide the filtering.

As an alternative, and my preferred overall method is to switch entirely to Custom SQL, utilising this with parameters, as this will enable you to craft and test your own sql to create your own high-performance, low-loading query, but as parameters are parsed before the query is executed, you can place the filtering deep down in the query without the need for a secondary look-up table or filtered derived statement - a select distinct as you are currently using it is still going to produce a large plan, as unless the category column is indexed, the engine shall still need to read every record from the table.

So using parameters, your new query will look something like:

Select a1.*

From (
    Select *
    From table a
    Join lookup_table b On On a.category_id = b.category_id
    And b.category = <parameters.pCategory>
)a1

(I've placed the filter condition directly onto the join as this can improve performance in some circumstances, though this actually shouldn't make much difference) And when used in conjunction with the Set parameter action, you can now use parameters as in/out updateable variables which shall update as the user interacts directly with the viz, instead of the user needing to manually update as they go. If you haven't used these before, I wrote an article about it here: https://community.tableau.com/s/news/a0A4T00000313S0UAI/psst-have-you-had-a-go-with-variables-in-tableau-yet

Steve

Upvotes: 0

Tom E
Tom E

Reputation: 421

You'll need to filter on the partitioned field before you make the inner join.

I haven't used tableau before so don't know if this is possible but just an idea. You could create a parameter which is set by the chosen category in tableau, which could be referenced in the where statement of the partitioned table?

SELECT *
FROM table a
INNER JOIN (

SELECT DISTINCT category_id, category
FROM table
Where category = @chosen_category
) b
ON a.category_id = b.category_id;

Upvotes: 1

Related Questions