CODEWITHSUNDEEP
google-cloud-platformgoogle-bigquerydataformgoogle-bigquery-dataform
Pierre Honoré
Pierre Honoré

Reputation: 21

Error "Not found: Dataset *****:dataform was not found in location US." while repository located in EU

I am currently working with a SQL workflow using Dataform. Within this workflow, I have declared certain tables and established dependencies among them. Everything is functioning as expected in this regard. However, I encountered an issue when attempting to create a table that depends on other tables which have not been declared explicitly.

The problem arises from the fact that the query doesn't compile due to the error message:

Not found: Dataset *****:dataform was not found in location US

Even though my repository is configured in the EU.

For illustration, here's an example:

simple_table

config {
  type: "table",
  description: ""
  }
SELECT 1 AS number
UNION ALL
SELECT 2 AS number

dependent_table

config {
  type: "table",
  description: ""
}
SELECT * FROM ${ref("simple_table")}

I tried to change the repository location, it didn't work.

Upvotes: 2

Views: 2079

Answers (4)

matox
matox

Reputation: 987

I had the same problem when going through Dataform tutorial Create and execute a SQL workflow in Dataform.

In the docs, it is mentioned that this is expected behaviour:

After defining the table type, Dataform throws a query validation 
error because quickstart-source does not exist in BigQuery yet. 
This error is resolved when you execute the SQL workflow later 
in this tutorial.

The issue was resolved once I executed all actions in the workspace.

Upvotes: 0

Achilles M
Achilles M

Reputation: 21

If you are running with the Dataform CLI then it's likely that your credentials are locked to that region. When you initialise the credentials with dataform init-creds bigquery, you are asked to choose between

[1] US (default)
[2] EU
[3] other

Chances are that you selected the default option.

Upvotes: 0

Matt Bennett
Matt Bennett

Reputation: 11

You can't combine data from multiple regions, so one of the tables is in the US, and one is in the EU, or they're all in the US and your project is set to EU. As mentioned above, changing the location in dataform.json will fix this.

If you're dealing with data across different regions, the way I've resolved this in the past is to set up a daily data transfer that moves the data I need to the primary region, then use that as your source.

Upvotes: 1

gibbon.cz
gibbon.cz

Reputation: 120

Check the dataform.json in your project root. In my case there was

  "defaultLocation": "EU"

that was causing the same issue. In your case, "US" is probably set there.

Upvotes: 0

Related Questions