CODEWITHSUNDEEP
pythonbashvariablesmatillion
FreeZey
FreeZey

Reputation: 2812

Matillion Bash Script - How to Set an Environment Variable Value?

If I create an environment variable in a Matillion called "var_foo" (with a Type of "Text", a Behaviour of "Shared" and a default of "notset") I'm able to set and get the value of this variable in a Python script but not a Bash script. In Bash I can only get and not set it.

Python Example Code:

# In Python Script 1.  Set the environment variable 
# -------------------------------------------------
print('Before Set.  ' + var_foo)
context.updateVariable('var_foo', 'bar')
print('After Set.  ' + var_foo)

# In Python Script 2.  Get the environment variable 
# -------------------------------------------------
print('In Next Script.  ' + var_foo)

Python Task Output:

Before Set.  notset
After Set.  bar
In Next Script.  bar

But when I try to do something similar using two Bash scripts the variable value isn't retained from Script 1 into Script 2. It just outputs the default as if the variable hasn't had a value set.

Bash Example Code:

# In Bash Script 1.  Set the environment variable
# -----------------------------------------------
echo "Before Set.  "$var_foo
var_foo="bar"
echo "After Set.  "$var_foo

# In Bash Script 2.  Get the environment variable
# -----------------------------------------------
echo "In Next Script.  "$var_foo

Bash Task Output:

Before Set.  notset
After Set.  bar
In Next Script.  notset

Does Bash have something equivalent to "context.updateVariable" that allows me to explicitly identifies the variable I'm setting is part of the Matillion environment? Because it seems to read it automatically but when setting the value it re-creates it as a local variable.

Upvotes: 0

Views: 941

Answers (1)

53epo
53epo

Reputation: 909

I agree, the Bash Script component has no equivalent of the Python Script context.updateVariable.

I think the way a Bash Script works is:

  1. all Matillion ETL variables get initialized as a Linux Bash environment variable with the same name
  2. the initial value is a string literal containing whatever the job sees as the current value of the Matillion variable - falling back to the default value, if any

So it's a one-way trip. The Bash Script is working with Linux Bash environment variables, and they all just disappear when the script finishes.

According to Matillion's variable lifecycle documentation, updates to variables never persist beyond one job execution anyway.

There is a possible way around all this, by updating the default value. You have to make a REST API call, but curl is available to do that.

In my Matillion ETL instance, I have a project-wide Text environment variable named today_yyyymmdd

Then, I have a job scheduled to run soon after midnight. It just runs a Bash Script containing this curl command:

curl -u myuser:mypass -s -X POST -H "Content-Type: application/json" "http://localhost:8080/rest/v1/group/name/${project_group_name// /%20}/project/name/${project_name// /%20}/environment/name/${environment_name// /%20}/variable/name/today_yyyymmdd/set/value/$(date '+%Y%m%d')"

All the other jobs that run later in the day can use ${today_yyyymmdd}, and it will always contain today's date as a string formatted in a standard way.

Upvotes: 1

Related Questions