CODEWITHSUNDEEP
pythonlinux
Omroth
Omroth

Reputation: 1119

Python and environment variables not working

In a Docker container on Ubuntu, I'm running a .sh file which calls python3 my_script_file.py. In that file, I do:

commands = []
commands.append("export OMP_NUM_THREADS=4")
commands.append("echo $OMP_NUM_THREADS")

I then execute the commands array with:

def execute_commands_list(commands, no_bash_script = False):
    for command in commands:
        if not no_bash_script and gc.output_to_bash_script:
            write_command_to_bash_script(command)
        else:
            print("     [  " + command + "  ]")

            ret = subprocess.call(command, shell=True, executable="/bin/bash")

            assert ret==0 or ret==2, "bash return code error: " + str(ret) + " " + command

Here is the output:

     [  export OMP_NUM_THREADS=4  ]
     [  echo $OMP_NUM_THREADS  ]

(that's a blank line). I try running echo $OMP_NUM_THREADS in the terminal after execution myself, and get nothing. If I run those both commands manually however the env var is set.

(I have also tried os.environ, and that has failed as well)

What am I missing?

(For reference, the .sh file is:)

#!/bin/bash -p

python3 my_script_file.py

Upvotes: 1

Views: 4912

Answers (1)

Avihoo Mamka
Avihoo Mamka

Reputation: 4786

That's probably because the scope of export is limited only to the current shell it's being running on. Meaning that only the shell the subprocess ran from, this environment variable will be visible to.

If you want to make this environment variable globally exposed, and given you're running from a linux machine of some sort, I'd suggest writing it to /etc/environment in the format of KEY=VALUE.

For example in your case, instead of

export OMP_NUM_THREADS=4

Replace it with:

echo OMP_NUM_THREADS=4 >> /etc/environment

Note! Since environment is located under /etc there's a good chance its permissions are set to allow write only to a sudo user, so make sure you include that in your subprocess call.

You can use something like the following:

echo 'OMP_NUM_THREADS=4' | sudo tee -a /etc/environment > /dev/null

tee prints the text to stdout, too. In order to mute it so it behaves more similar to shell appending (>>), route the stdout to /dev/null.

Upvotes: 3

Related Questions