CODEWITHSUNDEEP
pythonmultithreadingenvironment-variables
proofit404
proofit404

Reputation: 386

What is a safe way to "bound" environment variable in python?

Suppose we have a library A witch behavior depends on value of some environment variable. A_CONFIG_PATH actually. Some of my tasks use this library with different A_CONFIG_PATH for each task. I do it in a way of

import os
import A

def task(**kw):
    os.environ['A_CONFIG_PATH'] = '/home/me/current/task/config/path'
    A.do_some_stuff(kw)

This fine until all tasks process synchronously. But now I need concurrency in this tasks processing.

So how I can guarantee that each task will not corrupt another with its own A_CONFIG_PATH if I run each task in separate thread/process or something like this.

Upvotes: 2

Views: 2511

Answers (1)

shx2
shx2

Reputation: 64368

There are several ways to tackle the problem:

  1. run each task in a subprocess, instead all in one process in different threads
  2. change the task to take the A_CONFIG_PATH value as a parameter, instead of reading it from environment (reading from env is just as bad as relying on global variables...)
  3. instead of assigning a string to os.environ[A_CONFIG_VALUE], use a threading.local object, which allows each thread to have its own value. You'd need to slightly change the part which reads the value, though.

An example using threading.local:

#one time init
os.environ['A_CONFIG_PATH'] = threading.local()

# set the value    
def task(**kw):
    os.environ['A_CONFIG_PATH'].value = '/home/me/current/task/config/path'
    A.do_some_stuff(kw)

# read the value
config_path = os.environ['A_CONFIG_PATH'].value

EDIT: since you said the env var is being read using os.getenv, you can combine the third solution with this hack, replacing os.getenv with your own:

# one time init
orig_getenv = os.getenv
def getenv_hacked(key, default=None):
    if key == 'A_CONFIG_PATH':
       return orig_getenv(key, default).value
     else:
       return orig_getenv(key, default)
os.getenv = getenv_hacked
os.environ['A_CONFIG_PATH'] = threading.local()

Upvotes: 3

Related Questions