How to divide a script in a smaller script and a module and connect to workspace variable?

Question

I start up sessions in Jupyter with Python with a partly tailor-made script for the application I work with. The script contains both application dependent dictionaries and functions, but some function are of a general kind. I would like to make a general module and make the start-up script contain only application parts. The difficulty is that I want the functions in the general module have application dictionaries as default. So how to connect such workspace dictionaries to the imported functions?

A very simplifed example below illustrate the problem. First you see the original total startup script. This codes works.

import numpy as np
import matplotlib.pyplot as plt

parDict = {}
parDict['Y'] = 0.5
parDict['qSmax'] = 1.0
parDict['Ks'] = 0.1

def par(parDict=parDict, *x, **x_kwarg):
   """ Set parameter values if available in the predefined dictionaryt parDict. """
   x_kwarg.update(*x)
   x_temp = {}
   for key in x_kwarg.keys():
      if key in parDict.keys():
         x_temp.update({key: x_kwarg[key]})
      else:
         print(key, 'seems not an accessible parameter')
   parDict.update(x_temp)

And I can in the notebook give a command like par(Y=0.4) and then inspect the results in the dictionary parDict.

Second (below) you see an attempt to break out the general functions into a module and this functions are imported in the start-up script. And below the actual module. This code does not work. The error message is: name 'parDict' is not defined How to fix it?

parDict = {}
parDict['Y'] = 0.5
parDict['qSmax'] = 1.0
parDict['Ks'] = 0.1

from test_module import par

And test_module.py

def par(parDict=parDict, *x, **x_kwarg):
   """ Set parameter values if available in the predefined dictionaryt parDict. """
   x_kwarg.update(*x)
   x_temp = {}
   for key in x_kwarg.keys():
      if key in parDict.keys():
         x_temp.update({key: x_kwarg[key]})
      else:
         print(key, 'seems not an accessible parameter')
   parDict.update(x_temp)

If I in the function take away the default argument parDict then it works, but I must then have a lengthier call like par(parDict, Y=0.4). I would like avoid this lengthy call and provide the default parDict automatically. One idea is to in the start-up script make a new function from the imported function and here make the connection to the dictionary. But seems a clumsy way to do it, or the only choice?

Answer 1

Now I have put together an even better solution that I want to share with you! The key is to make use of Python class-concept that give the possibility to combine a dictionary with associated functionality and you import the class from a module. In the setup-file you then after import of the class populate the dictionary with application parameter names and value and you make a short name for the associated function. This could be described in terms of "abstract datatype" if you like.

The start-up script now look like this:

from test_module3 import ParDict

p = {}
p['Y'] = 0.5
p['qSmax'] = 1.0
p['Ks'] = 0.1

parDict = ParDict('Test3', p)

par = parDict.parX

and the updated test_module3 is now somewhat more complex:

class ParDict(object):
   """"Tool to associate a dictionary parDict with robustified way to   
       modifiy parameter values."""
          
   def __str__(self):
      return self.id

   def __init__(self, id, parDict):
      self.id = id 
      self.parDict = parDict
  
   def parX(self, *x, **x_kwarg):
      x_kwarg.update(*x)
      x_temp = {}
      for key in x_kwarg.keys():
         if key in self.parDict.keys():
            x_temp.update({key: x_kwarg[key]})
         else:
            print(key, 'seems not an accessible parameter')
      self.parDict.update(x_temp)
  

Finally out in the Jupyter notebook you can now change parmeter values in parDict as before with simplified command-line par(Y=0.4, Ks=0.08) etc.

Answer 2

The details of the tentative solution I sketched above look like this. The key idea is to in the start-up script import the function parX() from the module, but then make a new function par() where parDict is default parameter and thus "connected" to the function at this level, while at the module level it does not work to do this connection. Thus in the start-up script the dictionary parDict is implicitly a global variable that the function par() can operator on.

parDict = {}
parDict['Y'] = 0.5
parDict['qSmax'] = 1.0
parDict['Ks'] = 0.1

from test_module2 import parX

def par(*x, **x_kwarg):
   parX(parDict, *x, **x_kwarg)

and adjusted test_module2.py is:

def parX(parDict, *x, **x_kwarg):
   """ Set parameter values if available in the predefined dictionary parDict. """
   x_kwarg.update(*x)
   x_temp = {}
   for key in x_kwarg.keys():
       if key in parDict.keys():
          x_temp.update({key: x_kwarg[key]})
       else:
          print(key, 'seems not an accessible parameter')
   parDict.update(x_temp)

First tests show that it works. In this way I avoid introducing parDict as a default parameter, and simply access it as a global variable. So uncertainty of when default parameters are evaluated discussed in a link above is avoided. Can here be any problems with this code?

Suggestions for better ways to do this?

Generally I want to avoid using global variable and introducing them instead as default parameters improve readability, but it may introduce problems as the link above explains, and I dropped that idea. The reason I want to use workspace dictionaries like parDict as global variables is to have very short command-line functions like: par(),... to administrate the dictionaries with and later operate on their content. The name of the workspace dictionaries are always the same but the content is application dependent. The general functions like par() that operate on the dictionaries need to be gradually updated, and extended etc, and good to do that centrally from a module, rather than from each individual start-up script.

Answer 3

At the place where you try to define the function par the dictionary parDict is undefined. You could move the definition of parDict into test_module.py (before the function definition).

As a side note, please be aware of the dangers of mutable default arguments in Python: https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments

They can cause unexpected behavior.

My suggestion:

def par(parDict=None, *x, **x_kwarg):
   """ Set parameter values if available in the predefined dictionaryt parDict. """

    if parDict is None:
        parDict = {}
        parDict['Y'] = 0.5
        parDict['qSmax'] = 1.0
        parDict['Ks'] = 0.1

    x_kwarg.update(*x)
    x_temp = {}
    for key in x_kwarg.keys():
       if key in parDict.keys():
          x_temp.update({key: x_kwarg[key]})
       else:
          print(key, 'seems not an accessible parameter')
    parDict.update(x_temp)

    return parDict 

Then call like this:

parDict = par(Y=2)