CODEWITHSUNDEEP
pythonpython-3.xparameter-passingcommand-line-arguments
hechth
hechth

Reputation: 95

Best way to pass function specified in file x as commandline parameter to file y in python

I'm writing a wrapper or pipeline to create a tfrecords dataset to which I would like to supply a function to apply to the dataset.

I would like to make it possible for the user to inject a function defined in another python file which is called in my script to transform the data.

Why? The only thing the user has to do is write the function which brings his data into the right format, then the existing code does the rest.

I'm aware of the fact that I could have the user write the function in the same file and call it, or to have an import statement etc.

So as a minimal example, I would like to have file y.py

def main(argv):
    # Parse args etc, let's assume it is there.

    dataset = tf.data.TFRecordDataset(args.filename)
    dataset = dataset.map(args.function)
    # Continue with doing stuff that is independent from actual content

So what I'd like to be able to do is something like this

    python y.py --func x.py my_func

And use the function defined in x.py my_func in dataset.map(...)

Is there a way to do this in python and if yes, which is the best way to do it?

Upvotes: 1

Views: 62

Answers (2)

BenedictWilkins
BenedictWilkins

Reputation: 1253

  1. Pass the name of the file as an argument to your script (and function name)
  2. Read the file into a string, possibly extracting the given function
  3. use Python exec() to execute the code

An example:

file = "def fun(*args): \n  return args"
func = "fun(1,2,3)"

def execute(func, file):
    program = file + "\nresult = " + func
    local = {}
    exec(program, local)
    return local['result']

r = execute(func, file)
print(r)

Similar to here however we must use locals() as we are not calling exec in global scope.

Note: the use of exec is somewhat dangerous, you should be sure that the function is safe - if you are using it then its fine!

Hope this helps.

Upvotes: 1

hechth
hechth

Reputation: 95

Ok so I have composed the answer myself now using the information from comments and this answer.

    import importlib, inspect, sys, os
    # path is given path to file, funcion_name is name of function and args are the function arguments

    # Create package and module name from path
    package = os.path.dirname(path).replace(os.path.sep,'.')
    module_name = os.path.basename(path).split('.')[0]

    # Import module and get members
    module = importlib.import_module(module_name, package)
    members = inspect.getmembers(module)

    # Find matching function
    function = [t[1] for t in members if t[0] == function_name][0]
    function(args)

This exactly solves the question, since I get a callable function object which I can call, pass around, use it as a normal function.

Upvotes: 0

Related Questions