CODEWITHSUNDEEP
pythonjson
louise55
louise55

Reputation: 11

How to use a config file to run a python file multiple times with different settings?

I have one python script with it's settings in one seperate json config file. The json file looks like this: 

{
  "connection" : {
      "db_server" : "server",
      "db_name" : "table1", 
      "db_user" : "user1", 
}}

Now I need to run the same python file more than one time, each with other settings in the config file. The other settings would look like this:

{
  "connection" : {
      "db_server" : "server",
      "db_name" : "table2", 
      "db_user" : "user2", 
}}

I don't need to change anything in the Python script. I open the json file in my Python script like this:

with open('settings.json') as json_data_file:
    data = json.load(json_data_file)
    json_data_file.close()

Since you cannot add comments in a json file, I don't know the easiest way to do this. I want the Python script to run simultaneously two times, each time with other settings for the json file.

Thanks in advance!

Upvotes: 1

Views: 2199

Answers (2)

battr
battr

Reputation: 128

A simple solution is a new script that parses your JSON file, imports your Python scripts, and then executes that scripts with different parameters using concurrent.

An example (adapted from the example for ThreadPoolExecutor):

import concurrent.futures
import json
from YourModule import MainFunction

# First, open and load your JSON file
parameter_dict = json.load(open('yourfilename.json'))

# Do some parsing to your parameters in order
# (In your case, servers, tables, and users)
parameters_to_submit = [[entry['db_server'], entry['db_table'], entry['db_user'] for entry in parameter_dict.values()]

# Now, generate a ThreadPool to run your script multiple times
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    # Submit the function + parameters to the executor
    submitted_runs = {
        executor.submit(MainFunction, params[0], params[1], params[2]): params
        for params in parameters_to_submit
    }

    # as the results come in, print the results
    for future in concurrent.futures.as_completed(submitted_runs):
        url = submitted_runs[future]
        try:
            data = future.result()
        except Exception as exc:
            print(f'Script generated an exception: {exc}')
        else:
            # if need be, you could also write this data to a file here
            print(f'Produced result {data}')

Upvotes: 0

lnyng
lnyng

Reputation: 965

  • After launching the the python script, you can just modify the config file inplace and run it a second time. This won't affect the already-running python program because they already read the config.
  • Or you can have multiple config files with different names, and run the script with some command line argument (i.e. sys.argv[1]) to choose which config file to use. I personally recommend this approach.

Upvotes: 3

Related Questions