CODEWITHSUNDEEP
pythonparallel-processingfabric
SQA777
SQA777

Reputation: 372

How can I get this to run in parallel using Fabric?

I asked a related question earlier:

How to issue commands on remote hosts in parallel using Fabric without using a fabfile?

I'm having problems executing tests in parallel on various test hosts.

My code looks like this:

@parallel  
def run_test(arg_list):
    # arg_list is a list of dictionary.  Each entry in
    # arg_list has a 'ip_address' and a 'test_config_file'

    for x in arg_list:
        ip_address = x['ip_address']
        test_config_file = x['test_config_file']
        env['host_string'] = ip_address
        # The test program "test_localhost.py" is already on all the Test hosts
        cmd = "/root/test_localhost.py --ip_addr=" + ip_address + " --config=" + test_config_file
        run(cmd)


if __name__ == '__main__':

    env.parallel = True

    # Each test host will have unique test_config_files
    arg_list = list()
    arg_list.append({'ip_address':'10.10.10.10', 'test_config_file': "config_01.json"})
    arg_list.append({'ip_address':'10.10.10.11', 'test_config_file': "config_02.json"})

    execute(run_test, arg_list)

I've run this code against 2+ test hosts. I have a script that checks to see if the tests are running on the test hosts. The tests are not run in parallel.

Instead, the tests are run sequentially - the "test_localhost.py" is first run on 10.10.10.10, then after it is finished, it is run on 10.10.10.11.

Is there anything else I need to do to make the tests run in parallel?

Note: I can't use fabfile because I am sending different test configuration files for each test host.

Upvotes: 3

Views: 4294

Answers (1)

2ps
2ps

Reputation: 15926

Here’s what I would do. The trick for different parameters for different hosts is to add info to the env context_manager to let you grab a hold of host-specific arguments for a task. Just make sure that the host matches up to the key that you use for the dictionary and the parallel commands work fine. Finally, tasks in fabric are usually run through the fab command. I’ve never tried it using a native python script and am unsure of the ramifications or if any special handling needs to be done. The typical format for a fabric project is to have tasks defined in a file called fabfile.py and those tasks can be run using fab.

In a file called fabfile.py:

from fabric.decorators import task, parallel
from fabric.operations import run
from fabric.context_managers import env

@task
def run_test(arg_list):
    # arg_list is a list of dictionary.  Each entry in
    # arg_list has a 'ip_address' and a 'test_config_file'
    env.hosts = [ x['ip_address'] for x in arg_list ]
    env.host_data = dict()
    for x in arg_list:
        env.host_data[x['ip_address']] = x
    execute(run_command)

@parallel
def run_command():
    context = env.host_data[env.host]
    cmd = '/root/test_localhost.py --ip_addr=%(ip_address)s --config=%(test_config_file)s' % context
    run(cmd)

@task 
def run_my_test():
    arg_list.append({'ip_address':'10.10.10.10', 'test_config_file': "config_01.json"})
    arg_list.append({'ip_address':'10.10.10.11', 'test_config_file': "config_02.json"})
    run_test(arg_list)

From the command line run:

fab run_my_test

Upvotes: 3

Related Questions