CODEWITHSUNDEEP
pythonfabric
Flasking
Flasking

Reputation: 101

How to transfer env.host to fabric when using multiprocessing?

I have hundreds of box can be accessed only with different gateway, like below shows:

gateway1:ip1,ip2
gateway2:ip3,ip4 ...

All job need done in one minute, so I use multiprocess below command fab -f ytj_sto.py doitnow, errors below.

[]
None
None
***Warning*** Host None via ssh is down

Code:

@parallel(pool_size=20)
def coll():
    print env.hosts
    print env.host
    print env.gateway
    if _is_ssh_ok(env.host):
        d = patt()

def doitnow():
    p=Pool(20)
    with open('ytj_sto.hn','r') as f:
        for line in f.readlines():
            line = line.strip()
            if not len(line) or line.startswith('#'):
                continue
            env.gateway = line.split(':')[0]
            env.hosts = line.split(':')[1].split(',')
            result = p.apply_async(coll, args=())
            result.get()
    p.close()
    p.join()

EDITED: I have used fab -H -g solved the issue,thanks all

  def fabfun(Hosts,Gate,des,func1):
        with settings(hide('running'), warn_only=True):
           local(("fab -H %s -g %s -f %s %s ") % (Hosts,Gate,des,func1))

p=Pool(20)
starttime = time.asctime( time.localtime(time.time()) )
print('Waiting for all job done...%s' % starttime)
with open('ytj_sto.hn','r') as f:
    for line in f.readlines():
        line = line.strip()
        if not len(line) or line.startswith('#'):
            continue
        Hosts = line.split(':')[1]
        Gate = line.split(':')[0]
        p.apply_async(fabfun, args=(Hosts,Gate,des,func1))
    p.close()
    p.join()

Upvotes: 0

Views: 326

Answers (2)

2ps
2ps

Reputation: 15926

If you want to set env variables dynamically like you are, you should use execute. That way the executed task takes on the env values that you set at runtime. But unfortunately, because fabric is not fully thread-safe, you can only kick off tasks like this in parallel in your situation once per gateway because env is a global singleton.

A simple but integral aspect of Fabric is what is known as the “environment”: a Python dictionary subclass, which is used as a combination settings registry and shared inter-task data namespace.

The environment dict is currently implemented as a global singleton, fabric.state.env, and is included in fabric.api for convenience. Keys in env are sometimes referred to as “env variables”.

from fabric.context_managers import env

@parallel(pool_size=20)
def check_and_patt():
    if _is_ssh_ok(env.host):
        d = patt()

def doitnow():
    p=Pool(20)
    with open('ytj_sto.hn','r') as f:
        for line in f.readlines():
            line = line.strip()
            if not len(line) or line.startswith('#'):
                continue
            env.gateway = line.split(':')[0]
            env.hosts = line.split(':')[1].split(',')
            result = execute(check_and_patt)

Upvotes: 1

YellowShark
YellowShark

Reputation: 2269

I think you can just supply env to the coll function as a parameter, like so:

@parallel(pool_size=20)
def coll(env): # <-- updated
    print env.hosts
    print env.host
    print env.gateway
    if _is_ssh_ok(env.host):
        d = patt()

def doitnow():
    p=Pool(20)
    with open('ytj_sto.hn','r') as f:
        for line in f.readlines():
            line = line.strip()
            if not len(line) or line.startswith('#'):
                continue
            env.gateway = line.split(':')[0]
            env.hosts = line.split(':')[1].split(',')
            result = p.apply_async(coll, args=(env,)) # <-- updated
            result.get()
    p.close()
    p.join()

There's a few quirks to using the multiprocessing library. This info might be especially relevant to you scenario:

Global variables

Bear in mind that if code run in a child process tries to access a global variable, then the value it sees (if any) may not be the same as the value in the parent process at the time that Process.start was called.

However, global variables which are just module level constants cause no problems.

Upvotes: 0

Related Questions