CODEWITHSUNDEEP
pythonparallel-processingmultiprocessingemcee
V.I.Raj
V.I.Raj

Reputation: 21

Identical steps while running emcee in parallel with Pool

I am running an MCMC process in Python using emcee. My python code is a wrapper around a different software. emcee walkers sample the input parameter space to this software. The software reads a text file as its input. My code generates this text file using each step of the emcee chain. To speed up the sampling, I want multiple processes that evaluate the log-likelihood, running in parallel. However, each of these processes need to write to different input text files, to prevent interference between the processes. For this reason I think I cannot use the simple standard example on the emcee website.

Ideally, I would want each process running in its own subdirectory, generating different input files so that the different processes do not interfere with each other. To do this, I use Pool from multiprocessing to create parallel processes manually rather than the default option from emcee. I understand that in order to make different parallel processes generate different random numbers, I need to supply different seeds to each PRNG. My current code is as follows :

def log_likelihood(x):
    #update input text file and run software
    return log_likelihood

def run_mcmc(inp_directory)
    os.chdir(inp_directory)
    seedint = int(dusty_inpdir.split('_')[-1]) #sets a different seed for each process
    np.random.seed(seedint)
    sampler = emcee.EnsembleSampler(nwalkers, ndim, log_likelihood)
    sampler.run_mcmc(initial_pos,ntrials)

pool = Pool()
inp_dirlist = ['proc_0','proc_1','proc_2','proc_3']
pool.map(run_mcmc,inp_dirlist)

In spite of giving different seeds to each process, all 4 processes generate exactly the same set of emcee steps.

Is np.random.seed not the relevant parameter to seed the emcee PRNG? Is there a better way to have different processes working in different directories?

Upvotes: 2

Views: 1103

Answers (1)

OriolAbril
OriolAbril

Reputation: 8823

Using np.random.seed sets the global state of the numpy random number generator. To get each emcee sampler to have a different seed, you should use the seed argument:

sampler = emcee.EnsembleSampler(nwalkers, ndim, log_likelihood, seed=seedint)

Upvotes: 0

Related Questions