CODEWITHSUNDEEP
snakemake
goi42
goi42

Reputation: 65

Snakemake: --use-conda with --cluster

When I run with --cluster and --use-conda, Snakemake does not appear to set the conda environment before submitting to the cluster, and my jobs fail accordingly. Is there a trick I am missing to set the conda environment before cluster submission?

EDIT:

I get snakemake in a conda environment like:

channels:
  - bioconda
  - conda-forge
dependencies:
  - snakemake-minimal=5.19.3
  - xrootd=4.12.2

Reproducer:

I create a directory with Snakefile, dothing.py, and environment.yml:

Snakefile:

shell.prefix('unset PYTHONPATH; unset LD_LIBRARY_PATH; unset PYTHONHOME; ')

rule dothing:
    conda: 'environment.yml'
    output: 'completed.out'
    log: 'thing.log'
    shell: 'python dothing.py &> {log} && touch {output}'

dothing.py:

import uncertainties

print('it worked!')

environment.yml:

name: testsnakeconda
channels:
  - conda-forge
dependencies:
  - uncertainties=3.1.4

If I run locally like

snakemake --cores all --use-conda

It runs with no problems:

Building DAG of jobs...
Creating conda environment environment.yml...
Downloading and installing remote packages.
Environment for environment.yml created (location: .snakemake/conda/e0fff47f)
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Job counts:
        count   jobs
        1       dothing
        1

[Tue Jun 30 16:19:38 2020]
rule dothing:
    output: completed.out
    log: thing.log
    jobid: 0

Activating conda environment: /path/to/environment.yml
[Tue Jun 30 16:19:39 2020]
Finished job 0.
1 of 1 steps (100%) done
Complete log: /path/to/.snakemake/log/2020-06-30T161824.906217.snakemake.log

If I try to submit using --cluster like

snakemake --cores all --use-conda --cluster 'condor_qsub -V -l procs={threads}' --latency-wait 30 --max-jobs-per-second 100 --jobs 50

there is no message about setting up a conda environment and the job fails with an error:

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cluster nodes: 50
Job counts:
        count   jobs
        1       dothing
        1

[Tue Jun 30 16:20:49 2020]
rule dothing:
    output: completed.out
    log: thing.log
    jobid: 0

Submitted job 0 with external jobid 'Your job 9246856 ("snakejob.dothing.0.sh") has been submitted'.
[Tue Jun 30 16:26:00 2020]
Error in rule dothing:
    jobid: 0
    output: completed.out
    log: thing.log (check log file(s) for error message)
    conda-env: /path/to/.snakemake/conda/e0fff47f
    shell:
        python dothing.py &> thing.log && touch completed.out
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
    cluster_jobid: Your job 9246856 ("snakejob.dothing.0.sh") has been submitted

Error executing rule dothing on cluster (jobid: 0, external: Your job 9246856 ("snakejob.dothing.0.sh") has been submitted, jobscript: /path/to/.snakemake/tmp.a7fpixla/snakejob.dothing.0.sh). For error details see the cluster log and the log files of the involved rule(s).
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /path/to/.snakemake/log/2020-06-30T162049.793041.snakemake.log

and I can see that the problem is that the uncertainties package is not available:

$ cat thing.log
Traceback (most recent call last):
  File "dothing.py", line 1, in <module>
    import uncertainties
ImportError: No module named uncertainties

EDIT: verbose output without --cluster:

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 10
Rules claiming more threads will be scaled down.
Job counts:
        count   jobs
        1       dothing
        1
Resources before job selection: {'_cores': 10, '_nodes': 9223372036854775807}
Ready jobs (1):
        dothing
Selected jobs (1):
        dothing
Resources after job selection: {'_cores': 9, '_nodes': 9223372036854775806}

[Thu Jul  2 21:51:18 2020]
rule dothing:
    output: completed.out
    log: thing.log
    jobid: 0

Activating conda environment: /path/to/workingdir/.snakemake/conda/e0fff47f
[Thu Jul  2 21:51:33 2020]
Finished job 0.
1 of 1 steps (100%) done
Complete log: /path/to/workingdir/.snakemake/log/2020-07-02T215117.964474.snakemake.log
unlocking
removing lock
removing lock
removed all locks

verbose output with --cluster:

Building DAG of jobs...
Checking status of 0 jobs.
Using shell: /usr/bin/bash
Provided cluster nodes: 50
Job counts:
    count   jobs
    1   dothing
    1
Resources before job selection: {'_cores': 9223372036854775807, '_nodes': 50}
Ready jobs (1):
    dothing
Selected jobs (1):
    dothing
Resources after job selection: {'_cores': 9223372036854775806, '_nodes': 49}

[Thu Jul  2 21:40:23 2020]
rule dothing:
    output: completed.out
    log: thing.log
    jobid: 0

Jobscript:
#!/bin/sh
# properties = {"type": "single", "rule": "dothing", "local": false, "input": [], "output": ["completed.out"], "wildcards": {}, "params": {}, "log": ["thing.log"], "threads": 1, "resources": {}, "jobid": 0, "cluster": {}}
 cd /path/to/workingdir && \
/path/to/miniconda/envs/envname/bin/python3.8 \
-m snakemake dothing --snakefile /path/to/workingdir/Snakefile \
--force -j --keep-target-files --keep-remote \
--wait-for-files /path/to/workingdir/.snakemake/tmp.5n32749i /path/to/workingdir/.snakemake/conda/e0fff47f --latency-wait 30 \
 --attempt 1 --force-use-threads \
--wrapper-prefix https://github.com/snakemake/snakemake-wrappers/raw/ \
   --allowed-rules dothing --nocolor --notemp --no-hooks --nolock \
--mode 2  --use-conda  && touch /path/to/workingdir/.snakemake/tmp.5n32749i/0.jobfinished || (touch /path/to/workingdir/.snakemake/tmp.5n32749i/0.jobfailed; exit 1)

Submitted job 0 with external jobid 'Your job 9253728 ("snakejob.dothing.0.sh") has been submitted'.
Checking status of 1 jobs.
...
Checking status of 1 jobs.
[Thu Jul  2 21:46:23 2020]
Error in rule dothing:
    jobid: 0
    output: completed.out
    log: thing.log (check log file(s) for error message)
    conda-env: /path/to/workingdir/.snakemake/conda/e0fff47f
    shell:
        python dothing.py &> thing.log && touch completed.out
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
    cluster_jobid: Your job 9253728 ("snakejob.dothing.0.sh") has been submitted

Error executing rule dothing on cluster (jobid: 0, external: Your job 9253728 ("snakejob.dothing.0.sh") has been submitted, jobscript: /path/to/workingdir/.snakemake/tmp.5n32749i/snakejob.dothing.0.sh). For error details see the cluster log and the log files of the involved rule(s).
Cleanup job metadata.
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /path/to/workingdir/.snakemake/log/2020-07-02T214022.614691.snakemake.log
unlocking
removing lock
removing lock
removed all locks

Upvotes: 1

Views: 2229

Answers (1)

JulverN
JulverN

Reputation: 121

What worked for me is to set the full path to your python interpreter inside the rule itself...

rule dothing:
    conda: 'environment.yml'
    output: 'completed.out'
    log: 'thing.log'
    shell: '/full_path_to_your_environment/bin/python dothing.py &> {log} && touch {output}'

and the full path to your python script if it's a package installed in that specific environnement (which is my case).

rule dothing:
    conda: 'environment.yml'
    output: 'completed.out'
    log: 'thing.log'
    shell: '/full_path_to_your_environment/bin/python /full_path_to_your_environment/package_dir/dothing.py &> {log} && touch {output}'

by /full_path_to_your_environment/ I mean the hash name conda and snakemake gave to your env the first time they installed it (e.g. /path/to/workingdir/.snakemake/conda/e0fff47f)

It's a bit uggly but still did the trick.

Hope that it'll help

Upvotes: 2

Related Questions