CODEWITHSUNDEEP
pythonnextflownextflow-tower
jlo-gmail
jlo-gmail

Reputation: 5038

Simple Python Script in Nextflow fails

I am attempting to run Python from inside a Nextflow pipeline launched via Nextflow-Tower. I forked the nextflow-io/hello repo and made sure I could launch it successfully from my own repo. What I am doing wrong? If there is a simple python pipeline repo that can be launched(foolproof) using Nextflow-Tower that would be helpful.

My dir structure is:

root
--main.nf
--nextflow.config
--bin
----test.py

nextflow.config

process.container = "us-east1-docker.pkg.dev/xxxx-yyyy/pass/python3:latest"
docker.enabled = true

main.nf

#!/usr/bin/env nextflow
nextflow.enable.dsl=2 

process sayHello {
  input: 
    val x
  output:
    stdout
  script:
    """
    echo '$x world!'
    """
}

process processPython {
  input: 
    val y
  output:
    stdout
  script:
    """
    #!/usr/bin/env python3
    test.py
    """
}

workflow {
  Channel.of('Bonjour', 'Ciao', 'Hello', 'Hola') | sayHello | view

  Channel.of('foo', 'bar', 'cluster', 'frack') | processPython | view
}

test.py

import sys
print(f"Python Version: {sys.version}")

Error report

Error executing process > 'processPython (1)'

Caused by:
  Process `processPython (1)` terminated with an error exit status (127)

Command executed:

  #!/usr/bin/env python3
  test.py

Command exit status:
  127

Command output:
  (empty)

Command error:
  /usr/bin/env: 'python3': No such file or directory

Work dir:
  gs://nf-tower-6897ad32-d3d4-4bda-ade1-97d25c0e680c/scratch/4wCvYTX4GV6S3k/98/707cf12d829d888f4f5179ee1ae55e

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

Upvotes: 0

Views: 1007

Answers (2)

jlo-gmail
jlo-gmail

Reputation: 5038

The solution was to include shebang IN test.py:

#!/usr/bin/env python3
import sys
print(f"Python Version: {sys.version}")

process:

process processPython {

  container = 'us-east1-docker.pkg.dev/xxxx-yyyy/pass/python3:latest'

  input: 
    val y
  output:
    stdout
  script:
    """
    python3 --version
    test.py
    """
}

Upvotes: 0

Patrick H.
Patrick H.

Reputation: 293

The execution of the workflow is fine and seems to work. But the process fails, here because you do not have an executable known as python3 in your container. (As can be seen from the Command error: output section). However, the process is anyway not correct. There are two solutions:

Either use a python script in the script block of the nextflow process directly:

script:
"""
#!/usr/bin/env python

import sys
print(f"Python Version: {sys.version}")
"""

Or write the python code in a separate, executable file in your projectDir/bin directory (like you already did) and use the script section of the Nextflow process as bash-wrapper script.

script:
"""
test.py
"""

I personally prefer to use the second solution (bash wrapper script), even if the script is only very short. There are two reasons: 1) From my experience, IDEs are more helpful if you don't have all your code written as multiline-strings. 2) I like to throw all processes in a single modules.nf file and want to keep my process script blocks rather short in this file to keep the overview.

Note that I replaced python3 by python which should be okay for your testing case. If you want to be explicit here, make sure you use a container which knows python3 - Although to be honest, I think that this is not very typical anymore, as python2 is pretty much dead and one rarely needs both python2 and python3 in a single container. Given that you should only compose the image of things you really need for each task you might have either python2 or python3.

Upvotes: 0

Related Questions