CODEWITHSUNDEEP
pythontestingjenkinsdistributed
sagargp
sagargp

Reputation: 1070

Shell scripts have different behavior when launched by Jenkins

I have a ton of scripts I need to execute, each on a separate machine. I'm trying to use Jenkins to do this. I have a Python script that can execute a single test and handles time limits and collection of test results, and a handful of Jenkins jobs that run this Python script with different args. When I run this script from the command line, it works fine. But when I run the script via Jenkins (with the exact same arguments) the test times out. The script handles killing the test, so control is returned all the way back to Jenkins and everything is cleaned up. How can I debug this? The Python script is using subprocess.popen to launch the test.

As a side note, I'm open to suggestions for how to do this better, with or without Jenkins and my Python script. I just need to run a bunch of scripts on different machines and collect their output.

Upvotes: 1

Views: 1830

Answers (2)

sagargp
sagargp

Reputation: 1070

I solved my own problem and even though it's a bit of a corner case, I'll still answer it here. The script that Jenkins was launching itself launched a threaded Python server using ThreadedTCPServer implemented more or less exactly from here. That threaded server wasn't properly exiting, so it left some pipes open. Although the server process died, the leaky pipes made it impossible for Jenkins to properly determine that the process was over (Jenkins waits for an EOF from the child process to determine if the process completed). The solution was a reimplementation of the socket server that exited properly. Hope this helps someone in the future!

Upvotes: 2

Charles Duffy
Charles Duffy

Reputation: 295413

To debug this:

  • Add set -x towards the top of your shell script.
  • Set a PS4 which prints the line number of each line when it's invoked: PS4='+ $BASH_SOURCE:$FUNCNAME:$LINENO:'
  • Look in particular for any places where your scripts assume environment variables which aren't set when Hudson is running.

If your Python scripts redirect stderr (where logs from set -x are directed) and don't pass it through to Hudson (and so don't log it), you can redirect it to a file from within the script: exec 2>>logfile

There are a number of tools other than Jenkins for kicking off jobs across a number of machines, by the way; MCollective (which works well if you already use Puppet), knife ssh (which you'll already have if you use Chef -- which, in my not-so-humble opinion, you should!), Rundeck (which has a snazzy web UI, but shouldn't be used by anyone until this security bug is fixed), Fabric (which is a very good choice if you don't have mcollective or knife already), and many more.

Upvotes: 1

Related Questions