CODEWITHSUNDEEP
dockerjenkins
Yu Chen
Yu Chen

Reputation: 7500

Jenkins docker container simply hangs and never executes steps

I'm trying to run a Python image in Jenkins to perform a series of unit tests with pytest, but I'm getting some strange behavior with Docker.

My Jenkinsfile pipeline is

  agent {
      docker { image 'python:3.6-jessie' }
    }

    stages {
      stage('Run tests') {
        steps {
            withCredentials([
                string(credentialsId: 'a-secret', variable: 'A_SECRET')
            {

              sh label: "Install dependencies", script: 'pip install -r requirements.txt'
              sh label: 'Execute tests', script: "pytest mytests.py"
            }
     }
  }
}

However, when I run the pipeline, Docker appears to be executing a very long instruction (with significantly more -e environment variables than I defined as credentials?), followed by cat.

The build then simply hangs and never finishes:

         Jenkins does not seem to be running inside a container
            $ docker run -t -d -u 996:994 
    -w /var/lib/jenkins/workspace/myproject 
    -v /var/lib/jenkins/workspace/myproject:/var/lib/jenkins/workspace/myproject:rw,z
    -v /var/lib/jenkins/workspace/myproject@tmp:/var/lib/jenkins/workspace/myproject@tmp:rw,z 
-e ******** -e ******** python:3.6-jessie cat

When I SSH into my instance and run docker ps, I see 

CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
240d00459d92        python:3.6-jessie   "cat"               About a minute ago   Up About a minute                       kind_wright

Why is Jenkins running cat? Why does Jenkins say I am not running inside a container, when it has clearly created a container for me? And most importantly, why are my pip install -r requirements and other steps not executing?

Upvotes: 5

Views: 6147

Answers (3)

RealFrogo
RealFrogo

Reputation: 11

I had a similar case, using a Windows docker environment as Jenkins agent, but with a hanging "bat" call just the same when trying to run my pipeline using a docker image. Trying to figure out where the problem is (as I didn't get any hints whatsoever), I stumbled upon this question's answer here which gave me a push in the right direction.

Turned out the problem was not with environment variables, but with the name of the jenkins job: I had spaces in the name of the job (e.g. "My Jenkins Job"), which probably weren't properly escaped when dealing with the container. When I changed that (e.g. to "My-Jenkins-Job"), everything worked fine. So it might be a similar cause (values empty or at least not as expected), but for a different reason.

Upvotes: 1

Kevin Yue
Kevin Yue

Reputation: 392

I solved this problem by appending the -u root parameter to the args:

agent {
    docker {
        image 'alpine/k8s:1.27.4'
        args "-u root"
    }
}

I know using root might not be a good idea, but it works and I think it's okay for my CI environment.

Upvotes: 0

Yu Chen
Yu Chen

Reputation: 7500

I finally figured this out. If you have empty global environment variables in your Jenkins configuration, it appears that you'll get a malformed docker run command since Jenkins will write the command, with your empty string environment variable, as docker run -e some_env_var=some_value -e = ...

This will cause the container to simply hang.

A telltale sign that this is happening is you'll get the error message:

invalid argument "=" for "-e, --env" flag: invalid environment variable: =

This is initially difficult to diagnose since Jenkins (rightfully) hides your actual credentials with ***, so the empty environment strings do not show up as empty.

You need to check your Jenkins global configuration and make sure you don't have any empty environment variables accidentally defined:

enter image description here

If these exist, you need to delete them and rerun.

Upvotes: 5

Related Questions