CODEWITHSUNDEEP
dockerjenkinskubernetescontinuous-integrationsh
Würden
Würden

Reputation: 107

Jenkins sh script hangs when run in specific container

I'm trying to automate deployments using the official ArgoCD docker image (https://hub.docker.com/r/argoproj/argocd/dockerfile)

I've created a declarative jenkins pipeline using the kubernetes plugin for the agents and define the pod using yaml, the container definition looks like this:

pipeline {
    agent {
        kubernetes {
            yaml """
kind: Pod
metadata:
  name: agent
spec:
  containers:
  - name: maven
    image: maven:slim
    command:
    - cat
    tty: true
    volumeMounts:
      - name: jenkins-maven-cache
        mountPath: /root/.m2/repository
  - name: argocd
    image: argoproj/argocd:latest
    command:
    - cat
    tty: true
    ...

I'm trying to run commands inside that container, that step in the pipeline looks like this:

stage('Build') {
    steps {
        container('maven') {
            sh 'echo testing' // this works just fine
        }
    }
}
stage('Deploy') {
    steps {
        container('argocd') {
            sh "echo testing" // this does not work
            // more deploy scripts here, once sh works
        }
    }
}

So I have two containers, one where the sh script works just fine and another where it doesn't. The sh scripts in the "argocd" container just hangs for 5 minutes and then Jenkins kills it, the exit message is: process apparently never started in /home/jenkins/agent/workspace/job-name@tmp/durable-46cefcae (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)

I can't do echo a simple string in this particular container.

It works fine in other containers like the official for Maven from Docker, I use to build the spring boot application. I can also run commands directly in the argocd container manually from commandline with docker exec, but jenkins just won't in the pipeline for some reason. What could it be?

I am running the latest version (1.33) of the durable task plugin.

Update: Turns out that the image for argo-cd (continuous deployment tool) argoproj/argocd:latest does not include other commands except argocd, so the issue was with the container image I tried to use and not Jenkins itself. My solution was to install the Argo-CD CLI into a custom docker container and use that instead of the official one.

Upvotes: 8

Views: 9220

Answers (3)

Scott Kausler
Scott Kausler

Reputation: 11

While gkc's answer is correct I wanted to shed some more light on why. I ran sh as a non-root user with LAUNCH_DIAGNOSTICS=true and saw a number of permission denied errors writing to various files managed by the durable task plugin.

I am using the Kubernetes plugin with multiple containers in the pod. As far as I can tell, the files were created and owned by a user in the jnlp container. So when running sh in a second container the user needs permission to those files. This should work if both containers run as the same user, or if the second container is running as root.

One can still run a command as another user by setting runAsUser: 0 then using sudo to switch to the other user:

sh "sudo -H -u newuser bash -c 'whoami'"

Upvotes: 1

gkc
gkc

Reputation: 459

I've just encountered a similar issue with a custom docker image created by myself. It turns out, I was using USER nobody in Dockerfile of that image and somehow, this way Jenkins agent pod was unable to run cat command or any other shell command from my pipeline script. Running specific container with root user worked for me.

So in your case I would add securityContext: runAsUser: 0 like below.

...
  - name: argocd
    image: argoproj/argocd:latest
    command:
    - cat
    tty: true
    securityContext:
      runAsUser: 0
...

Kubernetes reference: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-container

Upvotes: 9

David
David

Reputation: 377

If the issue is Jenkins related here are some things that may help to solve the problem:

  1. Issues with the working directory, if you updated Jenkins from some older version the workdir was /home/jenkins while in the recent versions it should be /home/jenkins/agent or if you are running it in Windows the path should start with C:\dir and not with /dir
  2. You can try a new clean install with apt-get --purge remove jenkins and then apt-get install jenkins
  3. This is not your case as you run latest version of durable task plugin. But for other people reference versions prior to 1.28-1.30 caused the same issue.

If your Jenkins is clean the issue should be investigated in a different way, it seems that it's not returning an exit code to the sh command and/or the script is executed in a different shell. I would try to do an sh file to be placed in the working directory of the container 

#!/bin/bash
echo "testing"
echo $?

and try to run it with source my_script.sh or with bash my_script.sh

$? is the exit code of the latest bash operation, having it printed will make sure that your script is terminated correctly. The source command to run the script will make it run in the same shell that is calling it so the shell variables are accessible. Bash command will run it in another subshell instead.

Upvotes: -2

Related Questions