CODEWITHSUNDEEP
javamysqldockerkubernetesjenkins
Dev Fh
Dev Fh

Reputation: 644

Jenkins agent as kubernetes pods with kubedock to run testcontainers getting connection refused

I have Jenkins pipeline to build and run integration tests which require docker environment to run testcontainers that create Mysql database in Kubernetes with Kubedock.

Jenkins agent pod and Kubedock pods are getting created successfully however test are failling with the connection error below.

Below are the application logs when attempting to connect to Mysql database in maven test.

2024-10-17T06:25:41.663Z  INFO 231 --- [           main] org.testcontainers.DockerClientFactory   : Testcontainers version: 1.20.2
2024-10-17T06:25:42.403Z  INFO 231 --- [           main] o.t.d.DockerClientProviderStrategy       : Found Docker environment with Environment variables, system properties and defaults. Resolved dockerHost=tcp://127.0.0.1:2475
2024-10-17T06:25:42.403Z  WARN 231 --- [           main] o.t.d.DockerClientProviderStrategy       : Could not determine Docker OS type
2024-10-17T06:25:42.404Z  INFO 231 --- [           main] org.testcontainers.DockerClientFactory   : Docker host IP address is 127.0.0.1
2024-10-17T06:25:42.418Z  INFO 231 --- [           main] org.testcontainers.DockerClientFactory   : Connected to docker: 
  Server Version: 0.17.0-2-g6353a5f
  API Version: 1.25
  Operating System: kubernetes
  Total Memory: 0 MB
2024-10-17T06:25:42.421Z  WARN 231 --- [           main] o.testcontainers.utility.ResourceReaper  : 
********************************************************************************
Ryuk has been disabled. This can cause unexpected behavior in your environment.
********************************************************************************
2024-10-17T06:25:46.458Z  INFO 231 --- [           main] tc.mysql:8.0.39                          : Creating container for image: mysql:8.0.39
2024-10-17T06:25:46.467Z  INFO 231 --- [           main] o.t.utility.RegistryAuthLocator          : Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: mysql:8.0.39, configFile: /root/.docker/config.json, configEnv: DOCKER_AUTH_CONFIG). Falling back to docker-java default behaviour. Exception message: Status 404: No config supplied. Checked in order: /root/.docker/config.json (file not found), DOCKER_AUTH_CONFIG (not set)
2024-10-17T06:25:57.153Z  INFO 231 --- [           main] tc.mysql:8.0.39                          : Container mysql:8.0.39 is starting: f279144bfdbe5dbfa6e90c1103675e65572dc335843cb4c99fcc5d4d2f1b1c16
2024-10-17T06:25:57.193Z  INFO 231 --- [eam--1797563069] z.co.customtech.AbstractIntegrationTest  : STDOUT: 2024-10-17 06:25:50+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.39-1.el9 started.
2024-10-17T06:25:57.194Z  INFO 231 --- [eam--1797563069] z.co.customtech.AbstractIntegrationTest  : STDOUT: 2024-10-17 06:25:54+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-10-17T06:25:57.194Z  INFO 231 --- [eam--1797563069] z.co.customtech.AbstractIntegrationTest  : STDOUT: 2024-10-17 06:25:54+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.39-1.el9 started.
2024-10-17T06:25:57.194Z  INFO 231 --- [eam--1797563069] z.co.customtech.AbstractIntegrationTest  : STDOUT: 2024-10-17 06:25:55+00:00 [Note] [Entrypoint]: Initializing database files
2024-10-17T06:25:57.194Z  INFO 231 --- [eam--1797563069] z.co.customtech.AbstractIntegrationTest  : STDOUT: 2024-10-17T06:25:55.178476Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2024-10-17T06:25:57.194Z  INFO 231 --- [eam--1797563069] z.co.customtech.AbstractIntegrationTest  : STDOUT: 2024-10-17T06:25:55.190788Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.39) initializing of server in progress as process 80
2024-10-17T06:25:57.234Z  INFO 231 --- [           main] tc.mysql:8.0.39                          : Waiting for database connection to become available at jdbc:mysql://127.0.0.1:38259/blogger using query 'SELECT 1'
2024-10-17T06:25:57.289Z  INFO 231 --- [eam--1797563069] z.co.customtech.AbstractIntegrationTest  : STDOUT: 2024-10-17T06:25:55.568910Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-10-17T06:26:15.307Z  INFO 231 --- [eam--1797563069] z.co.customtech.AbstractIntegrationTest  : STDOUT: 2024-10-17T06:26:15.156186Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-10-17T06:26:56.726Z  INFO 231 --- [eam--1797563069] z.co.customtech.AbstractIntegrationTest  : STDOUT: 2024-10-17T06:26:56.586681Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2024-10-17T06:27:57.691Z ERROR 231 --- [           main] tc.mysql:8.0.39                          : Could not start container

java.lang.IllegalStateException: Container is started, but cannot be accessed by (JDBC URL: jdbc:mysql://127.0.0.1:38259/blogger), please check container logs
    at org.testcontainers.containers.JdbcDatabaseContainer.waitUntilContainerStarted(JdbcDatabaseContainer.java:210) ~[jdbc-1.20.2.jar:na]
    at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:500) ~[testcontainers-1.20.2.jar:na]
    at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:354) ~[testcontainers-1.20.2.jar:na]

Kubedock Pod

[GIN-debug] Listening and serving HTTP on :2475
I1017 06:23:48.773547       1 main.go:66] start cleaning lingering objects...
I1017 06:23:49.026240       1 main.go:68] finished cleaning lingering objects...
I1017 06:24:49.026772       1 main.go:66] start cleaning lingering objects...
I1017 06:24:50.706661       1 main.go:68] finished cleaning lingering objects...
[GIN] 2024/10/17 - 06:25:42 |[97;42m 200 [0m|     449.762µs |       127.0.0.1 |[97;44m GET     [0m "/info"
[GIN] 2024/10/17 - 06:25:42 |[97;42m 200 [0m|      30.871µs |       127.0.0.1 |[97;44m GET     [0m "/version"
[GIN] 2024/10/17 - 06:25:42 |[97;42m 200 [0m|      427.77µs |       127.0.0.1 |[97;44m GET     [0m "/images/json"
[GIN] 2024/10/17 - 06:25:46 |[97;42m 200 [0m|    3.9216542s |       127.0.0.1 |[97;44m GET     [0m "/images/mysql:8.0.39/json"
[GIN] 2024/10/17 - 06:25:46 |[97;42m 201 [0m|      455.79µs |       127.0.0.1 |[97;46m POST    [0m "/containers/create"
W1017 06:25:46.619401       1 container.go:226] user not set, will run as user defined in image
I1017 06:25:50.707259       1 main.go:66] start cleaning lingering objects...
I1017 06:25:50.818169       1 main.go:68] finished cleaning lingering objects...
I1017 06:25:51.925403       1 deploy.go:223] reverse proxy for 38259 to 3306
I1017 06:25:51.925440       1 tcpproxy.go:37] start reverse-proxy 0.0.0.0:38259->10.244.1.147:3306
I1017 06:25:56.969437       1 copy.go:30] copy archive to f279144bfdbe:/
I1017 06:25:56.969671       1 exec.go:59] exec kubedock-f279144bfdbe:[tar -xf - -C /]
[GIN] 2024/10/17 - 06:25:57 |[97;42m 200 [0m| 10.554630777s |       127.0.0.1 |[90;43m PUT     [0m "/containers/f279144bfdbe5dbfa6e90c1103675e65572dc335843cb4c99fcc5d4d2f1b1c16/archive?noOverwriteDirNonDir=false&path=%2F&copyUIDGID=false"
W1017 06:25:57.156293       1 containers.go:36] container f279144bfdbe5dbfa6e90c1103675e65572dc335843cb4c99fcc5d4d2f1b1c16 already running
[GIN] 2024/10/17 - 06:25:57 |[97;42m 204 [0m|     112.701µs |       127.0.0.1 |[97;46m POST    [0m "/containers/f279144bfdbe5dbfa6e90c1103675e65572dc335843cb4c99fcc5d4d2f1b1c16/start"
[GIN] 2024/10/17 - 06:25:57 |[97;42m 200 [0m|    3.135396ms |       127.0.0.1 |[97;44m GET     [0m "/containers/f279144bfdbe5dbfa6e90c1103675e65572dc335843cb4c99fcc5d4d2f1b1c16/json"
[GIN] 2024/10/17 - 06:25:57 |[97;42m 200 [0m|      44.687µs |       127.0.0.1 |[97;44m GET     [0m "/images/mysql:8.0.39/json"
[GIN] 2024/10/17 - 06:25:57 |[97;42m 200 [0m|        19.1µs |       127.0.0.1 |[97;44m GET     [0m "/version"
[GIN] 2024/10/17 - 06:25:57 |[97;42m 200 [0m|    3.106078ms |       127.0.0.1 |[97;44m GET     [0m "/containers/f279144bfdbe5dbfa6e90c1103675e65572dc335843cb4c99fcc5d4d2f1b1c16/json"
[GIN] 2024/10/17 - 06:25:57 |[97;42m 200 [0m|    3.112197ms |       127.0.0.1 |[97;44m GET     [0m "/containers/f279144bfdbe5dbfa6e90c1103675e65572dc335843cb4c99fcc5d4d2f1b1c16/json"
I1017 06:26:27.211428       1 tcpproxy.go:68] accepted connection for 0.0.0.0:38259 to 10.244.1.147:3306
W1017 06:26:27.222916       1 tcpproxy.go:99] error dialing 10.244.1.147:3306: dial tcp 10.244.1.147:3306: connect: connection refused (attempt: 0)
W1017 06:26:27.226730       1 tcpproxy.go:99] error dialing 10.244.1.147:3306: dial tcp 10.244.1.147:3306: connect: connection refused (attempt: 1)

See jenkins pipeline

def pod =
'''
apiVersion: v1
kind: Pod
metadata:
  name: java-docker-build-pod
spec:
  serviceAccountName: kubedock
  volumes:
    - name: socket
      hostPath:
        path: /var/run/docker.sock
  containers:
    - name: kubedock
      image: joyrex2001/kubedock:latest
      command:
        - /usr/local/bin/kubedock
      args:
        - server
        - --service-account=kubedock
        - --reverse-proxy
        - --timeout=2m
        - -v=3
        - --inspector

      tty: true

    - name: maven
      image: maven:3.9.9-sapmachine-17
      tty: true
      command: ["cat"]

'''

pipeline {
   agent {
       kubernetes {
           yaml pod
       }
   }
  options {
     skipStagesAfterUnstable()
   }
   triggers {
     pollSCM 'H/2 * * * *'
   }

   environment {
        TESTCONTAINERS_RYUK_DISABLED = "true"
        TESTCONTAINERS_CHECKS_DISABLE = "true"
        DOCKER_HOST = "tcp://127.0.0.1:2475"
   }

   stages {
         stage('Test') {
                    steps {
                        container('maven') {
                            script {
                                sh "mvn test"
                             }
                        }

                    }
              }
   }
}

Upvotes: 0

Views: 207

Answers (1)

Dev Fh
Dev Fh

Reputation: 644

Here is what worked - I've switched from kubedock to using Docker in Docker(DIND). e.g https://www.docker.com/blog/how-to-use-testcontainers-on-jenkins-ci/

In my spring boot application i've added Maven Wrapper files (mvnw and mvnw.cmd) - So that i no longer have to use maven container in jenkinsFile pipeline.

Here is is the final JenkinsFile.

def pod =
'''
apiVersion: v1
kind: Pod
metadata:
  labels:
    name: worker
spec:
  serviceAccountName: kubedock
  containers:
    - name: java17
      image: eclipse-temurin:17.0.9_9-jdk-jammy
      resources:
        requests:
          cpu: 1000m
          memory: 2048Mi
      imagePullPolicy: Always
      tty: true
      command:
        - cat
    - name: dind
      image: docker:dind
      imagePullPolicy: Always
      tty: true
      env:
        - name: DOCKER_TLS_CERTDIR
          value: ""
      securityContext:
        privileged: true

'''

pipeline {
   agent {
       kubernetes {
           yaml pod
       }
   }
  options {
     skipStagesAfterUnstable()
   }
   triggers {
     pollSCM 'H/2 * * * *'
   }

   environment {
        DOCKER_TLS_VERIFY = 0
        DOCKER_HOST = 'tcp://localhost:2375'
   }

   stages {
         stage('Test') {
                    steps {
                        container('java17') {
                            script {
                                sh "./mvnw test"
                             }
                        }

                    }
              }
   }
}

Upvotes: 0

Related Questions