ChromeHeadless hangs when running karma tests inside Jenkins agent docker image (with Kubernetes plugin)

Question

I have an angular application which runs with the default test suite (using karma to start a headless chrome). Jenkins runs with the kubernetes plugin to start up new instances of jenkins agents every time a job is submitted. (If this bug is fixed, I'll create an article in stackoverflow just about running headless chrome tests in docker containers, as this appears to be a common issue).

I based the build off of this (minus the protractor parts, as protractor is not used).

Everything builds and runs properly until the browser tests complete.

+ npm run test:vt --watch=false --progress=false --browsers=ChromeHeadless --headless --disable-gpu --window-size=800x600 --disable-dev-shm-usage --no-sandbox

> {my_company}/{my_project}docs@0.0.0-PLACEHOLDER test:vt /home/jenkins/agent/workspace/UI_{my_project}_PR-192
> ng test {my_project}

[33m04 09 2019 16:00:58.826:WARN [karma]: [39mNo captured browser, open http://localhost:9876/
[32m04 09 2019 16:00:58.832:INFO [karma-server]: [39mKarma v3.1.4 server started at http://0.0.0.0:9876/
[32m04 09 2019 16:00:58.832:INFO [launcher]: [39mLaunching browsers ChromeHeadlessNoSandbox with concurrency unlimited
[32m04 09 2019 16:00:58.839:INFO [launcher]: [39mStarting browser ChromeHeadless
[33m04 09 2019 16:01:16.236:WARN [karma]: [39mNo captured browser, open http://localhost:9876/
[32m04 09 2019 16:01:16.432:INFO [HeadlessChrome 76.0.3809 (Linux 0.0.0)]: [39mConnected on socket FcDILSSqmJFawfnrAAAA with id 26860248
HeadlessChrome 76.0.3809 (Linux 0.0.0): Executed 0 of 863 SUCCESS (0 secs / 0 secs)
...{bunches 'o lines}...
[1A[2KHeadlessChrome 76.0.3809 (Linux 0.0.0): Executed 796 of 863 (skipped 67) SUCCESS (0 secs / 32.499 secs)
[1A[2KHeadlessChrome 76.0.3809 (Linux 0.0.0): Executed 796 of 863 (skipped 67) SUCCESS (33.59 secs / 32.499 secs)
TOTAL: 796 SUCCESS
TOTAL: 796 SUCCESS

Then the terminal just appears to hang here. I let it sit for 24 hours with no change.

I put a monitor on the memory and cpu usage. The cpu drops to noise level (<10mcpu) as though it's waiting for an input. So, I added the flag "--password-store=basic" in order to prevent ChromeHeadless from hanging while waiting for input (I tried finding the karma github bug report that recommended this, but couldn't find it again). Unfortunately, this was unsuccessful.

Any ideas on how I can trouble shoot this, or fix it?

The docker image is built as such (with the last two RUN commands being used to setup the Chrome binaries):

FROM jenkins/jnlp-slave:3.29-1

ARG DOCKER_VERSION=18.06.1~ce~3-0~debian
ARG DC_VERSION=1.24.1

USER root
RUN apt-get update && \
    apt-get install -qq -y --no-install-recommends \
      apt-transport-https \
      ca-certificates \
      curl \
      gnupg2 \
      software-properties-common && \
    curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add - && \
    apt-key fingerprint 0EBFCD88 && \
    add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian stretch stable" && \
    add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian stretch stable" && \
    apt-get update && \
    apt-get install -qq -y --no-install-recommends docker-ce=${DOCKER_VERSION} && \
    curl -L https://github.com/docker/compose/releases/download/${DC_VERSION}/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose && \
    chmod +x /usr/local/bin/docker-compose && \
        curl -sL https://deb.nodesource.com/setup_12.x | bash - && \
        apt-get install nodejs && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
RUN apt-get install -qq -y --no-install-recommends nodejs npm

# Set the Chrome repo.
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
    && echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list
# Install Chrome.
RUN apt-get update && apt-get -y install google-chrome-stable

ENTRYPOINT ["jenkins-slave"]

My karama.conf is setup as such:

module.exports = function (config) {
  config.set({
    basePath: '',
    frameworks: ['jasmine', '@angular-devkit/build-angular'],
    plugins: [
      require('karma-jasmine'),
      require('karma-chrome-launcher'),
      require('karma-jasmine-html-reporter'),
      require('karma-coverage-istanbul-reporter'),
      require('@angular-devkit/build-angular/plugins/karma')
    ],
    client: {
      clearContext: false // leave Jasmine Spec Runner output visible in browser
    },
    coverageIstanbulReporter: {
      dir: require('path').join(__dirname, '../../coverage'),
      reports: ['html', 'lcovonly'],
      fixWebpackSourcePaths: true
    },
    reporters: ['progress', 'kjhtml'],
    port: 9876,
    colors: true,
    logLevel: config.LOG_INFO,
    autoWatch: true,
        browsers: ['ChromeHeadlessNoSandbox'],
        customLaunchers: {
            ChromeHeadlessNoSandbox: {
                base: 'ChromeHeadless',
                flags: [
            "--no-sandbox",
             // required to run without privileges in Docker
            "--disable-web-security",
            "--disable-gpu",
            "--remote-debugging-port=9222"
                ]
            }
        },
    singleRun: false,
        junitReporter:{
           outputDir:'test-reports',
           // results will be saved as $outputDir/$browserName.xml
                outputFile:'junit-report.xml',
           // if included, results will be saved as $outputDir/$browserName/$outputFile
                suite:'',
           // suite will become the package name attribute in xml testsuite element
                useBrowserName:false,
           // add browser name to report and classes names
                nameFormatter:undefined,
           // function (browser, result) to customize the name attribute in xml testcase element
                classNameFormatter:undefined,
           // function (browser, result) to customize the classname attribute in xml testcase element
                properties:{
           }   // key value pair of properties to add to the section of the report
        }
  });
};

The Jenkinsfile section that runs the test is:

  stage('Test') {
    echo "Running tests"
    sh 'npm run test:vt --watch=false --progress=false --browsers=ChromeHeadless  --headless --disable-gpu --window-size=800x600 --disable-dev-shm-usage --no-sandbox --password-store=basic'
  }

Answer 1

I use karma and CromeHeadless installed on the same machine as Jenkins (small projects and sites). With this configuratios I have never watch a Chrome instance that keeps working once the tests are finished. You could try to add this to your karma.conf:

    browsers: ['ChromeHeadless'],
    singleRun: true,
    restartOnFileChange: true

For example, my karma.conf looks like this (sending important lines only):

    customLaunchers: {
      ChromeHeadless: {
        base: 'Chrome',
        flags: [
          '--headless',
          '--disable-gpu',
          '--no-sandbox',
          '--remote-debugging-port=9222',
        ]
      }
    },
    browsers: ['ChromeHeadless'],
    singleRun: true,
    restartOnFileChange: true

The single run option means that "If true, Karma will start and capture all configured browsers, run tests and then exit with an exit code of 0 or 1 depending on whether all tests passed or any tests failed." (from the karma config file site)