CODEWITHSUNDEEP
dockerwaittomcat9sigtermbash-trap
IPlato
IPlato

Reputation: 311

Unable to trap signals in docker entrypoint script

I have a docker entrypoint script that is supposed to trap signals sent to processes in the container. The main application is tomcat - java process embedded in the docker-entrypoint.sh which is passed to dumb-init. The process mapping in the container looks like so:

root@mycontainer:/usr/local/tomcat/webapps/datarouter-example# ps -ef
    UID        PID  PPID  C STIME TTY          TIME CMD
    root         1     0  0 05:21 ?        00:00:00 dumb-init -- /docker-entrypoint.sh
    root         6     1  0 05:21 ?        00:00:00 bash /docker-entrypoint.sh
    root        14     6  1 05:21 ?        00:08:57 /jdk-13.0.1/bin/java -Djava.util.logging.config.file=....

Dockerfile:

FROM maven:3.6.3-jdk-13 as maven_builder

WORKDIR /app
COPY . /app
RUN ["mvn","clean","install","-T","2C","-DskipTests=true"]


FROM tomcat:9.0.31-jdk13-openjdk-buster

ARG dumbInitVersion='1.2.2'

# install dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
        sudo \
        wget \
        dpkg-dev \
    && rm -rf /var/lib/apt/lists/*

ARG webappDir
WORKDIR $CATALINA_HOME/webapps/$webappDir/
ENV CATALINA_OUT $CATALINA_HOME/logs/catalina.out
ENV CATALINA_PID $CATALINA_HOME/catalina.pid
COPY docker-entrypoint.sh /
COPY --from=maven_builder /app/target/datarouter-example $CATALINA_HOME/webapps/$webappDir/

RUN wget https://github.com/Yelp/dumb-init/releases/download/v"$dumbInitVersion"/dumb-init_"$dumbInitVersion"_amd64.deb; \
    dpkg -i dumb-init_*.deb; \
    rm dumb-init_*.deb

EXPOSE 8080
ENTRYPOINT ["dumb-init", "--", "/docker-entrypoint.sh"]

And this is the simplified docker-entrypoint.sh:

        #!/usr/bin/env bash

        start(){
                echo "$(date +'%F %T,%3N') Starting tomcat..." >> "$CATALINA_OUT"
                catalina.sh start && tail -f "$CATALINA_OUT"
        }

        stop(){
                echo "$(date +'%F %T,%3N') starting stop" >> "$CATALINA_OUT"

   # some fancy shutdown logic, e.g. calling our application's shutdown endpoint to cleanup (does not stop tomcat)

                echo "$(date +'%F %T,%3N') Stopping tomcat..." >> "$CATALINA_OUT"
                catalina.sh stop  >> "$CATALINA_OUT"

                wait $(cat "${CATALINA_PID}")

                tailId=$(pgrep tail)
                if [[ -n "$tailId" ]]; then
                        kill "$tailId"
                fi
                exit
        }

        trap stop SIGINT SIGQUIT SIGHUP SIGTERM

    start

Now the issue. Every time a docker stop command or a kill 1 from inside the container is issued, the java application would be sent a signal and start to shutdown first before the stop function even gets to the first line. I am sure because of the Terminated log:

Terminated
++ stop
+++ date '+%F %T,%3N'
++ echo '2020-03-13 14:34:54,480 starting stop'

and in catalina.out:

2020-03-12 04:08:27.079 INFO [Thread-13] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-apr-8080"]
2020-03-12 04:08:27,079 starting stop
2020-03-12 04:08:27.085 INFO [Thread-13] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-apr-8081"]
2020-03-12 04:08:27.090 INFO [Thread-13] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["https-openssl-apr-8443"]
2020-03-12 04:08:27.096 INFO [Thread-13] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
......

I'm really confused of why this is happening and how to fix this so that catalina.sh stop gets to be executed.

Upvotes: 6

Views: 7006

Answers (1)

IPlato
IPlato

Reputation: 311

After trying a few changes, the following script traps the SIGTERM and executes the expected steps.

Changes from original docker-entrypoint, Dockerfile:

  • Switched from dumb-init to tini, just to try it out.
  • Switched from catalina start to catalin run and putting it in background myself and waiting on it
  • Removed the tail process

I think the tail is what was causing the original described behaviour. The bigest downside of the new entrypoint script is that I lost the container logs when running docker logs .... Since we bind mount the logs directory, we're still able to get the logs, but will be investigating further to get back docker logs ...

I'm still in search for a nicer solution if anyone has an advice or other solutions.

#!/usr/bin/env bash

trap stop SIGTERM SIGINT SIGQUIT SIGHUP ERR

start(){
   local catalinaPid

   touch "${CATALINA_PID}"
   catalina.sh run >> ${CATALINA_OUT} 2>&1 &
   catalinaPid=$!
   echo "$catalinaPid" > "${CATALINA_PID}"

   wait "$catalinaPid"
}

stop(){
   echo "$(date +'%F %T,%3N') starting stop" >> "$CATALINA_OUT"

   # some fancy shutdown logic, e.g. calling our application's shutdown endpoint to cleanup (does not stop tomcat)

   echo "$(date +'%F %T,%3N') Stopping tomcat..." >> "$CATALINA_OUT"
   catalina.sh stop 20 >> "$CATALINA_OUT"

   wait $(cat "${CATALINA_PID}")

   exit
}

start

Upvotes: 6

Related Questions