While loop in bash script breaks systemd service

Question

I'm on Debian and I have a systemd service that calls a bash script.

The script contains an infinite while loop, as I need it to check something every X seconds infinitely.

The systemd service crashes once it hits the "while true; do" line.

The script runs fine if I execute it manually. Why doesn't systemd like it? What do I do?

Here are the service and the script. As I've indicated, an echo statement before the "while true; do" prints. The echo statement after the "while true; do" line does not print.

/etc/systemd/system/stream.service:

[Service]
WorkingDirectory=/home/pi/
ExecStart=/home/pi/joi_main.sh
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=stream_service
User=pi
Group=pi
Environment=NODE_ENV=production
[Install]
WantedBy=multi-user.target

/home/pi/joi_main.sh:

#!/bin/bash -e

today=`/bin/date '+%Y_%m_%d__%H_%M_%S'`
exec 2> "/home/pi/stream_logs/$today.$RANDOM.log"
exec 1>&2

#Wait 120s for system to finish booting
sleep 120

#Initial config
export AUDIODEV=mic_mono
export AUDIODRIVER=alsa
sudo sysctl fs.pipe-max-size=1048576

echo "This line prints"

# Check if video buffer is full every minute. if full, the stream needs to restart
while true; do
    echo "This line doesn't"
    if grep "100% full" /home/pi/video_buffer_usage.txt; then
        echo "Buffer is full!"
    # Kill existing processes
        pkill -f “raspivid|rec|buffer|ffmpeg”
    # Wait 10s
         sleep 10
         ./joi_stream.sh &
    fi
    sleep 60
done

Journalctl seems completely unhelpful, but here it is. No errors. Why is "session closed"?

Mar 31 02:13:41 raspberrypi sudo[1369]: pi : TTY=unknown ; PWD=/home/pi ; USER=root ; COMMAND=/sbin/sysctl fs.pipe-max-size=1048576
Mar 31 02:13:41 raspberrypi sudo[1369]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 31 02:13:41 raspberrypi sudo[1369]: pam_unix(sudo:session): session closed for user root

(Please don't tell me to start yet another systemd service for just this while loop. I want it to be part of this main script because it needs to run after everything else, and if I turn off the main service I don't want the while loop running either, so maintaining two systemd services would only add troube.)

Answer 1

The contents of ./joi_stream.sh were not shared, but here's a problem I see with your systemd solution. It doesn't directly explain your behavior, but may be related:

In your systemd configuration, you redirect both STDOUT and STDERR to syslog, but in your script, you redirect STDERR (file descriptor "2") to a file, and redirect STDOUT (file descriptor "1') to STDERR.

exec 2> "/home/pi/stream_logs/$today.$RANDOM.log"
exec 1>&2

If your ./joi_stream.sh expected your redirection of these file descriptors to another file to work, it may not. If the file is just for logging, I would get rid of these lines and let the systemd journal handle that-- it will tag the logs with your unit you can review your logs specifically:

 journalctl -u your-unit-name.service

Also, in systemd, you wouldn't normally put in a sleep to wait until the systemd has booted. Instead, you would use a .timer unit.

The .timer file would instruct to run the main logic every minute, so the "while" loop would not be required. The timer unit would contain directives like:

# Run for the first time 2 minutes after boot # and every minute after that OnBootSec=120 OnUnitActiveSec=60

It would be timer unit which is enabled to start on boot. Timer files can be super-simple. Just create a .timer file in /etc/systemd/system and give it the same name as the service file you want it to activate:

[Unit]
Description=Runs my service every minute

[Timer]
# Run for the first time 2 minutes after boot 
# and every minute after that
OnBootSec=120 
OnUnitActiveSec=60

[Install]
WantedBy=timers.target

To start and test your timer immediately, run:

sudo systemctl start my-service.timer

You can review the status of timers with:

sudo systemctl list-timers

The systemd solution is more robust than the rc.local solution. If your rc.local solution dies for any reason, it will not restart. However, if your script dies will run under systemd, the timer will still run it again a minute later.

Answer 2

FYI, everything works if I call /home/pi/joi_main.sh from /etc/rc.local instead of using a systemd service. I'll use rc.local and kill the service.