Why cross compiled program could crash this way?

Question

I cross compiled program for Raspberry Pi Zero but for some reason I get a crash. Program is very simple, just prints string to std::cout:

#include <iostream>

int main(int argc, char const *argv[])
{
    std::cout << "Hello!" << std::endl;
    return 0;
}

I build program using Docker and CMake:

FROM debian:bookworm

ENV GNU_HOST=arm-linux-gnueabihf
ENV C_COMPILER_ARM_LINUX=$GNU_HOST-gcc
ENV CXX_COMPILER_ARM_LINUX=$GNU_HOST-g++

ENV CROSS_INSTALL_PREFIX=/usr/$GNU_HOST
ENV CROSS_TOOLCHAIN=/arm.toolchain.cmake

RUN echo "set(CMAKE_SYSTEM_NAME Linux)" >> $CROSS_TOOLCHAIN && \
  echo "set(CMAKE_SYSTEM_PROCESSOR arm)" >> $CROSS_TOOLCHAIN && \
  echo "set(CMAKE_C_COMPILER /usr/bin/$C_COMPILER_ARM_LINUX)" >> $CROSS_TOOLCHAIN && \
  echo "set(CMAKE_CXX_COMPILER /usr/bin/$CXX_COMPILER_ARM_LINUX)" >> $CROSS_TOOLCHAIN && \
  echo "set(CMAKE_PREFIX_PATH $CROSS_INSTALL_PREFIX)" >> $CROSS_TOOLCHAIN

RUN apt-get update && \
  apt-get --no-install-recommends install -y autoconf \
    automake \
    build-essential \
    ca-certificates \
    curl \
    # C/C++ cross compilers
    gcc-$GNU_HOST \
    g++-$GNU_HOST \
    git \
    gnupg \
    libssl-dev \
    openssh-client \
    pkg-config \
    software-properties-common \
    wget \
    cmake && \
  rm -rf /var/lib/apt/lists/*

Here is C++ compiler version:

/usr/bin/arm-linux-gnueabihf-g++ --version
arm-linux-gnueabihf-g++ (Debian 12.2.0-14) 12.2.0

Here is information about binary file.

pi@raspberrypi:~$ file camera-bot
camera-bot: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=f45663f061217dc844ca45f314e2af084df83869, for GNU/Linux 3.2.0, with debug_info, not stripped

pi@raspberrypi:~$ objdump -j .interp -s ./camera-bot

./camera-bot:     file format elf32-littlearm

Contents of section .interp:
 0154 2f6c6962 2f6c642d 6c696e75 782d6172  /lib/ld-linux-ar
 0164 6d68662e 736f2e33 00                 mhf.so.3.       
pi@raspberrypi:~$ 

pi@raspberrypi:~$ ldd camera-bot
        /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so (0xb6ee0000)
        libstdc++.so.6 => /lib/arm-linux-gnueabihf/libstdc++.so.6 (0xb6d00000)
        libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0xb6cc0000)
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb6b47000)
        libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0xb6b00000)
        /lib/ld-linux-armhf.so.3 (0xb6f23000)

pi@raspberrypi:~$ g++ --version
g++ (Raspbian 12.2.0-14+rpi1) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

pi@raspberrypi:~$ gdb camera-bot
GNU gdb (Raspbian 13.1-3) 13.1

Reading symbols from camera-bot...
(gdb) b main
Breakpoint 1 at 0x892: file /workspaces/RaspberryCamera/src/main.cpp, line 19.
(gdb) run
Starting program: /home/pi/camera-bot 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x0044f952 in ?? ()
(gdb) bt
#0  0x0044f952 in ?? ()
#1  0x0040079e in _start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Update #1

Even if I do static linking for the C++ standard library, I get crash:

Reading symbols from camera-bot...
(gdb) b main
Breakpoint 1 at 0x889a: file /workspaces/RaspberryCamera/src/main.cpp, line 19.
(gdb) run
Starting program: /home/pi/camera-bot 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x0045795c in bool std::has_facet<std::collate<wchar_t> >(std::locale const&) ()
(gdb) bt
#0  0x0045795c in bool std::has_facet<std::collate<wchar_t> >(std::locale const&) ()
#1  0xbefff6e2 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Answer 1

I found that Docker image debian:bookworm that I used for the cross compilation has /usr/bin/arm-linux-gnueabihf compiler that was built for ARMv8 but Raspberry Pi Zero requires ARMv6. There are some other ARM related options that are different for device compiler and Docker image compiler. I guess that this was the reason of crash.

So I changed Docker image to ghcr.io/tttapa/docker-arm-cross-toolchain:armv6-rpi-linux-gnueabihf created by tttapa and also enabled static linking for the C++ standard library in my CMakeLists.txt.

# Dockerfile.dev
FROM ghcr.io/tttapa/docker-arm-cross-toolchain:armv6-rpi-linux-gnueabihf

USER root

RUN apt-get update && \
  apt-get --no-install-recommends install -y autoconf \
    automake \
    build-essential \
    ca-certificates \
    curl \
    gnupg \
    libssl-dev \
    openssh-client \
    pkg-config \
    software-properties-common \
    wget \
    cmake && \
  rm -rf /var/lib/apt/lists/*

After this steps I can run cross compiled binary on my Raspberry Pi Zero without any crash.

P.S. I don't know if this should be accepted answer because I did not provided full version information (including build options) for the GCC from Docker image and from device in my question. Maybe this answer will save some time for someone who will faced the same crash.