Having trouble changing from python3.12 to python3.11 within a dockerfile with ros2:devel base

Question

Hello I am using the osrf/ros2:devel base image for my dockerfile which includes python3.12, but due to some hardware API constraints I need to use python3.11. In my dockerfile removing python3.12 is easy enough but getting 3.11 is becoming a real pain.

I have tried:

RUN apt-get install -y python3.11

and

RUN apt install -y wget libffi-dev gcc build-essential curl tcl-dev tk-dev uuid-dev lzma-dev liblzma-dev libssl-dev libsqlite3-dev

RUN wget https://www.python.org/ftp/python/3.11.9/Python-3.11.9.tgz
RUN tar -zxvf Python-3.11.9.tgz
RUN cd Python-3.11.9 && ./configure --prefix=/opt/python3.11 && make && make install

RUN rm Python-3.11.9.tgz
RUN rm -r Python-3.11.9/

RUN ln -s /opt/python3.11/python3.11 /usr/bin/python#python3-pip

Which seem to be the most common answers but both end up exit code: 100. Any clue how to get this done?