Getting error 'Unable to load python3' while dockerizing Scala Application which runs python code using scalapy

Question

I'm trying to dockerize my scala application - in which I've used Scalapy to run python code. My scala version (2.13.1), sbt version (1.2.8).

Main Class in Scala:

import me.shadaj.scalapy.interpreter.CPythonInterpreter
import me.shadaj.scalapy.py

object Main extends App {
  val pythonCode =
    """
      |import os
      |
      |myFile = os.path.join("src", "main", "resources", "myFile.txt")
      |
      |with open(myFile, "w") as file:
      |    file.write("Hello World!")
      |""".stripMargin

  py.local {
    CPythonInterpreter.execManyLines(pythonCode)
  }
}

I have added a libraryDependency in project/plugins.sbt:

libraryDependencies += "ai.kien" %% "python-native-libs" % "0.2.2"

Here is my build.sbt:

import com.typesafe.sbt.packager.docker.{DockerChmodType, DockerPermissionStrategy, ExecCmd}
import ai.kien.python.Python

lazy val root = (project in file("."))
  .settings(
    name := "my-project"
  )

ThisBuild / version := "0.1.0-SNAPSHOT"

ThisBuild / scalaVersion := "2.13.1"

ThisBuild / scapegoatVersion := "1.4.0"

enablePlugins(JavaAppPackaging, DockerPlugin)
packageName in Docker:= "my-project"

version in Docker:= "0.0.3"
dockerExposedPorts:= Seq(8083)

dockerChmodType := DockerChmodType.UserGroupWriteExecute
dockerPermissionStrategy := DockerPermissionStrategy.CopyChown
dockerAdditionalPermissions += (DockerChmodType.UserGroupPlusExecute, "/var/run/")
daemonUserUid in Docker := None
daemonUser in Docker    := "root"

dockerCommands ++= Seq(
  ExecCmd("RUN", "apt-get", "update"),
  ExecCmd("RUN", "apt-get", "install", "-y", "python3")
)

fork:= true

lazy val python = Python("/usr/bin/python3")

lazy val javaOpts = python.scalapyProperties.get.map {
  case (k, v) => s"""-D$k=$v"""
}.toSeq

javaOptions ++= javaOpts

libraryDependencies ++= Seq(
  "me.shadaj" %% "scalapy-core" % "0.5.2"
)

Now the program is running fine if I run it in intellij idea with sbt-shell - but if I run with the run button it gives:

Exception in thread "main" java.lang.UnsatisfiedLinkError: Unable to load library 'python3':
libpython3.so: cannot open shared object file: No such file or directory
libpython3.so: cannot open shared object file: No such file or directory
Native library (linux-x86-64/libpython3.so) not found in resource path (/home/zaryab/IdeaProjects/My_Project/target/scala-2.13/classes:/home/zaryab/.ivy2/cache/aopalliance/aopalliance/jars/aopalliance-1.0.jar)

At this point its fine because its working with sbt-shell. But now I want to dockerize it. For this I had added Some Docker Commands in build.sbt - and I'm making my application image using sbt-native-packager.

The DockerFile which is automatically created by sbt-native-packager:

FROM openjdk:8
WORKDIR /opt/docker
COPY --chown=root:root opt /opt
EXPOSE 8083
USER root
ENTRYPOINT ["/opt/docker/bin/notary-scraping"]
CMD []
RUN ["apt-get", "update"]
RUN ["apt-get", "install", "-y", "python3"]

But when I run the container of this image, it is giving same error which I get when running with run button.

I've run different commands to check 'python3' is in container or not:

First I run 'docker exec -it my_container /bin/bash'

Then 'which python3' - it prints '/usr/bin/python3'

Python3 is in container but still getting 'unable to load python3'. What should I do now to properly dockerize it?

Answer 1

You are using base Image openjdk:8 which has python3 installed already, so it will look for libpython3 in it's own installation rather than the path you have given in build.sbt ("/usr/bin/python3").

Now what you need to do is to change the base image like amazoncorretto:11-alpine3.18-jdk and then install python3 and other required modules via commands.

Your build.sbt will look like.

import com.typesafe.sbt.packager.docker.{DockerChmodType, DockerPermissionStrategy, ExecCmd}
import ai.kien.python.Python

lazy val root = (project in file("."))
  .settings(
    name := "my-project"
  )

ThisBuild / version := "0.1.0-SNAPSHOT"

ThisBuild / scalaVersion := "2.13.1"

ThisBuild / scapegoatVersion := "1.4.0"

enablePlugins(JavaAppPackaging, DockerPlugin)
packageName in Docker:= "my-project"

version in Docker:= "0.0.3"
dockerExposedPorts:= Seq(8083)

dockerChmodType := DockerChmodType.UserGroupWriteExecute
dockerPermissionStrategy := DockerPermissionStrategy.CopyChown
dockerAdditionalPermissions += (DockerChmodType.UserGroupPlusExecute, "/var/run/")
daemonUserUid in Docker := None
daemonUser in Docker    := "root"

dockerBaseImage := "amazoncorretto:11-alpine3.18-jdk"

dockerCommands ++= Seq(
  ExecCmd("RUN", "apt-get", "update"),
  ExecCmd("RUN", "apt-get", "install", "-y", "python3")
)

fork:= true

lazy val python = Python("/usr/bin/python3")

lazy val javaOpts = python.scalapyProperties.get.map {
  case (k, v) => s"""-D$k=$v"""
}.toSeq

javaOptions ++= javaOpts

libraryDependencies ++= Seq(
  "me.shadaj" %% "scalapy-core" % "0.5.2"
)


dockerCommands ++= Seq(
  ExecCmd("RUN", "apk", "add", "--no-cache", "bash"),
  ExecCmd("RUN", "apk", "add", "--no-cache", "python3"),
  ExecCmd("RUN", "apk", "add", "--no-cache", "py3-pip")
//other modules if required
)

also you need to specify path in python code is (start with "/"),

  |myFile = os.path.join("/src/main/resources", "myFile.txt")

This should work for you.