CODEWITHSUNDEEP
gccdlldatabrickscython
reda igbaria
reda igbaria

Reputation: 1454

databricks python dynamic linking

I'm trying to compile a python code using cython and run it as an executable on databricks runtime.

I'm using the following commands for compilation:

cython --embed -3 main.py
gcc  -I/usr/include/python3.11 -ldl  -lm  -lpython3.11 main.c

it produces an executable, but the problem that it is using the system python version and not the databricks provdided one, which is at path /databricks/python3/bin/python

I'm using databricksruntime/python:15.4-LTS docker image from databricks, I want to link the databricks python so I can connect my code to spark. Tried searching for a python dll in / couldn't find other than the python system version.

Also, here is the result of running sysconfig.get_config_var('LIBDIR') from databricks python:

/usr/lib/x86_64-linux-gnu

and the result of sysconfig.get_paths():

{
  'stdlib': '/usr/lib/python3.11', 
  'platstdlib': '/databricks/python3/lib/python3.11', 
  'purelib': '/databricks/python3/lib/python3.11/site-packages', 
  'platlib': '/databricks/python3/lib/python3.11/site-packages',
  'include': '/usr/include/python3.11',
  'platinclude': '/usr/include/python3.11',
  'scripts': '/databricks/python3/bin',
  'data': '/databricks/python3'
}

Upvotes: 0

Views: 22

Answers (0)

Related Questions