CODEWITHSUNDEEP
pythoncythoncythonize
Asym
Asym

Reputation: 1938

Cython unable to import and execute a function: Storing unsafe C derivative of temporary Python reference

I'm new to Cython and just trying to experiment with a simple script. I'm unable to import a function from a module and use it in another file, but if I declare the function in the same file it works just fine. What could be the problem here. Am I missing something?

This is the original code snippet (test_cy.pyx):

# test_cy.pyx

import json

cpdef char* say_hello():
  message = json.dumps({"message": "Hello world"})
  return message.encode()

cdef char* fn(int n):
  cdef char* hello = say_hello()
  return hello

def test():
  cdef char* n = fn(1000)
  print(n)

I can run it like (After compiling):

>>> import test_cy
>>> test_cy.test()
b'Hello world'

However, if I move the say_hello function to another file (utils.pyx):

# utils.pyx

import json

cpdef char* say_hello():
  message = json.dumps({"message": "Hello world"})
  return message.encode()

... And import it in my original test_cy.pyx file like so:

# test_cy.pyx

from utils import say_hello

cdef char* fn(int n):
  cdef char* hello = say_hello()
  return hello

def test():
  cdef char* n = fn(1000)
  print(n)

I'm unable to get it to compile as I get the following error:

>>> python3 setup.py build_ext --inplace
...
Error compiling Cython file:
------------------------------------------------------------
...
from utils import say_hello

cdef char* fn(int n):
  cdef char* hello = say_hello()
      ^
------------------------------------------------------------

test_cy.pyx:4:7: Storing unsafe C derivative of temporary Python reference
Traceback (most recent call last):
  File "setup.py", line 11, in <module>
    sources=["utils.pyx"]
  File "/Users/asim/.pyenv/versions/3.6.9/lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 1101, in cythonize
    cythonize_one(*args)
  File "/Users/asim/.pyenv/versions/3.6.9/lib/python3.6/site-packages/Cython/Build/Dependencies.py", line 1224, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: test_cy.pyx

Here's my setup.py file:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize

setup(
  ext_modules = cythonize([
    Extension("test_cy",
              sources=["test_cy.pyx"]
    ),
    Extension("utils",
              sources=["utils.pyx"]
    )
  ])
)

Please help me as something so simple like importing a function from another module shouldn't be this complicated. I might be missing something trivial. Thanks!

Upvotes: 2

Views: 458

Answers (1)

DavidW
DavidW

Reputation: 30888

cpdef char* say_hello():
  message = json.dumps({"message": "Hello world"})
  return message.encode()

This is a disaster waiting to happen (although Cython doesn't seem to be able to generate a warning). The data that your char* points to is held by the result of message.encode() - a temporary Python object that is destroyed right after it's created. Therefore your char* is instantly invalid.

Think about this in C terms: to return a char* someone has to own the char* and you should know exactly who that owner is. There isn't an easy, sensible way to returning a char* without taking responsibility for allocating/deallocating yourself. Unless you're prepared to do that and understand how to do that I suggest you delete all the char* from your code and just return Python objects.

The error you're getting is specifically because Cython uses cimport to know about Cython functions.

from utils cimport say_hello

Without cimport it believes that say_hello is just an ordinary Python function returning an ordinary Python object. However, even with this change your code is still invalid - you've just moved the error somewhere Cython can't see it.

Upvotes: 1

Related Questions