Verifying compatibility in compiling extension types, and using them with cdef

Question

Standing questions:

• Why do other errors in Cython compilation point to the specific line of error, while this doesn't?

Prior to update:

Due to difficulty in compiling extension types, as referenced in the 'won't compile' link below, it was thought that the AssertionError was related to the extension types (after their instantiation was transferred to the main pyx file).

Someone from this forum post said that “If you actually want extension types, and want to declare their types in your code, you need to declare their C attributes in a .pxd file.”

If that is true in this case, the problem is that the pyx files won’t compile for me when using pxd files. There is no error when compiling the extension type in one pyx file, but then I receive the error extTypeName is not a type identifier when the extension type is used after its import.

I was told in the comments that extension types can be defined in the pyx file with the main function using them. When doing so, I receive an error I don’t know how to trace (AssertionError: unexpected type int and base type tuple object for indexing). The full traceback is posted below.

---

Difficult to trace error

Traceback when trying to compile with the extension types declared in the same pyx file (which I’m not sure is the source of the issue):

Traceback (most recent call last):
  File "setup.py", line 37, in <module>
    ext_modules = [Extension("HintToCRdict", ["HintToCRdict.pyx"])]
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/Library/Python/2.7/site-packages/Cython/Distutils/build_ext.py", line 164, in run
    _build_ext.build_ext.run(self)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 337, in run
    self.build_extensions()
  File "/Library/Python/2.7/site-packages/Cython/Distutils/build_ext.py", line 171, in build_extensions
    ext.sources = self.cython_sources(ext.sources, ext)
  File "/Library/Python/2.7/site-packages/Cython/Distutils/build_ext.py", line 324, in cython_sources
    full_module_name=module_name)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Main.py", line 688, in compile
    return compile_single(source, options, full_module_name)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Main.py", line 638, in compile_single
    return run_pipeline(source, options, full_module_name)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Main.py", line 495, in run_pipeline
    err, enddata = Pipeline.run_pipeline(pipeline, source)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Pipeline.py", line 365, in run_pipeline
    data = phase(data)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Pipeline.py", line 53, in generate_pyx_code_stage
    module_node.process_implementation(options, result)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/ModuleNode.py", line 131, in process_implementation
    self.generate_c_code(env, options, result)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/ModuleNode.py", line 359, in generate_c_code
    self.body.generate_function_definitions(env, code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 435, in generate_function_definitions
    stat.generate_function_definitions(env, code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 1944, in generate_function_definitions
    self.generate_function_body(env, code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 1698, in generate_function_body
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 5851, in generate_execution_code
    if_clause.generate_execution_code(code, end_label, is_last=i == last)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 5894, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/UtilNodes.py", line 321, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 6402, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/UtilNodes.py", line 321, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 6402, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 5851, in generate_execution_code
    if_clause.generate_execution_code(code, end_label, is_last=i == last)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 5894, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/UtilNodes.py", line 89, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 6015, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 6192, in generate_execution_code
    self.body.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 441, in generate_execution_code
    stat.generate_execution_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 4786, in generate_execution_code
    self.generate_rhs_evaluation_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/Nodes.py", line 5073, in generate_rhs_evaluation_code
    self.rhs.generate_evaluation_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/ExprNodes.py", line 699, in generate_evaluation_code
    self.generate_subexpr_evaluation_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/ExprNodes.py", line 714, in generate_subexpr_evaluation_code
    node.generate_evaluation_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/ExprNodes.py", line 699, in generate_evaluation_code
    self.generate_subexpr_evaluation_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/ExprNodes.py", line 714, in generate_subexpr_evaluation_code
    node.generate_evaluation_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/ExprNodes.py", line 705, in generate_evaluation_code
    self.generate_result_code(code)
  File "/Library/Python/2.7/site-packages/Cython/Compiler/ExprNodes.py", line 3737, in generate_result_code
    self.type, self.base.type)
AssertionError: unexpected type int and base type tuple object for indexing

Answer 1

The extension types were not an issue in compilation (at least when declared within the same pyx file as the main cdef function); The error is arising elsewhere.

Resolved:

1. The AssertionError was caused by simple index access to this tuple.

Where a tuple was declared as:

cdef:
    tuple curIRs

…and set in nested loops as:

for deriv1 in xrange(len(L1)):
    for deriv2 in xrange(len(L2)):
        curIRs = (deriv2, deriv1)

…and tuple indexes are accessed:

d1[ix] = (curIRs[0], curIRs[1])

It was an apparently innocuous oversight that curIRs wasn’t just stored instead of essentially copying it by index access. However, this is where the error occurred for an unknown reason:

When (curIRs[0], curIRs[1]) is replaced with curIRs, the AssertionError ceases.

2. As stated in the comments, the main function in the pyx file can't be defined with cdef, since it is invoked in a python script.

---

Pierre de Buyl's suggestion in the comments was very helpful: 'remove cdef declarations of potential problem variables until the error is resolved.' i.e. where there is likely no error in Python, an incompatibility may be uncovered in Cython

Answer 2

A reply with what I can take from your message.

1. The use of Python objects in Cython is possible but limited as soon as you cdef some parts. Try by "un"-cdefing the dict.
2. The error you have suggests that you have an expression that is x[y] where x is a tuple and y an int. In principle it should work all right (indexing a tuple with an int shoud be ok). So you can look at square brackets in the Cython to find out where it comes from.
3. You mentioned having removed the pxd file. Have you then move the full declaration of the extension class/other type definitions to the pyx file?