Process image stream in Lua by importing image pointer from C++

Question

I am working on a deep learning task in Lua and I need to import an image stream into Lua from C++. The C++ script is reading data from a shared memory and I have the pointer to the images in C++. I would like to get this pointer in Lua and give the image stream to my Neural Network.
Since Lua does not use pointers and as far as my research the way to go is to wrap this pointer into userdata and write accessor methods. As explained in this User Data With Pointer Example.
My problem is that I need to feed the neural network with raw pixel data and so far the material available online suggests that processing on data should be done in C++ and there is no way to get that object in Lua directly.
What should be the correct approach here? How can I get object values in Lua from pointer in C++?

Answer 1

Well, you have to make the data accessible to Lua in some way.

I can think of four general approaches:

1. converting the data

   • writing C(++) code that traverses the data to create a Lua data structure made of tables & primitive values (booleans, numbers, strings)
   • converting the data to an interchange format (JSON, CBOR, … or in the case of image data: PNM / PAM) for which you have code on both sides or can trivially write it

   (I won't say any more on these options here.)

2. passing it in as userdata and writing accessor functions, so you can write the conversion code in Lua instead of C(++)

3. (only for pointer-free data, and images usually are that) passing the raw data area to Lua as a string

4. (LuaJIT only) using ffi.cdef to declare C structs that match the in-memory format for free access & conversion (int, float, doesn't matter – can also work with pointers)

(2) is basically just (1a) with most of the code shifted to the Lua side. (3) is the same with free accessor code (string.byte( data, i[, j]) and a bunch of others, newly added in 5.3: string.unpack(fmt, data[, startpos] ) ).

Before deciding on one of those approaches, look at what format the other side wants.

Maybe your data is already in that format, in which case you would only have to route through the pointer in some way. (Maybe there's a function accepting a Lua table, but maybe there's also one that accepts a string or pointer to raw image data.) If this applies, use (light) userdata or strings for communication. (This will probably be the fastest, and if this way of passing the data doesn't exist yet consider asking upstream whether they can add it.)

Failing that, you'll have to actually touch the data. The string version is potentially the easiest, but also likely to be the slowest. (If the data is one contiguous block of bytes, you can simply lua_pushlstring( L, ptr, len_in_bytes ) to get everything into Lua, but that creates a copy of the "string" internal to the Lua state which might be relatively slow.) After that, you can use the standard string handling functionality of Lua.

If you're using LuaJIT, it's probably a very good idea to use the ffi library. A very brief outline of what you may need (using a linked list as example):

ffi = require "ffi"
-- sample structure, yours will differ
ffi.cdef [[
    typedef struct Node_s { struct Node_s *next; int value; } Node;
]]

-- in your case you will get a pointer from C, which you can then
-- ffi.cast( "Node*", ptr )
-- after that, you can access the fields (ptr.next, ptr.value etc.)
-- we'll just use some dummy sample data here:
do
    local p = ffi.new( "Node" )
    local q = ffi.new( "Node" )
    p.next = q
    p.value, p.next.value = 23, 42
    ptr = tonumber( tostring( p ):match( ": (0x%x*)" ) )
end

data = ffi.cast( "Node*", ptr )
print( "first value:", data.value )
--> first value:    23
print( "second value:", data.next.value )
--> second value:   42
print( "third value:", data.next.next.value )
--> Segmentation fault   [so be just as careful as you're in C(++)!]

Lastly, if you're on vanilla Lua, you may want to use (full) userdata and manually written accessors to avoid the speed loss from data copying. (The full userdata is required so you can set a metatable to get the accessors, but it can simply be a pointer to your externally allocated data.) The book "Programming in Lua" has good information on that (first edition is online, direct link to the relevant chapter.)