Sockets between Raspbian-Windows openCV videostream

Question

This is my first time asking here but I've been looking around for a solution long time and I've been unable to find it, so I will try to solve this and, if more people have the same problem, help those with this thread. Hope it is not duplicated exactly!

I have a server in Windows7, C++, Visual Studio 2015, where I have some routines running in openCV. I'm taking some of those tasks out from my computer to a remote Raspberry Pi 3, so I programmed a client in Python 3 which uses sockets as well.

Both the server (C++) and the client (Python) are properly working when sending strings such as "Hello World!" and that stuff. The problem came when I started to send binary data.

I have an image in openCV (client-python side) and I used imencode function:

data = cv2.imencode(".jpg", image, [int(cv2.IMWRITE_JPEG_QUALITY), 90])[1].tostring()

Then I have a buffer where I'm storing the information to be sent, so I add that data to the buffer:

buffer.append(data)

Then, when the buffer is populated, I sent the first image through my socket client:

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect("some_address and some port")
sock.send(data)
sock.close()

In the meanwhile, in the server side (C++), I initialized the windows socket, solved the address, created a listening socket, linking the listening socket to the server address and the wait for the client to connect to the listening socket. When the client connects, the listening socket is closed and I call to start receiving method:

do {
    iResult = recv(ClientSocket, recvbuf, recvbuflen, 0);
    if (iResult > 0) {
        totalBytes += iResult;
        std::cout << recvbuf << std::endl;
        ss << recvbuf;

        std::cout << ss.str() << std::endl;
        std::cout << "Bytes received: " << iResult << std::endl;

        // Echo the buffer back to the sender
        iSendResult = send(ClientSocket, recvbuf, iResult, 0);
        if (iSendResult == SOCKET_ERROR) {
            std::cout << "send failed with error: " << WSAGetLastError() << std::endl;
            closesocket(ClientSocket);
            WSACleanup();
            break;
        }
        std::cout << "Bytes sent: " << iSendResult << std::endl;
    }
    else if (iResult == 0) {
        std::cout << "Connection closing..." << std::endl;
    } else {
        std::cout << "recv failed with error: " << WSAGetLastError() << std::endl;
        closesocket(ClientSocket);
        WSACleanup();
        break;
    }
} while (iResult > 0);
addData(ss.str());

Then, addData pushes back the information received into a queue buffer (inside a mutex region), casted as uint8:

std::vector<uint8_t> v(data.begin(), data.end());
lck.lock();
buffer.push(v);
lck.unlock();

From another part of my program, I'm reading (as well inside a mutex region) the oldest element in that queue

std::vector<uint8_t> & ImageServer::getNextImage()
{
    lck.lock();
    if (buffer.size() > 0)
    {
        frameJPG = buffer.front();
        buffer.pop();
    }

    lck.unlock();
    return frameJPG;
}

and I try to decode it using openCV function again:

try {
    vec = is->getNextImage();
    img = cv::imdecode(vec, CV_LOAD_IMAGE_UNCHANGED);

    if (img.data == NULL)
        std::cout << "Image in buffer couldn't be properly decoded" << std::endl;
    else
        std::cout << "IMAGE RECOVERED!" << std::endl;
} catch (...) {
    std::cout << "AN UNKNOWN ERROR OCURRED" << std::endl;
}

It is always going into img.data == NULL. That means that there is not enough data (not the case) or the data can't be properly formatted. Why? I'm messing things up when storing it into a stringstream as a first buffer to gather the whole data stream? It is split into 1024 bytes packages...

I've checked:

1. Size of the received data in the server equals to data size in the client side.
2. b"Hello World!" strings are being properly sent and received.
3. Both Raspbian and Windows 7 are little endian
4. Both OpenCV installations are openCV3.x

After 1000 different test with different castings I'm stuck just in the same point: a binary stream which I cannot parse. Does OpenCV behave in a different way in Windows and in Python so it is not able to decode in C++ what was encoded in Python? Do sockets receive information unsorted? Do the sockets add some header that I have to trim in some way? Is it caused by the OS architecture (x86 vs x64)?

Thank you so much in advance.

If you need more code, I can post it too, but I would like to keep it clean!

Regards, maije

Answer 1

Yes!, Öö Tiib, that was it. On Friday I was suspicious about that, and you are right. This morning I changed it and it worked smoothly, without any other modification in the code but in the add and the get data methods! (to fit into the new data structure).

So here it is as I'm doing it in a first test:

std::vector<uint8_t> v;
// Receive until the peer shuts down the connection
do {
    iResult = recv(ClientSocket, recvbuf, recvbuflen, 0);
    if (iResult > 0) {
        totalBytes += iResult;
        for (auto i = 0; i < iResult; ++i)
            v.push_back(recvbuf[i]);
    //some more lines unreleated to the answer

Is there any more efficient way to fill the vector v? Thank you so much in advance for your very appreciated help on this!

PS: I'm so sorry I cannot upvote your answer since I have no reputation to do so :-(

Answer 2

I think that your bytes get cut with ss << recvbuf; since << treats that recvbuf as null-terminated string. Instead it should be treated as buffer that contains iResult bytes. That means that it is not null-terminated, its last byte may be anything and there may be null values in middle of it.

I suggest to drop that ss and instead to build a std::vector<uint8_t> directly from pieces received in recvbuf.

You can insert all contents of recvbuf to end of vector v like that:

v.insert(v.end(), recvbuf, recvbuf + iResult);