Python video stream to C++ server

Question

I am working to learn C++ through a project where I stream video from one device to display on another. I have a PiCamera with the following client code that captures frames and sends them via TCP stream.

import io
import socket
import struct
import time
import picamera


def stream():
    servver_ip = "0.0.0.0" #intentionally left out
    client_socket = socket.socket()
    client_socket.connect((server_ip, 8123))
    connection = client_socket.makefile("wb") 

    try:
        camera = picamera.PiCamera()
        camera.resolution = (640, 480)
        camera.start_preview()
        time.sleep(2)
        start = time.time()
        stream = io.BytesIO()
        for image in camera.capture_continuous(stream, 'jpeg'):
            connection.write(struct.pack("<L", stream.tell()))
            connection.flush()
            stream.seek(0)
            connection.write(stream.read()) 
            stream.seek(0)
            stream.truncate()
        connection.write(struct.pack("<L", 0)) 
    except Exception as e:
        raise e
    finally:
        connection.close()
        client_socket.close()

def main():
    while True:
        try:
            stream()
        except ConnectionRefusedError as e:
            print("Connection refused. It the server on?")
            time.sleep(2)
        except Exception as e:
            print(e)
            break 

if __name__ == "__main__":
    main()

The code above comes directly from a PiCamera recipie and works great when I have a Python script on the other end. However, when I attempt to receive and display the stream with the following C++ code I get only partial frames and it doesn't flow well at all. Sometimes I get 0 data, 1 frame, or a slow choppy mess. Changing the usleep increment seems to get some improvement but I fear it's not the correct answer. I've also tried using the MSG_WAITALL flag in recv but that seems to keep any data from coming through which might suggest I have an incorrect value for my buffer size.

// the standard stuff
#include <iostream>
#include <string>
#include <unistd.h>

// opencv
#include <opencv2/opencv.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/video/video.hpp>

//socket stuffs
#include <sys/socket.h>
#include <sys/types.h>
#include <netdb.h>
#include <netinet/in.h>


int main(int argc, const char * argv[]) {

    int key;

    int yes = 1;

    int height = 480;
    int width = 640;

    char* listenPort = "8123";

    int bytes = 0;

    int status;
    struct addrinfo hints; // define our interface
    struct addrinfo *serviceinfo; // will point to results
    struct sockaddr_storage their_addr;
    memset(&hints, 0, sizeof(hints)); // make sure the struct is empy
    hints.ai_family = AF_UNSPEC; // don't care if IP4 or IP6
    hints.ai_socktype = SOCK_STREAM; // TCP
    hints.ai_flags = AI_PASSIVE; // fill in my IP for me

    // look up address info for this machine. Will populate service info
    if ((getaddrinfo(NULL, listenPort, &hints, &serviceinfo)) == -1){
        fprintf(stderr, "getaddinfo error: %s\n", gai_strerror(status));
        exit(1);
    }

    // make the socket
    int sockfd = socket(serviceinfo->ai_family, serviceinfo->ai_socktype, serviceinfo->ai_protocol);

    //bind to the correct port
    if ((bind(sockfd, serviceinfo->ai_addr, serviceinfo->ai_addrlen)) == -1){
        fprintf(stderr, "failed to bind: %s\n", gai_strerror(status));
        exit(1);
    }

    // allow us to reuse the port
    if ((setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(yes))) == -1){
        perror("setsockopt");
        exit(1);
    }

    // number of connections to let in the queue
    int backlog = 1;

    // start listening for incoming connections
    if ((listen(sockfd, backlog)) == -1){
        perror("listen");
        exit(1);
    }

    // start accepting incoming connections
    socklen_t addr_size = sizeof(their_addr);
    int new_fd = accept(sockfd, (struct sockaddr *)&their_addr, &addr_size); // use this for all send and receive calls
    if (new_fd == -1){
        perror("accept");
        exit(1);
    }

    // empty image object
    cv::Mat img = cv::Mat::zeros(height, width, CV_8UC1);

    if (!img.isContinuous()){
        img = img.clone();
    }

    int image_size = img.total() * img.elemSize();

    cv::Mat rawImage = cv::Mat::zeros(1, image_size, CV_8UC1);

    if (!rawImage.isContinuous()){
        rawImage = rawImage.clone();
    }
    cv::Mat flipped = cv::Mat::zeros(height, width, CV_8UC1);

    std::cout << "Press q to quit" << std::endl;
    cv::namedWindow("Sentry", cv::WINDOW_AUTOSIZE);

    while (key != 'q'){
        std::cout << "Capturing" << std::endl;

        if ((bytes = recv(new_fd, rawImage.data, image_size, 0)) == -1){
            perror("Failed to receive bytes!");
            exit(1);
        }

        img = cv::imdecode(rawImage, CV_LOAD_IMAGE_COLOR);
        cv::flip(img, flipped, 1);


        if (flipped.data != NULL) {
            // show image. using a global here
            cv::imshow("Sentry", flipped);
        }

        memset(rawImage.data, 0x0, sizeof(rawImage));

        // look for quit key
        key = cv::waitKey(10);

        // pause for half a second
        usleep(500000);
    };

    cv::destroyAllWindows();

    freeaddrinfo(serviceinfo); // clear the linked list

    close(sockfd);

    return 0;

}

I'm looking for any tips, answers or just to be pointed in the right direction. Thanks in advance.

EDIT: Working solution
First, thanks Kevin for helping me out. My issue was I did not realize my initial Python client was sending the image size. Doing some searching and working with the answers below I started grabbing those first 4 bytes and began resizing my cv::Mat rawImage. Instead of writing the logic to check recv to ensure I'm getting all the data, I'm using MSG_WAITALL. That flag in combination with getting the correct payload size got everything working smoothly. Very humbling experience.

#define CHUNKSIZE 500

int main(int argc, const char * argv[]) {

    // skipping code from before

    int32_t image_size = 0;
    cv::Mat rawImage;
    long data_received = 0;
    long success;

    cv::namedWindow("Sentry", cv::WINDOW_AUTOSIZE);

    while (key != 'q'){
        std::cout << "Capturing" << std::endl;

        // Very important!! Grab the image_size here
        success = recv(new_fd, &image_size, 4 , NULL);
        if (success == -1){
            perror("Failed to receive file size!");
            exit(1);
        }
        // if your image size is extremely large, it's probably 
        // because you're grabing the wrong number of bytes above
        if (image_size > 300000){
            std::cout << "Image size is too large " << image_size << std::endl;
            exit(1);
        }

        if (image_size > 0) {

            //
            rawImage = cv::Mat::zeros(1, image_size, CV_8UC1);
            success = recv(new_fd, rawImage.data, image_size, MSG_WAITALL);
            if (success == -1){
                perror("Failed to receive");
                exit(1);
            }

            img = cv::imdecode(rawImage, CV_LOAD_IMAGE_COLOR);

            if (img.data != NULL) {
                // show image. using a global here
                cv::imshow("Sentry", img);
            } else {
                std::cout << "Image is null!" << std::endl;
            }

            memset(&rawImage, 0x0, sizeof(rawImage));

            // look for quit key
            key = cv::waitKey(10);
        } else {
            std::cout << "No message yet" << std::endl;

        }
        image_size = 0;
        // pause for half a second
        usleep(500000);
    };

    cv::destroyAllWindows();

    freeaddrinfo(serviceinfo); // clear the linked list

    close(sockfd);

    return 0;

}

Answer 1

I too had this same problem when I created a live stream Raspberry Pi to my PC using wifi with C++.

I solved it by only sending chunks of image. Test the chunks at around 500 bytes then increase it once you see its working. Dont send the entire file or else the program will only get part of the frame.

Also, dont read the return code of send() or recv() functions as @Sam stated, I have actually found it to be unreliable when creating a live stream. Just send it in chunks and it should be okay.

I have my source code if you want to take a look it at, I know its very hard to create a live stream server using C++ because there is almost no documentation, so I want to help you out since it gave me sooo much head aches! Let me know if you want the source good luck!

Client should look something like this, this wont compile! I tried to make it easily readable:

//Client    
#define CHUNK_SIZE 500

int main()
{
    //do all the connection stuff here and connect to it
    int filesize;
    int dataReceived = 0;
    char received_message[10];
    std::vector<char> imageData;
    while (1)
    {
        recv(Connection, received_message, 11, NULL); //recieved File size of image
        filesize = atoi(received_message); //Turning File size char into an integer 

        imageData.resize(filesize); //Setting up vector to recieve image data

        while (dataReceived < filesize) //keep running until we get enough bytes
        {
            if (filesize - dataReceived >= CHUNK_SIZE) //if the amount of bytes left to recieve is greater than or equal to chunk size, then lets receive a CHUNK_SIZE of the image
            {
                recv(Connection, &imageData[dataReceived], CHUNK_SIZE, 0);
                dataReceived += CHUNK_SIZE;
            }
            else //Else, if the amount of bytes left to recieve is smaller than the CHUNK_SIZE then lets receive a specific amount of bytes
            {
                recv(Connection, &imageData[dataReceived], fileSize - dataReceived, 0);
                dataReceived += fileSize - dataReceived;
            }
         Sleep(Amount of sleep); //Note, python code will always be slower than C++ so you must set a Sleep() function.
        }
        std::cout << "Got the image, it is stored in the imageData vector!" << std::endl;
        std::cout << "Press enter to get another image!" << std::endl;
        std::cin.get(); 

    }
} 

Answer 2

The shown code mostly ignores the return value from recv().

if ((bytes = recv(new_fd, rawImage.data, image_size, 0)) == -1){

It is true that an error results in recv() returning -1.

But if recv() did not return -1, it doesn't mean that recv() has read image_size bytes, here.

But the shown code now appears to assume that the entire image has been read, image_size bytes. This is a false assumption. When it comes to network sockets, if the receiver has, to date, received fewer bytes, recv() will immediately return.

A positive return value from recv() indicates how many bytes were actually received and placed into the buffer. If this is fewer bytes than you expected, you are responsible for calling recv() again. Note that you can't just

recv(new_fd, rawImage.data, image_size, 0)

again, because the initial part of rawImage.data already contains however many bytes were received by the first call to recv(). You will need to work out some simple logic, here, to read the rest of it (keeping in mind that the second call to recv() isn't guaranteed to read the rest of it, either).