Python Opencv depth error in Image transimission project

Question

First off, I'm trying to do something like screenshare, I'm confronting an error. I tried to send the image row by row than I figured out that it could work only in LAN since the connection speed was far better. Then I transmitted exclusively each pixel and constructed the array on my own which would be pretty slow. Here is the code: Server side:

def getImg(socket,y,x):
    pixels = []
    
    print(x*y*3)
    for a in range(x*y*3):

        pixels.append(struct.unpack('i', socket.recv(4))[0])

        if a%(x*y)==0:
            pass
    return np.array(pixels).reshape(y,x,3)

Client side:

def sendSS(img):
    

    y, x, color = img.shape
    print(y*x*3)
    for a in range(y):
        for b in range(x):
            for m in img[a][b]:
                socket.send(struct.pack('i', m))

The error is this:

(<class 'cv2.error'>, error("OpenCV(4.5.1) c:\\users\\appveyor\\appdata\\local\\temp\\1\\pip-req-build-wvn_it83\\opencv\\modules\\imgproc\\src\\color.simd_helpers.hpp:94: error: (-2:Unspecified error) in function '__cdecl cv::impl::`anonymous-namespace'::CvtHelper<struct cv::impl::`anonymous namespace'::Set<3,4,-1>,struct cv::impl::A0xeee51b91::Set<3,4,-1>,struct cv::impl::A0xeee51b91::Set<0,2,5>,2>::CvtHelper(const class cv::_InputArray &,const class cv::_OutputArray &,int)'\n> Unsupported depth of input image:\n>     'VDepth::contains(depth)'\n> where\n>     'depth' is 4 (CV_32S)\n"), <traceback object at 0x0000023221307740>)

The way I constructed the numpy array works fine:

import pyautogui
import cv2
import numpy as np
import pickle
import struct
from pprint import pprint as pp
ss = np.array(pyautogui.screenshot())
y,x,color = ss.shape
pixels = []
for a in range(y):
    for b in range(x):
        for m in ss[a][b]:
            pixels.append(struct.pack('i',m))
pixels2 = []
for val in pixels:
    pixels2.append(struct.unpack('i',val)[0])


pixels2 = np.array(pixels2).reshape((1080,1920,3))

cv2.imwrite('name.png',pixels2)

How can I transmit faster and solve this problem?

edit: Here is the part of the code where the error occurs, it says the depth is 4 but I'm sure the dimensions of the image are 1080,1920,3 because I can print it.

image = getImg(aSocket,y,x)

            SS = cv2.cvtColor(image,cv2.COLOR_RGB2BGR) # error occurs in this line 

            aSocket.send('image is gotten'.encode('utf-8'))
            print('image is gotten')
            cv2.imwrite(f'{ip[0]}\{count}.png',SS)

Answer 1

You need to pass the new shape as tuple instead for reshape, eg. arr.reshape((y,x,3)). You might also get another error if OpenCV expects different data type (8 bits per channel, unsigned) than what it got from your program (32 bit signed integer), see documentation for cvtColor(). You can specify the data type when you create the array np.array(..., dtype=np.uint8).

Regarding performance, in Python for loops over large amounts of data can be slow and often you can make your code significantly faster if you can avoid the loop and use library routine instead. Also, you should read and write larger chunks with send() and recv(). If you are not familiar with socket programming, you should at minimum check out the python documentation Socket Programming HOWTO. You eliminate the loops over the image data by directly interpreting the data as bytes. The socket code follows the examples in linked documentation page.

def send_im(sock, im):
    totalsent = 0
    im_bytes = im.tobytes()
    
    while totalsent < len(im_bytes):
        sent = sock.send(im_bytes[totalsent:])
        if sent == 0:
            raise RuntimeError("socket error")
            
        totalsent += sent

Receiver needs to know the size of expected input.

def recv_im(sock, dtype, shape):
    # expected number of bytes
    size = np.dtype(float).itemsize * np.prod(shape)
    im_bytes = bytearray(size) # received bytes
    
    bytes_recvd = 0
    chunk_size = 2048
    
    while bytes_recvd < size:
        chunk = sock.recv(min(size-bytes_recvd, chunk_size))
        if chunk == b'':
            raise RuntimeError("socket connection broken")
        
        im_bytes[bytes_recvd:(bytes_recvd+chunk_size)] = memoryview(chunk)
        bytes_recvd += len(chunk)
        
    # intrepret bytes in correct data type and shape and return image
    return np.frombuffer(im_bytes, dtype=dtype).reshape(shape)

In general, you should always use a serialization library instead of directly sending raw bytes over network, as those libraries usually handle various important details such as byte order, message sizes, data types, conversions, and so on.