CODEWITHSUNDEEP
javascreenshotvideo-capturevideo-encoding
Mister Smith
Mister Smith

Reputation: 28168

Alternatives for generating a video feed from screenshots

I'm working in a remote administration toy project. For now, I'm able to capture screenshots and control the mouse using the Robot class. The screenshots are BufferedImage instances.

First of all, my requirements: - Only a server and a client. - Performance is important, since the client might be an Android app.

I've thought on opening two socket connections, one for mouse and system commands and the second one for the video feed.

How could I convert the screenshots to a video stream? Should I convert them to a known video format or would it be ok to just send a series of serialized images?

The compression is another problem. Sending the screen captures in full resolution would result in a low frame rate, according to my preliminary tests. I think I need at least 24 fps to perceive movement, so I've to both downscale and compress. I could convert the BufferedImages to jpg files and then set the compression rate, but I don't want to store the files on disk, they should live in RAM only. Another possibility would be to serialize instances (representing an uncompressed screenshot) to a GZipOutputStream. What is the correct approach for this?

To summarize:

Thanks in advance.

UPDATE: my tests, client and server on same machine
-Full screen serialized BufferedImages (only dimension, type and int[]), without compression: 1.9 fps.
-full screen images through GZip streams: 2.6 fps.
-Downscaled images (640 width) and GZip streams: 6.56 fps.
-Full screen images and RLE encoding: 4.14 fps.
-Downscaled images and RLE encoding: 7.29 fps.

Upvotes: 6

Views: 2266

Answers (5)

Louis Ricci
Louis Ricci

Reputation: 21086

  • Break the screen up into a grid squares (or strips)
  • Only send the grid square if it's different from the previous

// server start

sendScreenMetaToClient(); // width, height, how many grid squares
...

// server loop ImageBuffer[] prevScrnGrid while(isRunning) {

ImageBuffer scrn = captureScreen();
ImageBuffer[] scrnGrid = screenToGrid(scrn);
for(int i = 0; i < scrnGrid.length; i++) {
    if(isSameImage(scrnGrid[i], prevScrnGrid[i]) == false) {
        prevScrnGrid[i] = scrnGrid[i];
        sendGridSquareToClient(i, scrnGrid[i]); // send the client a message saying it will get grid square (i) then send the bytes for grid square (i)
    }
} }

Don't send serialized java objects just send the image data.

ByteArrayOutputStream imgBytes = new ByteArrayOutputStream();
ImageIO.write( bufferedImage, "jpg", imgBytes );
imgBytes.flush();

Upvotes: 5

blahdiblah
blahdiblah

Reputation: 34001

If you're trying to get 24fps video then there's no reason not to use modern video codecs. Why try and recreate that wheel?

Xuggler works fine for encoding h264 video and sounds like it would serve your needs nicely.

Upvotes: 0

Durandal
Durandal

Reputation: 20059

If its just screen captures, I would not compress them using a Video compression scheme, most likely you don't want lossy compression (blurred details in small text etc are the most common defects). For getting a workable "remote Desktop" feel, remember the previously sent Screenshot and send only the difference to get to the next one. If nothing (or very little) changes between frames this is very efficient. It will however not work well in certain situations like playing a video, game or scrolling a lot in a document.

Compressing the difference between two BufferedImage can be done with more or less elaborate methods, a very simple, yet reasonably effective method is simply to subtract one image from the other (resulting in zeros everywhere they are identical) and compressing the result with simple RLE (run length encoding).

Reducing the color precision can be used to further reduce the amount of data (depending on the use case you could omit the least significant N bits of each color channel, for most GUI applications look not much different if you reduce colors from 24 bits to 15 bits).

Upvotes: 6

Elias Vasylenko
Elias Vasylenko

Reputation: 1534

Firstly, I might suggest only capturing a small part of the screen, rather than downscaling and potentially losing information, perhaps with something like a sliding window which can be moved around by pushing the edges with a cursor. This is really just a small design suggestion though.

As for compression, I would think that a series of images would not compress separately as well as with a decent video compression scheme, especially as frames are likely to remain consistent between captures in this scenario.

One option would be to use Xuggle, which is capable of capturing the desktop via Robot in a number of video formats afaiu, but I can't tell if you can stream and decode with this.

For capturing jpegs and converting them, then you can also use this.

Streaming these videos seems to be a little more complicated, though.

Also, it seems that the abandoned Java Media Framework supports this functionality.

My knowledge in this area is not fantastic tbh, so sorry if I have wasted your time, but it looks like some more useful information on the feasibility of using Xuggle as a screensharer has been compiled here. This also appears to link to their own notes on existing approaches.

If it doesn't need to be pure Java I reckon this would all be much easier using just by interfacing with a native screen capture tool...

Maybe it would be easiest just to send video as a series of jpegs after all! You could always implement your own compression scheme if you were feeling a little crazy...

Upvotes: 3

vipw
vipw

Reputation: 7645

I think you described a good solution in your question. Convert the images to jpeg, but don't write them as files to disk. If you want it to be a known video format, use M-JPEG. M-JPEG is a stream of jpeg frames in a standard format. Many digital cameras, especially older ones, save videos in this format.

You can get some information about how to play an M-JPEG stream from this question's answers: Android and MJPEG

If network bandwidth is a problem, then you'll want to use an inter-frame compression system such as MPEG-2, h.264, or similar. That requires a lot more processing than M-JPEG but is far more efficient.

Upvotes: 2

Related Questions