Webcam stream in opencv using Java

Question

I want to stream my webcam in my java application for some image processing stuff. I am using OpenCV and Java Library files which came along with it.

I got how to capture image, but how to capture and display frames continuously like a video.

Please suggest modification in below code or an alternative way to do so.

I m using Java Swing Application development .

package openc;
import org.opencv.core.*;

public class openc {

private JFrame frame;

/**
 * Launch the application.
 */
public static void main(String[] args) {
    EventQueue.invokeLater(new Runnable() {
        public void run() {
            try {
                openc window = new openc();
                window.frame.setVisible(true);
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    });
}

/**
 * Create the application.
 */
public openc() {
    initialize();
}

/**
 * Initialize the contents of the frame.
 */
private void initialize() {
    frame = new JFrame();
    frame.setBounds(100, 100, 450, 300);
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);

    Canvas canvas = new Canvas();
    frame.getContentPane().add(canvas, BorderLayout.CENTER);

            System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
            VideoCapture camera = new VideoCapture(0);
            if(!camera.isOpened()){
                System.out.println("Error");
            }
            else {
                Mat iframe = new Mat();
                while(true){
                    if (camera.read(iframe)){
                        System.out.println("Frame Obtained");
                        System.out.println("Captured Frame Width " + 
                        iframe.width() + " Height " + iframe.height());
                        while(true){
                        Highgui.imwrite(canvas, iframe);
                        }


                    }
                }   
            }
            camera.release();

}

}

Answer 1

dic19 is right, you need to encapsulate the the process of frame graping and repainting the Swing component in a SwingWorker. However I would avoid Thread.sleep and usage of ImageIO, because the actual output on the Swing component will perform poor.

So, I suggest those changes:

1.: Initialize the VideoCapture in the constructor of the SwingWorker and remove the sleep within the loop of the doInBackground() method:

@Override
protected Void doInBackground() throws Exception {
    Mat webcamImage = new Mat();
    while (!isCancelled()) {
        camera.read(webcamImage);
        if (!webcamImage.empty()) {
            videoPanel.updateImage(webcamImage);
        } else {
            break;
        }
    }

    return null;
}

2.: In the Swing component, that renders the image, you should use System.arraycopy. It is faster then ImageIO.

public void updateImage(Mat matrix) {
    int type = BufferedImage.TYPE_BYTE_GRAY;
    if (matrix.channels() > 1) {
        type = BufferedImage.TYPE_3BYTE_BGR;
    }
    byte[] b = new byte[matrix.channels() * matrix.cols() * matrix.rows()];
    matrix.get(0, 0, b);
    image = new BufferedImage(matrix.cols(), matrix.rows(), type);
    final byte[] targetPixels = ((DataBufferByte) image.getRaster().getDataBuffer()).getData();
    System.arraycopy(b, 0, targetPixels, 0, b.length);

    repaint();
}

Finally, override the paintComponent() method:

@Override
public void paintComponent(Graphics g) {
    super.paintComponent(g);
    if (image == null) {
        return;
    }
    g.drawImage(this.image, 1, 1, this.image.getWidth(), this.image.getHeight(), null);
}

Search GitHub for samples, I'm sure you will find something helpfull there.

Answer 2

I got how to capture image, but how to capture and display frames continuously like a video.

I have never used OpenCV API so I don't know nothing about this library. However I think your problem is a tipical thread-blocking issue caused by infinite loops:

while(true){ // infinite loop

    if (camera.read(iframe)) {
        System.out.println("Frame Obtained");
        System.out.println("Captured Frame Width " + 
        iframe.width() + " Height " + iframe.height());

        while(true) { // another infinite loop
            Highgui.imwrite(canvas, iframe);
        }
    }
} 

You have two infinite loops that are blocking the Event Dispatch Thread causing your GUI become unresponsive. Swing is single threaded and you have to take extra care on how you create/update your Swing components.

In this particular case I think you could use a SwingWorker to read camera data periodically in a background thread and update Canvas object in the EDT. Something like this:

SwingWorker<Void, Mat> worker = new SwingWorker<Void, Mat>() {
    @Override
    protected Void doInBackground() throws Exception {            
        while(!isCancelled()) {
            if (camera.read(iframe)) {
                publish(iframe);
            }
            Thread.sleep(500); // prudential time to avoid block the event queue
        }            
        return null;
    }

    @Override
    protected void process(List<Mat> chunks) {
        Mat lastFrame = chuncks.get(chunks.size() - 1);
        Highgui.imwrite(canvas, lastFrame);
    }        
};

worker.execute();