Convert a javacv Frame Into Bitmap using AndroidFrameConverter

Question

I am trying to convert a Frame of size 1280x720 into Bitmap using javacv AndroidFrameConverter and it takes long time to convert. Here is the sample code of converting.

FrameGrabber grabber = new FFmpegFrameGrabber(videoUrl);
frame = grabber.grab();
AndroidFrameConverter converter = new AndroidFrameConverter();
Bitmap originalBitmap = converter.convert(frame);

is there any other solution which is faster than this?

Answer 1

You can use this code and change filter according to your needs.

public class FilterAsync extends AsyncTask {

    private FFmpegFrameGrabber VIDEO_GRABBER;
    private FFmpegFrameRecorder videoRecorder;
    private File file = new File(Environment.getExternalStorageDirectory() + "/Download/Abc.mp4");
    private Context mContext;
    private FFmpegFrameFilter filter;
    private boolean isTrue = false;
    private ArrayList<String> videoPaths;
    private File myDirectory;


    public FilterAsync(Context context) {
        mContext = context;
        VIDEO_GRABBER = new FFmpegFrameGrabber(file);
        myDirectory = new File(Environment.getExternalStorageDirectory() + "/Folder/");
        if (!myDirectory.exists()) {
            myDirectory.mkdirs();
        }
        videoPaths = new ArrayList<>();
    }


    @Override
    protected void onPreExecute() {
        super.onPreExecute();

    }

    @Override
    protected Object doInBackground(Object[] params) {
        Frame tempVideoFrame;
        try {
            VIDEO_GRABBER.start();
            initVideoRecorder(myDirectory + "/video" + System.currentTimeMillis() + ".mp4");
            filter.start();
            while (VIDEO_GRABBER.grab() != null) {
                tempVideoFrame = VIDEO_GRABBER.grabImage();
                if (tempVideoFrame != null) {
                    filter.push(tempVideoFrame);
                    tempVideoFrame = filter.pull();
                    videoRecorder.record(tempVideoFrame);
                }
            }
            videoRecorder.stop();
            filter.stop();
            videoRecorder.release();
            VIDEO_GRABBER.stop();
            VIDEO_GRABBER.release();
        } catch (FrameGrabber.Exception e) {
            e.printStackTrace();
        } catch (FrameRecorder.Exception e) {
            e.printStackTrace();
        } catch (FrameFilter.Exception e) {
            e.printStackTrace();
        }

        return null;
    }


    @Override
    protected void onPostExecute(Object o) {
        super.onPostExecute(o);
    }


    private void initVideoRecorder(String path) {
        try {
            filter = new FFmpegFrameFilter("transpose=clock ,crop=w=640:h=480:x=0:y=0", VIDEO_GRABBER.getImageWidth(), VIDEO_GRABBER.getImageHeight());

            videoRecorder = FFmpegFrameRecorder.createDefault(path, VIDEO_GRABBER.getImageWidth(), VIDEO_GRABBER.getImageHeight());
           videoRecorder.setAudioChannels(VIDEO_GRABBER.getAudioChannels());
            videoRecorder.start();
        } catch (FrameRecorder.Exception e) {
            e.printStackTrace();
        }
    }
}

Answer 2

Frame returned by FrameGrabber.grab() is javacv specific, Bitmap is Android specific, I doubt there is a faster convertion method between the two because they don't know each other and both being optimized for their own use they cannot optimize a conversion process between both data formats.

You are moving 1280 * 720 * 4 Bytes, roughly 3.6 MB per frame, it's a lot for mobile devices, above all taking into account memory allocation.

I doubt you'll find another conversion tool without exchanging javacv for something else - can't imagine a tool made specifically to convert javacv's Frame to Android Bitmap to replace the built-in methods.

Besides, AndroidFrameConverter.convert() is already pretty clever - it creates the Bitmap only for the first time or if the specification has changed (like size etc.), then just copies the bytes.

So luckily it has the ability to reuse once created Bitmap instance, FFmpegFrameGrabber has also this ability, so on the first run they will create each an appropriate object but from the second frame on should reuse the objects they have - this way at least no unnecessary memory allocation will happen, which kills performance on mobile devices. That means that converting many frames on the average should be way faster than converting only one frame - just don't create new instances of the grabber and converter for each frame.

Parallelizing using GPU might be possible - create a Bitmap manually, partition Frame's Buffer and set pixels of the Bitmap in parallel. I've never done something like that myself so can't really comment more. Just keep in mind that parallel algorithms have their own overhead and are not always faster than sequential ones. And I'm not sure how portable GPU programming is. Parallelizing using CPU shouldn't be worth trying - how many cores do have current devices? 4? The overhead should be more than the gain.