Working with videos on a frame by frame basis

Question

Is there a way in JavaFX to work with frames, e.g. extract frames from a loaded video?

What I mean by working with frames? Triggering actions when a certain frame appears, e.g. using a canvas to show subtitles, or popups on top of videos. Or getting to a certain frame in the video, although this could probably be done using the time index (as could the above, triggering actions when the time index is hit.)

Also, would it be possible to create the following youtube features using JavaFX?

• Subtitles
• Markings along the timeline with hovering over such markings and displaying overlayed content
• Overlays (links, comments, etc.)
• Quality Level Adjustments
• Speed Adjustment

and features that are not there in youtube:

• Extracting Sequence, enriching with other content, adding into the timeline
• Extracting Frames commenting them, adding them back into the video sequence (maybe in such a way that sequences can be added after each other cutting the original one into smaller pieces and inserting a sequence like so 123 --> 12new3

Answer 1

Is there a way in JavaFX to work with frames, e.g. extract frames from a loaded video?

No, there's not - the JavaFX media support is very limited. If you want to load up a video in a certain supported format and play that in a JavaFX scene, then you're good to go. You can do the "normal" actions you'd expect in a video player, so you can pause, stop, seek, mute, etc. but not much more.

Or getting to a certain frame in the video, although this could probably be done using the time index

Yes, you can use a time index (have a look at currentTimeProperty() on MediaPlayer.) This is the only sensible way you're going to have for doing this at the JFX level.

If you want to go further, you're going to have to use the GStreamer framework (or some other native framework) directly. I'd have historically recommended Xuggler if you wanted to get into frame-level video extraction, but that's now been unsupported for a few years.