Import Subtitles in Processing

Question

I am working on this sketch on Processing which gets the videofeed from my webcam/smartphone and shows it when running. I want to import an .srt converted to txt subtitles file from a film to it. I can see that the text file has all these numbers that stand for the start and end subtitle frame before the actual text.

Here's an example:

{9232}{9331}Those swings are dangerous.|Stay off there. I haven't fixed them yet.
{9333}{9374}I think you're gonna live.

What I would like to do is to figure outa code that will

1. use the numbers and set them as start/end frames to run at the right time as in the film
2. display the subtitles
3. figure out how the '|'sign can be used as a symbol to trigger in the script the change of line.

I guess that might already be quite complicated but I just wanted to check whether someone has done anything similar in the past..

I guess what I want to do is save me from doing the whole

if ((current_frame > 9232) && ((current_frame < 9331)) {
    text("Those swings are dangerous.", 200, 500/2);
    text("Stay off there. I haven't fixed them yet..", 200, (500/2 + 35));
} 

thing for each subtitle...

I am quite new to processing so not that familiar with many commands apart from 'for' and 'if', a newbie at importing .txt files and an ignoramus in working with arrays. But I really want to find a nice way in the last two bits..

Any help in any form will be greatly appreciated :)

Cheers, George

Answer 1

For displaying the appropriate subtitle, you could do something like the following (explanation below, sorry in advance for the wall of text):

String[] subtitles = loadStrings("subtitles.txt");
int currentFrame = 0;
int subtitleIndex = -1;
int startFrame = -1, endFrame = -1;
int fontSize = 10; //change to suit your taste
String[] currentSubtitle;
...

//draw loop start:
//video drawing code goes here

if(currentFrame > endFrame){ //update which subtitle is now/next
  subtitleIndex++;
  startFrame = int(subtitles[subtitleIndex].split("\\}\\{")[0].substring(1));
  endFrame = int(subtitles[subtitleIndex].split("\\}\\{")[1].split("\\}")[0]);
  currentSubtitle = subtitles[subtitleIndex].split("\\}")[2].split("\\|");
}
if(currentFrame >= startFrame && currentFrame <= endFrame){
  for(int i = 0; i < currentSubtitle.length; i++){
    text(currentSubtitle[i], width/2, height - fontSize * (currentSubtitle.length - i));
  }
}
currentFrame++;
//draw loop end

Probably that looks pretty intimidating to you, so here's some walk-through commentary. Your program will be a type of state machine. It will either be in the state of displaying a subtitle, or not. We'll keep this in mind later when we're designing the code. First, you need to declare and initialize your variables.

The first line uses the loadStrings() function, which reads through a text file and returns a String array where each element in the array is a line in the file. Of course, you'll need to change the filename to fit your file.

Your code uses a variable called current_frame, which is a very good idea, but I've renamed it to currentFrame to fit the java coding convention. We'll start at zero, and later on our code will increment it on every frame display. This variable will tell us where we are in the subtitle sequence and which message should be displayed (if any).

Because the information for what frame each subtitle starts and ends on is encoded in a string, it's a bit tricky to incorporate it into the code. For now, let's just create some variables that represent when the "current" subtitle-- the subtitle that we're either currently displaying or will be displaying next-- starts and ends. We'll also create an index to keep track of which element in the subtitles array is the "current" subtitle. These variables all start at -1, which may seem a bit odd. Whereas we initialized currentFrame to 0, these don't really have a real "initial" value, at least not for now. If we chose 0, then that's not really true, because the first subtitle may not (probably doesn't) begin and end at frame 0, and any other positive number doesn't make much sense. -1 is often used as a dummy index that will be replaced before the variable actually gets used, so we'll do that here, too.

Now for the final variable: currentSubtitle. The immediate thought would be to have this be a plain String, not a String array. However, because each subtitle may need to be split on the pipe (|) symbols, each subtitle may actually represent several lines of text, so we'll create an array just to be safe. It's possible that some subtitles may be a single-element array, but that's fine.

Now for the hard part!

Presumably, your code will have some sort of loop in it, where on each iteration the pertinent video frame is drawn to the screen and (if the conditions are met), the subtitle is drawn over top of it. I've left out the video portion, as that's not part of your question.

Before we do anything else, we need to remember that some of our variables don't have real values yet-- all those -1s from before need to be set to something. The basic logic of the drawing loop is 1) figure out if a subtitle needs to be drawn, and if so, draw it, and 2) figure out if the "current" subtitle needs to be moved to the next one in the array. Let's do #2 first, because on the first time through the loop, we don't know anything about it yet! The criterion (in general) for moving to the next subtitle is if we're past the end of the current one : currentFrame > endFrame. If that is true, then we need to shift all of our variables to the next subtitle. subtitleIndex is easy, we just add one and done. The rest are... not as easy. I know it looks disgusting, but I'll talk about that at the end so as to not break the flow. Skip ahead to the bottom if you just can't wait :)

After (if necessary) changing all of the variables so that they're relevant to the current subtitle, we need to do some actual displaying. The second if statement checks to see if we're "inside" the frame-boundaries of the current subtitle. Because the currentSubtitle variable can either refer to the subtitle that needs to be displayed RIGHT NOW, or merely just the next one in the sequence, we need to do some checking to determine which one it is for this frame. That's the second if statement-- if we're past the start and before the end, then we should be displaying the subtitle! Recall that our currentSubtitle variable is an array, so we can't just display it directly. We'll need to loop through it and display each element on a separate line. You mentioned the text() command, so I won't go too in depth here. The tricky bit is the y-coordinate of the text, since it's supposed to be on multiple lines. We want the first element to be above the second, which is above the third, etc. To do that, we'll have the y-coordinate depend on which element we're on, marked by i. We can scale the difference between lines by changing the value of fontSize; that'll just be up to your taste. Know that the number you set it to will be equal to the height of a line in pixels.

Now for the messy bit that I didn't want to explain above. This code depends on String's split() method, which is performed on the string that you want split and takes a string as a parameter that instructs it how to split the string-- a regex. To get the startFrame out of a subtitle line in the file, we need to split it along the curly braces, because those are the dividers between the numbers. First, we'll split the string everywhere that "}{" occurs-- right after the first number (and right before the second). Because split() returns an array, we can reference a single string from it using an index between square braces. We know that the first number will be in the first string return by splitting on "}{", so we'll use index 0. This will return (for example) "{1234", because split() removes the thing you're splitting on. Now we need to just take the substring that occurs after the first character, convert it to an int using int(), and we're done!

For the second number, we can take a similar approach. Let's split on "}{" again, only we'll take the second (index 1) element in the returned array this time. Now, we have something like "9331}Those swings are dang...", which we can split again on "}", choose the first string of that array, convert to int, and we're done! In both of these cases, we're using subtitles[subtitleIndex] as the original String, which represents the raw input of the file that we loaded using loadStrings() at the beginning. Note that during all this splitting, the original string in subtitles is never changed-- split(), substring(), etc only return new sequences and don't modify the string you called it on.

I'll leave it to you to figure out how the last line in that sequence works :)

Finally, you'll see that there are a bunch of backslashes cluttering up the split() calls. This is because split() takes in a regex, not a simple string. Regexs use a lot of special notation which I won't get into here, but if you just passed split() something like "}{", it would try to interpret it and it would not behave as expected. You need to escape the characters, telling split() that you don't want them to be interpreted as special and you just want the characters themselves. To do that, you use a backlash before any character that needs to be escaped. However, the backslash itself is yet another special character, so you need to escape it, too! This results in stuff like "\\{" -- the first backslash escapes the second one, which escapes whatever the third character is. Note that the | character also needs to be escaped.

Sorry for the wall of text! It's nice to see questions asked intelligently and politely, so I thought I'd give a good answer in return.