Html Parsing with regex over multiple lines

Question

Currently I'm writing a program in Java, where I have to parse a html file and get all the names from the tables. I have to write it in pure Java, so I can't use Jsoup or something similar.

a snippet of the html is here:



Comic

Film



 0–9[Bearbeiten | Quelltext bearbeiten]


2 Guns

2 Guns (2013)


5 ist die perfekte Zahl (von Igort)

5 è il numero perfetto (2019)


13 rue de l'Espoir (von Paul Gillon)

Die tolle Masche (1961)


XIII

XIII – Die Verschwörung (Fernseh-Zweiteiler, 2008)


XIII – Die Verschwörung (Fernsehserie, 2011–2012)


20th Century Boys

20th Century Boys 1 (2008)


20th Century Boys 2 (2009)


20th Century Boys 3 (2009)


30 Days of Night

30 Days of Night (2007)


30 Days of Night: Dark Days (2010)


47:an Löken (von Lennart Elworth)

47:an Löken (1971)


47:an Löken blåser på (1972)


91:an Karlsson (von Rudolf Petersson)

Aber warum, Herr Feldwebel? (1946)


Zurück, marsch-marsch! (1947)


Rekruten rechts 'raus (1951)


Alla tiders 91:an Karlsson (1953)


91 Karlsson rycker in (1955)


91:an Karlsson slår knock out (1957)


91:an Karlsson muckar (tror han) (1959)


91:an och generalernas fnatt (1977)


300

300 (2007)


300: Rise of an Empire (2014)



 A[Bearbeiten | Quelltext bearbeiten]


Abattoir (Radical Studios)

Abattoir – Er erwartet dich! (2016)


Ace Drummond (von Eddie Rickenbacker und Clayton Knight)

Ace Drummond (Serial, 1936)


Accident Man (von Pat Mills u. a.)

Accident Man (2018)


Ada im Dschungel (von Altan)

Ada dans la jungle (1988)


The Addams Family (Zeitungscartoons)

Die Addams Family (Fernsehserie, 1964–1966)


Halloween with the New Addams Family (1977)


Addams Family (1991)


Die Addams Family in verrückter Tradition (1993)


Addams Family – Und die lieben Verwandten (1998)


Die neue Addams Familie (Fernsehserie, 1998–1999)


Adeles ungewöhnliche Abenteuer

Adèle und das Geheimnis des Pharaos (2010)


Agent Carter

Marvel’s Agent Carter (Fernsehserie, 2015–2016)


Air Hawk (von John Dixon)

Air Hawk (1981)


Alena (von Kim W. Andersson)

Alena (2015)


Alexander Nikopol (Die Geschäfte der Unsterblichen)

Immortal – New York 2095: Die Rückkehr der Götter (2004)


Allein

Seuls (2017)


Der alltägliche Kampf (von Manu Larcenet)

Le Combat ordinaire (2015)


Ally Sloper (von Marie Duval und C. H. Ross)

Ally Sloper (Kurzfilm, 1898)


Sloper's Visit to Brighton (Kurzfilm, 1898)


Ally Sloper (Kurzfilm-Serie, 1921)


Along with the Gods

Along with the Gods: The Two Worlds (2017)


Along with the Gods: The Last 49 Days (2018)


Alphonse and Gaston (von Frederick Burr Opper)

Alphonse and Gaston (Kurzfilm-Serie, 1902–1903)


Anmerkung zu Trickfilmen und Comiczeichnern[Bearbeiten | Quelltext bearbeiten]

At first I wanted to match only the which have two in them but I can't figure out why it isn't working. I checked my regex on www.regex101.com and it seems to be fine. But in Java I had to replace the with .*? to even get a match. This i my Java Code:

import utils.YearInterval;

import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Scanner;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class Parser {
    private Pattern comicBegin = Pattern.compile(".*?", Pattern.MULTILINE | Pattern.DOTALL);
    private Pattern comicEnd = Pattern.compile(".*?", Pattern.MULTILINE | Pattern.DOTALL); 
    private Pattern comicEnum = Pattern.compile(".*?(?:)?(.*?)(?:)?.*?.*?(?:)?(.*?)(?:)?.*?", Pattern.MULTILINE | Pattern.DOTALL);  // Tried also this \n(?:)?(.*?)(?:)?\n\n(?:)?(.*?)(?:)?\n
    private Scanner wikiComicFilmScanner = null;

    public static void main(String[] args) throws MalformedURLException, IOException {
        Path wikiComicFilmLocal = Paths.get("ressources/ListevonComicverfilmungen.html");
        Parser wp1 = new Parser("file:///" + wikiComicFilmLocal.toAbsolutePath());
        long start = System.currentTimeMillis();
        Map>> comicMap = wp1.contentToComicFilmsPerYear();
        System.out.println("Parsen ausgeführt in: " + (System.currentTimeMillis() - start) + "ms");
    }

    public Parser(String uri) throws MalformedURLException, IOException {
        wikiComicFilmScanner = new Scanner(new URL(uri).openStream(), "UTF-8");
    }


    public Map>> contentToComicFilmsPerYear() {
        Map>> comicMap = new HashMap<>();
        wikiComicFilmScanner.useDelimiter(comicBegin);
        if (wikiComicFilmScanner.hasNext()) {
            wikiComicFilmScanner.next();
        }
        wikiComicFilmScanner.useDelimiter(comicEnd);
        if (wikiComicFilmScanner.hasNext()) {
            String filmsPerYearEnumeration = wikiComicFilmScanner.next();
            Matcher matcherEnum = comicEnum.matcher(filmsPerYearEnumeration);
            while (matcherEnum.find()) {
                String comicTitle = matcherEnum.group(0);
                String filmTitle = matcherEnum.group(2);
                System.out.println("---------------------------------");
                System.out.println(comicTitle);
                System.out.println(filmTitle);
            }
        }
        return comicMap;
    }

}

Thanks so much!

Thomas Timbul · Accepted Answer

Depending on the source of the HTML a new line might not just be (Unix, new Mac), but (ancient Mac) or (Windows). Note that this depends on the creator of the HTML, not necessarily your own system.

As per Regular Expression to match cross platform newline characters the suggested expression to use for platform-independent newlines is " ?| ".

This will match any of the following:

Since in HTML you probably don't care about the number of occurrences, nor their order or possible mixing, you could also go with [ ]* which avoids creating a capturing group for each one.

Edit:

As @Toto pointed out, you can even more simply also use \R. So \R* should do the trick for you.

Html Parsing with regex over multiple lines

Answers (1)

Related Questions