Extract Main- and Subclauses from German Sentence with SpaCy

Question

In German, how can I extract the main- and subclauses (aka "subordinate clauses", "dependent clauses") from a sentence with SpaCy?

I know how to use SpaCy's tokenizer, part-of-speech tagging and dependency parser, but I cannot figure out how to represent the grammatical rules of German using the information SpaCy can extract.

Answer 1

The problem can be divided into two tasks: 1. Splitting the sentence in its constituting clauses and 2. Identifying which of the clauses is a main clause and which one is a sub-clause. Since there are pretty strict grammatical rules about the structure difference of sub-clauses and main clauses, I would go with a rule-based approach.

Split Sentence into Clauses

A clause contains a finite verb. In German, sub-clauses are separated by comma (",") from the "reigning" clause they depend on (either a main clause, or another sub-clause). Main clauses are separated from other main clauses either by comma or by one of the conjunctions "und", "oder", "aber" and "sondern" (if two main clauses are connected by "und" or "oder", the comma is omitted).

That's why the idea could possibly come to our mind, to split the sentence into chunks by comma and "und"/"oder"/"aber"/"sondern". But this leaves us with the problem, that such things as comma-separated parts which are not a clause exist (think of enumerations, or of appositions), as well as "und"- and "oder" do not always denote the beginning of a new clause (think of enumerations). Also, we could face situations, where the comma at the beginning of a subclause has been omitted. Even if this is against the (normative) grammatical rules of German, we still would want to identify these subclauses correctly.

That's why it is a better idea to start from the finite verbs in the sentence and make use of spacy's dependency parser. We may assume, that each finite verb is part of its own subclause. So we can start from a finite verb and walk through its "progeny" (its children and their children, and so on). This walk needs to stop as soon as it encounters another finite verb -- because this will be the root of another clause.

We then just need to combine the path of this walk into one phrase. This needs to take into account that a Clause can consist of multiple spans -- because a clause can be divided by a subclause (consider relative clauses which relate to an object in the main clause).

Identify whether a Clause is Main Clause or Subclause

Grammatically, in German, subclauses can be identified by the fact that the finite verb is in the last position, which is impossible in main clauses.

So we can make use of spacy's part-of-speech-tags to solve the problem. We can differentiate the different tags of verbs, whether the verb form is finite or infinite, and we can easily check if the last token in the clause (before the punctuation) is a finite or infinite verb form.

Code

import itertools as it
import typing as tp

import spacy


VERB_POS = {"VERB", "AUX"}
FINITE_VERB_TAGS = {"VVFIN", "VMFIN", "VAFIN"}


class Clause:
    def __init__(self, spans: tp.Iterable["spacy.tokens.Span"]):
        """Clause is a sequence of potentially divided spans.

        This class basically identifies a clause as subclause and
        provides a string representation of the clause without the
        commas stemming from interjecting subclauses.

        A clause can consist of multiple unconnected spans, because
        subclauses can divide the clause they are depending on. That's
        why a clause cannot just be constituted by a single span, but
        must be based on an iterable of spans.
        """

        self.spans = spans

    @property
    def __chain(self) -> tp.Iterable["spacy.tokens.Token"]:
        return [token for token in it.chain(*self.spans)]

    # We make this class an iterator over the tokens in order to
    #  mimic span behavior. This is what we need the following
    #  dunder methods for.
    def __getitem__(self, index: int) -> "spacy.tokens.Token":
        return self.__chain[index]

    def __iter__(self) -> tp.Iterator:
        self.n = 0
        return self

    def __next__(self) -> "spacy.tokens.Token":
        self.n += 1
        try:
            return self[self.n - 1]
        except IndexError:
            raise StopIteration

    def __repr__(self) -> str:
        return " ".join([span.text for span in self.inner_spans])

    @property
    def is_subclause(self) -> bool:
        """Clause is a subclause iff the finite verb is in last position."""
        return (
            self[-2].tag_ in FINITE_VERB_TAGS
            if self[-1].pos_ == "PUNCT"
            else self[-1].tag_ in FINITE_VERB_TAGS
        )

    @property
    def clause_type(self) -> str:
        return "SUB" if self.is_subclause else "MAIN"

    @property
    def inner_spans(self) -> tp.List["spacy.tokens.Span"]:
        """"Spans with punctuation tokens removed from span boundaries."""
        inner_spans = []
        for span in self.spans:
            span = span[1:] if span[0].pos_ == "PUNCT" else span
            span = span[:-1] if span[-1].pos_ == "PUNCT" else span
            inner_spans.append(span)

        return inner_spans


class ClausedSentence(spacy.tokens.Span):
    """Span with extracted clause structure.

    This class is used to identify the positions of the finite verbs, to
    identify all the tokens that belong to the clause around each finite
    verb and to make a Clause object of each clause.
    """

    @property
    def __finite_verb_indices(self) -> tp.List[int]:
        return [token.i for token in self if token.tag_ in FINITE_VERB_TAGS]

    def progeny(
        self,
        index: int,
        stop_indices: tp.Optional[tp.List[int]] = None,
    ) -> tp.List["spacy.tokens.Token"]:
        """Walk trough progeny tree until a stop index is met."""
        if stop_indices is None:
            stop_indices = []

        progeny = [index]  # consider a token its own child

        for child in self[index].children:
            if child.i in stop_indices:
                continue

            progeny += [child.i] + self.progeny(child.i, stop_indices)

        return sorted(list(set(progeny)))

    @property
    def clauses(self) -> tp.Generator["Clause", None, None]:
        for verb_index in self.__finite_verb_indices:
            clause_tokens = [
                self[index]
                for index in self.progeny(
                    index=verb_index, stop_indices=self.__finite_verb_indices
                )
            ]

            spans = []

            # Create spans from range extraction of token indices
            for _, group in it.groupby(
                enumerate(clause_tokens),
                lambda index_token: index_token[0] - index_token[1].i,
            ):
                tokens = [item[1] for item in group]
                spans.append(self[tokens[0].i : tokens[-1].i + 1])

            yield Clause(spans)

Example how to run

The following code snippet demonstrates how to use the above classes in order to split a sentence into its clauses:

import spacy


text = "Zu Hause ist dort, wo sich das W-LAN verbindet."  # Could also be a text with multiple sentences

language_model = "de_core_news_lg"
nlp = spacy.load(language_model)  # The spacy language model must be installed, see https://spacy.io/usage/models
document = nlp(text)
sentences = document.sents

for sentence in sentences:
    claused_sentence = ClausedSentence(sentence.doc, sentence.start, sentence.end)
    clauses = list(claused_sentence.clauses)
    for clause in clauses:
        print(f"{clause.clause_type}: {clause.inner_spans}")

Test Cases

I have not run a thorough testing on a larger corpus of different kinds of texts, but I have created some test cases in order to investigate the principal aptitude of the algorithm and potential pitfalls:

Divided Main Clause with Subclause

In meinem Bett, das ich gestern gekauft habe, fühle ich mich wohl.

SUB: das ich gestern gekauft habe
MAIN: In meinem Bett fühle ich mich wohl

Correct.

Main Clause with Subclause

Ich brauche nichts, außer dass mir ab und zu jemand Trost zuspricht.

MAIN: Ich brauche nichts 
SUB: außer dass mir ab und zu jemand Trost zuspricht

Correct.

Sequence of Main Clauses and Subclause

Er sieht in den Spiegel und muss erkennen, dass er alt geworden ist.

MAIN: Er sieht in den Spiegel und 
MAIN: muss erkennen
SUB: dass er alt geworden ist

The assignment of clause types is correct. The "und" could be assigned to the second main clause, though. This would require additionally taking into account whether the last token of a Clause is a conjunction, and if so, assign it to the next clause.

Subclause and Sequence of Main Clauses

Als er die Türklingel hört, rennt er die Treppe hinunter, geht zur Tür, schaut durch den Spion, und öffnet die Tür.

SUB: Als er die Türklingel hört
MAIN: rennt er die Treppe hinunter  und 
MAIN: geht zur Tür
MAIN: schaut durch den Spion
MAIN: öffnet die Tür

Correct. Same problem with the conjunction "und" as above.

Main Clause with Substantivated Verbs

Essen und Trinken hält Leib und Seele zusammen.

MAIN: Essen und Trinken hält Leib und Seele zusammen

Correct.

Main Clause and Subclause

Zu Hause ist dort, wo sich das W-LAN verbindet.

MAIN: Zu Hause ist dort 
SUB: wo sich das W-LAN verbindet

Correct.

Complex Sequence of Main and Subclauses

Angela Merkel, die deutsche Bundeskanzlerin, hat nicht erneut für den Vorsitz ihrer Partei kandidiert, obwohl sie stets der Auffassung war, Kanzlerschaft und Parteivorsitz würden in eine Hand gehören.

SUB: Angela Merkel, die deutsche Bundeskanzlerin, hat 
SUB: nicht erneut für den Vorsitz ihrer Partei kandidiert
SUB: obwohl sie stets der Auffassung war
SUB: Kanzlerschaft und Parteivorsitz würden
SUB: in eine Hand gehören

This is wrong. Correct would be:

MAIN: Angela Merkel, die deutsche Bundeskanzlerin, hat nicht erneut für den Vorsitz ihrer Partei kandidiert, 
SUB: obwohl sie stets der Auffassung war, 
MAIN: Kanzlerschaft und Parteivorsitz würden in eine Hand gehören.

The error is caused by SpaCy misidentifying "kandidiert" as finite verb, while it is a participle, and also misidentifying "gehören" as a finite verb form, while it is an infinite verb. Since this error is based in the underlying language model provided by SpaCy, it seems hard to correct this outpout independently from the language model. However, maybe there could be a rule-based way to override SpaCy's decision to tag these verb forms as infinite verbs. I didn't find a solution yet.