Generate all word forms using Lucene & Hunspell

Question

In an application I work on, we use Lucene Analyzer, especially it's Hunspell part. The problem I face is: I need to generate all word forms of a word, using a set of affix rules.

E.g. having the word 'educate' and affix rules ABC, generate all forms of word 'educate.' - educates, educated, educative, etc.

What I'd like to know is: is it possible to do this using Lucene's Hunspell implementation (we use a Hunspell dictionary (.dic) and affix file (.aff), so it has to be a Hunspell API)? Lucene's Hunspell API isn't that big, I went through it, and didn't find something suitable.

Nearest I could find on SO was this, but there are no answers related to hunspell.

Answer 1

Thanks to Peter Gromov - s. Lucene Github Issue 11662 LUCENE-10626 - starting from Version 9.5.0 you can generate all word forms by using a new method "getAllWordForms":

import org.apache.lucene.analysis.hunspell.AffixedWord;
import org.apache.lucene.analysis.hunspell.Dictionary;
import org.apache.lucene.analysis.hunspell.Hunspell;

Dictionary dictionary = new Dictionary(...);
Hunspell hObj = new Hunspell(dictionary);
String root = "educate";
List<AffixedWord> affixedWords = hObj.getAllWordForms(root);
for (AffixedWord affixedWord : affixedWords) {
    System.out.println(affixedWord.getWord());
}

If you need to generate all those word forms out of a generated one - for instance "educated" - use this:

import org.apache.lucene.analysis.hunspell.AffixedWord;
import org.apache.lucene.analysis.hunspell.Dictionary;
import org.apache.lucene.analysis.hunspell.Hunspell;

Dictionary dictionary = new Dictionary(...);
Hunspell hObj = new Hunspell(dictionary);
String word = "educated";
List<String> roots = hObj.getRoots(word);
for (String root : roots) {
    List<AffixedWord> affixedWords = hObj.getAllWordForms(root);
    for (AffixedWord affixedWord : affixedWords) {
        System.out.println(affixedWord.getWord());
    }
}

Answer 2

(The original question was about generating all forms for one given word. This answer focuses on the harder problem of generating all forms for all words of a dictionary. I post this here as this is what comes up when searching for the harder problem.)

Update on unmunching

As of 2021, Hunspell provides two tools which are called unmunch and wordforms for generating word forms. Their respective usage is:

# print all forms for all words whose stems are given in `stems.dic`
# and make use of affix rules defined in `affixes.aff`:
unmunch   stems.dic affixes.aff
# print the forms of ONE given word (a single stem with no affix flag)
# which are allowed by the reference dictionary defined by the pair of
# `stems.dic` and `affixes.aff`:
wordforms affixes.aff stems.dic word

So affixes.aff would be given by your language, and stems.dic would be either a reference dictionary for your language, or a custom dictionary with the stems of the new words you want to generate.

Unfortunately, Hunspell’s unmunch is deprecated¹ and does not work properly. It is inherited from MySpell, and my guess is that it does not support all features of Hunspell. When I tried using it with the reference French dictionary (Dicollecte, v7.0), it generated garbage words by applying affix rules it was not supposed to apply (such as: conjugating non-verbs), and failed to generate many expected words. Some of the defects I could pinpoint:

• apparently it does not properly support UTF-8?
• it does not understand FLAG long, which leads to many affix rules being applied out of the blue;
• it wrongly parses metadata attached to stems as affix flags, leading to even more arbitrary rules being applied;
• it does not understand 0 as meaning the empty string and thus generate garbage words containing 0;
• it seems to limit derivations to at most 2 rules, and thus misses many expected words.

wordforms should be more up-to-date, so you might try to emulate unmunch with wordforms (as the README suggests), but the latter only takes one unqualified stem, and compares it against the whole dictionary implied by stems.dic and affixes.aff. This takes a lot of time per stem and, worst, you would have to call wordforms in turn with all the stems in stems.dic. So you would have a quadratic time. For me, with the reference set of affixes for French, this is slow to the point of being unusable—even with only 10 stems! The unusable Bash code is, for illustration:

# /!\ EXTREMELY SLOW
aff='affixes.aff'
dic='stems.dic'
cat "$dic" | while read -r stem ; do # read each stem of the file
    stem="${stem%%/*}" # strip the stem from the optional slash (attached affix flags)
    wordforms "$aff" "$dic" "$stem" # generate all forms for this stem
done \
| sort -u # sort (according to the locale) and remove duplicates

Also, note that wordforms produces bare words, while unmunch was able to attach derived metadata (such as part-of-speech or gender), so with wordforms you lose information (which may or may not matter to you).

The lack of a replacement for unmunch is a known issue. Apparently Hunspell developers will not address it in a predictable future (something about funding?). This has led to several people reimplementing the functionality, you’ll find pointers throughout GitHub issues.

• In 2012 someone wrote an sh/awk script by adapting the source code of wordforms; maybe severely outdated, but I haven’t tried it.
• In 2014 someone wrote another sh/awk script to treat a Hindi dictionary; for me it worked slightly better than the built-in unmunch, as it does support FLAG long, but still has the other defects mentioned above.
• In December 2020 someone wrote a Perl module and a Perl program; looks great, but I’m not sure how to use them.
• In 2021 I wrote my own version, that does fix the defects mentioned above, and works for French. It does not support every feature though, see the comments in the source code for a list of unsupported features.

---

¹ From the repo’s README.

Answer 3

To look for all created forms of one word, assuming en_US.dic contains: word/abc, create a file:

1
word/abc

and save it as word.dic. Use:

unmunch word.dic en_US.aff

and you get all created forms of word.

Answer 4

I think what you're looking for is Hunspell's wordforms command:

Usage: wordforms [-s | -p] dictionary.aff dictionary.dic word
-s: print only suffixed forms
-p: print only prefixed forms

Example:

$ wordforms en_US.aff en_US.dic educate
educating
educated
educate
educates
educates

Read more in the documentation.

Answer 5

Hunspell comes with the unmunch command, which will create all word forms. You can call it like this:

 unmunch en_GB.dic en_GB.aff

Thus you might look in the hunspell source how this is implemented and whether it can be called from outside. The command was a bit buggy last time I checked when used on dictionaries with compounds - in those cases you cannot create all wordforms anyway, as there is an infinite number of them.