CODEWITHSUNDEEP
c#pdfitexttext-extraction
Euan Gordon
Euan Gordon

Reputation: 108

Extract text with iText with embedded fonts

I am trying to use iTextSharp (v5.5.12.1) to extract the text from the following PDF: https://structure.mil.ru/files/morf/military/files/ENGV_1929.pdf

Unfortunately, it seems like they are using a number of embedded custom fonts, which are defeating me.

For now, I have a working solution using OCR, but the OCR can be imprecise, reading some characters wrongly and also adding additional spaces between characters. It would be ideal if I could extract the text directly.

public static string ExtractTextFromPdf(Stream pdfStream, bool addNewLineBetweenPages = false)
{
    using (PdfReader reader = new PdfReader(pdfStream))
    {
        string text = "";

        for (int i = 1; i <= reader.NumberOfPages; i++)
        {
            text += PdfTextExtractor.GetTextFromPage(reader, i);
            if (addNewLineBetweenPages && i != reader.NumberOfPages)
            {
                text += Environment.NewLine;
            }
        }

        return text;
    }
}

Upvotes: 2

Views: 1196

Answers (1)

mkl
mkl

Reputation: 95918

The issue here is that the glyphs in the embedded font programs have non-standard glyph names (G00, G01, ...) and are identified only by glyph name. Thus, one has to establish a mapping from these glyph names to Unicode characters. One can do so e.g. by inspecting the fonts programs in the PDF (for example with font forge) and visually recognizing the glyphs by name. E.g. like here

AHMHDG+MSTT31c31400

(As you can recognize, there are some gaps for glyphs of the font in question which are not used in the document at hand. Some missing glyphs you can guess, some not.)

Then you have to inject these mappings into iText. As the mappings are hidden (private static members of the GlyphList class), you either have to patch iText itself or use reflection:

void InitializeGlyphs()
{
    FieldInfo names2unicodeFiled = typeof(GlyphList).GetField("names2unicode", BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.Static);
    Dictionary<string, int[]> names2unicode = (Dictionary<string, int[]>) names2unicodeFiled.GetValue(null);
    names2unicode["G03"] = new int[] { ' ' };

    names2unicode["G0A"] = new int[] { '\'' };
    names2unicode["G0B"] = new int[] { '(' };
    names2unicode["G0C"] = new int[] { ')' };

    names2unicode["G0F"] = new int[] { ',' };
    names2unicode["G10"] = new int[] { '-' };
    names2unicode["G11"] = new int[] { '.' };
    names2unicode["G12"] = new int[] { '/' };
    names2unicode["G13"] = new int[] { '0' };
    names2unicode["G14"] = new int[] { '1' };
    names2unicode["G15"] = new int[] { '2' };
    names2unicode["G16"] = new int[] { '3' };
    names2unicode["G17"] = new int[] { '4' };
    names2unicode["G18"] = new int[] { '5' };
    names2unicode["G19"] = new int[] { '6' };
    names2unicode["G1A"] = new int[] { '7' };
    names2unicode["G1B"] = new int[] { '8' };
    names2unicode["G1C"] = new int[] { '9' };
    names2unicode["G1D"] = new int[] { ':' };

    names2unicode["G23"] = new int[] { '@' };
    names2unicode["G24"] = new int[] { 'A' };
    names2unicode["G25"] = new int[] { 'B' };
    names2unicode["G26"] = new int[] { 'C' };
    names2unicode["G27"] = new int[] { 'D' };
    names2unicode["G28"] = new int[] { 'E' };
    names2unicode["G29"] = new int[] { 'F' };
    names2unicode["G2A"] = new int[] { 'G' };
    names2unicode["G2B"] = new int[] { 'H' };
    names2unicode["G2C"] = new int[] { 'I' };
    names2unicode["G2D"] = new int[] { 'J' };
    names2unicode["G2E"] = new int[] { 'K' };
    names2unicode["G2F"] = new int[] { 'L' };
    names2unicode["G30"] = new int[] { 'M' };
    names2unicode["G31"] = new int[] { 'N' };
    names2unicode["G32"] = new int[] { 'O' };
    names2unicode["G33"] = new int[] { 'P' };
    names2unicode["G34"] = new int[] { 'Q' };
    names2unicode["G35"] = new int[] { 'R' };
    names2unicode["G36"] = new int[] { 'S' };
    names2unicode["G37"] = new int[] { 'T' };
    names2unicode["G38"] = new int[] { 'U' };
    names2unicode["G39"] = new int[] { 'V' };
    names2unicode["G3A"] = new int[] { 'W' };
    names2unicode["G3B"] = new int[] { 'X' };
    names2unicode["G3C"] = new int[] { 'Y' };
    names2unicode["G3D"] = new int[] { 'Z' };

    names2unicode["G42"] = new int[] { '_' };

    names2unicode["G44"] = new int[] { 'a' };
    names2unicode["G45"] = new int[] { 'b' };
    names2unicode["G46"] = new int[] { 'c' };
    names2unicode["G46._"] = new int[] { 'c' };
    names2unicode["G47"] = new int[] { 'd' };
    names2unicode["G48"] = new int[] { 'e' };
    names2unicode["G49"] = new int[] { 'f' };
    names2unicode["G4A"] = new int[] { 'g' };
    names2unicode["G4B"] = new int[] { 'h' };
    names2unicode["G4C"] = new int[] { 'i' };
    names2unicode["G4D"] = new int[] { 'j' };
    names2unicode["G4E"] = new int[] { 'k' };
    names2unicode["G4F"] = new int[] { 'l' };
    names2unicode["G50"] = new int[] { 'm' };
    names2unicode["G51"] = new int[] { 'n' };
    names2unicode["G52"] = new int[] { 'o' };
    names2unicode["G53"] = new int[] { 'p' };
    names2unicode["G54"] = new int[] { 'q' };
    names2unicode["G55"] = new int[] { 'r' };
    names2unicode["G56"] = new int[] { 's' };
    names2unicode["G57"] = new int[] { 't' };
    names2unicode["G58"] = new int[] { 'u' };
    names2unicode["G59"] = new int[] { 'v' };
    names2unicode["G5A"] = new int[] { 'w' };
    names2unicode["G5B"] = new int[] { 'x' };
    names2unicode["G5C"] = new int[] { 'y' };
    names2unicode["G5D"] = new int[] { 'z' };

    names2unicode["G62"] = new int[] { 'Ш' };
    names2unicode["G63"] = new int[] { 'Р' };
    names2unicode["G6A"] = new int[] { 'И' };
    names2unicode["G6B"] = new int[] { 'А' };
    names2unicode["G6C"] = new int[] { 'М' };
    names2unicode["G6D"] = new int[] { 'в' };
    names2unicode["G6E"] = new int[] { 'Ф' };
    names2unicode["G70"] = new int[] { 'Е' };
    names2unicode["G72"] = new int[] { 'Б' };
    names2unicode["G73"] = new int[] { 'Н' };
    names2unicode["G76"] = new int[] { 'С' };
    names2unicode["G7A"] = new int[] { 'К' };
    names2unicode["G7B"] = new int[] { 'В' };
    names2unicode["G7C"] = new int[] { 'О' };
    names2unicode["G7D"] = new int[] { 'к' };
    names2unicode["G7E"] = new int[] { 'З' };
    names2unicode["G80"] = new int[] { 'Г' };
    names2unicode["G81"] = new int[] { 'П' };
    names2unicode["G82"] = new int[] { 'у' };
    names2unicode["G85"] = new int[] { '»' };
    names2unicode["G88"] = new int[] { 'т' };
    names2unicode["G8D"] = new int[] { '’' };
    names2unicode["G90"] = new int[] { 'У' };
    names2unicode["G91"] = new int[] { 'Т' };
    names2unicode["GA1"] = new int[] { 'Ц' };
    names2unicode["GA2"] = new int[] { '№' };
    names2unicode["GAA"] = new int[] { 'э' };
    names2unicode["GAB"] = new int[] { 'я' };
    names2unicode["GAC"] = new int[] { 'і' };
    names2unicode["GAD"] = new int[] { 'б' };
    names2unicode["GAE"] = new int[] { 'й' };
    names2unicode["GAF"] = new int[] { 'р' };
    names2unicode["GB0"] = new int[] { 'с' };
    names2unicode["GB2"] = new int[] { 'х' };
    names2unicode["GB5"] = new int[] { '“' };
    names2unicode["GB9"] = new int[] { 'п' };
    names2unicode["GBA"] = new int[] { 'о' };
    names2unicode["GBD"] = new int[] { '«' };
    names2unicode["GC1"] = new int[] { 'ф' };
    names2unicode["GC8"] = new int[] { 'а' };
    names2unicode["GCB"] = new int[] { 'е' };
    names2unicode["GCE"] = new int[] { 'ж' };
    names2unicode["GCF"] = new int[] { 'з' };
    names2unicode["GD2"] = new int[] { 'и' };
    names2unicode["GD3"] = new int[] { 'н' };
    names2unicode["GDC"] = new int[] { '–' };
    names2unicode["GE3"] = new int[] { 'л' };
}

After executing that method you can extract the text using your method:

InitializeGlyphs();

using (FileStream pdfStream = new FileStream(@"ENGV_1929.pdf", FileMode.Open))
{
    string result = ExtractTextFromPdf(pdfStream, true);
    File.WriteAllText(@"ENGV_1929.txt", result);
    Console.WriteLine("\n\nENGV_1929.pdf\n");
    Console.WriteLine(result);
}

The result:

From Notices to Mariners
Edition No 29/2019
(English version)
Notiсes to Mariners from Seсtion II «Сharts Сorreсtion», based on the original sourсe information, and
NAVAREA XIII, XX and XXI navigational warnings are reprinted hereunder in English. Original Notiсes to
Mariners from Seсtion I «Misсellaneous Navigational Information» and from Seсtion III «Nautiсal
Publiсations Сorreсtion» may be only briefly annotated and/or a referenсe may be made to Notiсes from
other Seсtions. Information from Seсtion IV «Сatalogues of Сharts and Nautiсal Publiсations Сorreсtion»
сonсerning the issue of сharts and publiсations is presented with details.
Digital analogue of English version of the extracts from original Russian Notices to Mariners is available
by: http://structure.mil.ru/structure/forces/hydrographic/info/notices.htm
СНАRTS СОRRЕСTIОN
Вarents Sea
 3493 Сharts 18012, 17052, 15005, 15004
Amend 1. Light to light Fl G 4s 1M at
 front leading lightbeacon 69111’32.2“N 33129’48.0“E
 2. Light to light Fl G 4s 1M at
 rear leading lightbeacon 69111’34.85“N 33129’44.25“E
 Cancel coastal warning
 MURMANSK 71/19
...

Beware, you will see that quite often instead of a Latin character a similar looking Cyrillic character is used. Apparently the document was created manually by someone who didn't consider typographical correctness very important..

So, if you want to search in the text, you should first normalize the text and your search terms (e.g. use the same character for the Latin 'c' and the Cyrillic 'с').

Upvotes: 1

Related Questions