CODEWITHSUNDEEP
pythonfileformattinglanguage-translationmachine-translation
Alexander R Johansen
Alexander R Johansen

Reputation: 2817

Loading large Translate Translation Memory Exchange (TMX) files

How do I load TMX files without exploding memory?

When loading in ParaCrawl datasets I would like to use the TMX format. In order to load the TMX file I have been using this [SO post] (TMX(Translation Memory eXchange) files in python)

However, for something like Swedish to English, simply loading in the dataset takes up 66GB of memory, and with the size of the datasets I have projected that it would take almost 500GB of memory to load in Spanish to English. The largest AWS instance is 128 GB memory, it seems like I am doing something wrong?

Upvotes: 0

Views: 1119

Answers (1)

MrKaikev
MrKaikev

Reputation: 265

Using an XML parser with streaming seems to work quite well. There are 3 levels within the TMX file <tmx><body><tu>.

import xmltodict
from gzip import GzipFile


language_pairs = []

def get_lang_pair(_, tree):
    lang_pair = {}
    for elem in tree['tuv']:
        language = elem['@xml:lang']
        text = elem['seg']
        lang_pair[language] = text

    language_pairs.append(lang_pair)
    return True

# <tmx>
# <body>
# <tu>
xmltodict.parse(
    GzipFile('en-da.tmx.gz'),
    item_depth=3, item_callback=get_lang_pair,
)

output:

[{'en': 'Galaxy S9 price: $799.99 full price or $33.33/month for 24 months with 0 percent APR Galaxy S9 Plus price: $929.99 full price or $38.74/month for 24 months with 0 percent APR Pre-order launch date: 1 March, 9:01 p.m Pacific Launch date: 16 March Trade-in promo: Up to $350 credit (iPhone X, 8, 8 Plus; Samsung GS8, GS8+, Note8) Other promos: $150 prepaid MasterCard for porting in a number, $10 off accessories for Verizon Up customers, 50 percent off Samsung tablet with 2-year agreement, $100 off Gear S3, and Samsung fast charge power bundle for $99.99',
  'da': 'Galaxy S9 pris: $799.99 fuld pris eller $33.33/måned i 24 måneder med 0% p.a Galaxy S9 Plus pris: $929.99 fuld pris eller $38.74/måned i 24 måneder med 0% p.a Forud for lanceringen dato: 1 Marts, 9:01 p.m Pacific Lanceringen dato: 16 Marts Handel-promo: Op til $350 kredit (iPhone X, 8, 8, Plus, Samsung GS8, GS8+, Note8) Andre promotions: $150 prepaid MasterCard for portering af et nummer, $10 off tilbehør til Verizon kunder Op, 50% off Samsung tablet med 2-årig aftale, $100 off Gear S3, og Samsung hurtig opladning power bundle for $99.99'},
 {'en': 'Room facilities: Air conditioning, Bath or Shower, Cable Channels, Desk, Dining table, Entire unit located on ground floor, Fan, Flat-screen TV, Free toiletries, Heating, Linen, Microwave, Private bathroom, Private entrance, Refrigerator, Seating Area, Shower, TV, Tile/Marble floor, Toilet, Toilet paper, Towels Max persons: 3 Size of the room: 13.94 m² Beds: 1x Large bed(s) (King size) bed Rate: US$ 55.71 ...',
  'da': 'Værelses faciliteter: Aircondition, Badekar eller bruser, Bruser, Egen indgang, Eget badeværelse, Fjernsyn, Fladskærms-tv, Flise-/marmorgulv, Gratis toiletartikler, Hele boligen er beliggende i stueetagen, Håndklæder, Kabelkanaler, Køleskab, Loftvifte, Mikrobølgeovn, Sengelinned, Siddekrog, Skrivebord, Spisebord, Toilet, Toiletpapir , Varme Max antal personer: 3 Størrelsen af \u200b\u200brummet: 13.94 m² Senge: 1x Large bed(s) (King size) bed Pris: US$ 55.71 ...'},

Upvotes: 4

Related Questions