PangeaMT with TDA data provides up to 50% more

Valencia, 1st October 2009.
Pangeanic conducted a series of tests with PangeaMT1 for specific language domains by combining its own statistical data with data obtained from TAUS’s TDA during late September. The aim of the test was to prove that increased amounts of trustable, regular data from TDA would help Pangeanic’s own technologies to improve output percentage quality, and to open up new domain developments.
PangeaMT is based on a Moses engine with an applied set of heuristics according to the language.
Data
Three domains were selected for the test in the English-Spanish language pair (no distinction as to Lat.Am/EU), with the following number of files:
– ECH (Electronics-Computer Hardware): 800
 tmx
– MBE (Marketing-Business-Economics): 76 tmx
– SOF (Software): 80 tmx
Valencia, 27th October 2009.

Pangeanic conducted a series of tests with PangeaMT1 for specific language domains by combining its own statistical data with data obtained from TAUS’s TDA during late September. The aim of the test was to prove that increased amounts of trustable, regular data from TDA would help Pangeanic’s own technologies to improve output percentage quality, and to open up new domain developments.

PangeaMT is a custom-built Moses-based engine. Initially developed for internal SMT use in aTMX workflow, Pangeanic is now offering SMT training services and on-demand translation services.

Data

Three domains were selected for the test in the English-Spanish language pair (no distinction as to Lat.Am/EU), with the following number of files:

– ECH (Electronics-Computer Hardware): 800
 tmx

– MBE (Marketing-Business-Economics): 76 tmx

– SOF (Software): 80 tmx





Electronics-Computer Hardware
English Spanish

Sentences (segments) 373803
Training Different file pairs 373803

Words 3934319 4457167

Vocabulary 219789 234920

Average sentence length 10,5 11,9

Sentences (segments) 2000
Test Different file pairs 2000

Common pairs with training 18

Words 20875 23564

Perplexity (Trigrams) 100 77




Software
English Spanish

Sentences (segments) 273537
Training Different file pairs 273537

Words 3190340 3710593

Vocabulary 117449 126331

Average sentence length 11,7 13,6

Sentences (segments) 2000
Test Different file pairs 2000

Common pairs with training 12

Words 22593 26392

Perplexity (Trigrams) 115 72








MBE
English Spanish

Sentences (segments) 71721
Training Different file pairs 71721

Words 873284 1006106

Vocabulary 76394 82585

Average sentence length 12,2 14

Sentences (segments) 2000
Test Different file pairs 2000

Common pairs with training 2

Words 23838 27544

Perplexity (Trigrams) 243 154




Perplexity is a measure that gives us an idea of the complexity of the task and how similar the test is to the training. The higher the perplexity, the higher the difficulty.

Results

Model training + optimization: Moses+MERT
Language models: 5-grams
# TMX files for each category
ECH: 800
MEB: 76
SOF: 80
Translation results English->Spanish
BLEU: ECH: 49.98
MEB: 24.39
SOF: 47.78
Meteor 0.8.3
ECH: 0.4312
MEB: 0.2610
SOF: 0.4377
The best scoring domain is Electronics-Computer Hardware, with almost 50% scoring in BLEU and 43 in METEOR.
Results in Software are also very high (47,78% and 43,7% respectively).
This is a new domain for our development and we have used almost exclusively TDA data plus one of our client’s.
Marketing-Business-Economics lags behind with around 25% in both. Specific, “imaginative” marketing TMs weigh a lot here, and there is less content from TDA. Marketing literature may be closer to human speech. The result also highlights the necessity to count on at least 2M for a customized development (client corpus was under 1M).
Nevertheless, the results surpass our expectations. A 50% BLEU scoring can translate in large increases in language production. Even the 25%, as an initial result for marketing leaves a lot of room for improvement once even more data is available.


Next time you think languages, think Pangeanic

Leave a Reply

Your email address will not be published. Required fields are marked *