PangeaMT with TDA data provides up to 50% more

Valencia, 1st October 2009.
Pangeanic conducted a series of tests with PangeaMT1 for specific language domains by combining its own statistical data with data obtained from TAUS’s TDA during late September. The aim of the test was to prove that increased amounts of trustable, regular data from TDA would help Pangeanic’s own technologies to improve output percentage quality, and to open up new domain developments.
PangeaMT is based on a Moses engine with an applied set of heuristics according to the language.
Data
Three domains were selected for the test in the English-Spanish language pair (no distinction as to Lat.Am/EU), with the following number of files:
- ECH (Electronics-Computer Hardware): 800
 tmx
- MBE (Marketing-Business-Economics): 76 tmx
- SOF (Software): 80 tmx
Valencia, 27th October 2009.

Pangeanic conducted a series of tests with PangeaMT1 for specific language domains by combining its own statistical data with data obtained from TAUS’s TDA during late September. The aim of the test was to prove that increased amounts of trustable, regular data from TDA would help Pangeanic’s own technologies to improve output percentage quality, and to open up new domain developments.

PangeaMT is a custom-built Moses-based engine. Initially developed for internal SMT use in aTMX workflow, Pangeanic is now offering SMT training services and on-demand translation services.

Data

Three domains were selected for the test in the English-Spanish language pair (no distinction as to Lat.Am/EU), with the following number of files:

- ECH (Electronics-Computer Hardware): 800
 tmx

- MBE (Marketing-Business-Economics): 76 tmx

- SOF (Software): 80 tmx





Electronics-Computer Hardware
EnglishSpanish

Sentences (segments)373803
TrainingDifferent file pairs373803

Words3934319 4457167

Vocabulary219789 234920

Average sentence length10,5 11,9

Sentences (segments)2000
TestDifferent file pairs2000

Common pairs with training18

Words2087523564

Perplexity (Trigrams)10077




Software
EnglishSpanish

Sentences (segments)273537
TrainingDifferent file pairs273537

Words31903403710593

Vocabulary117449126331

Average sentence length11,713,6

Sentences (segments)2000
TestDifferent file pairs2000

Common pairs with training12

Words2259326392

Perplexity (Trigrams)11572








MBE
EnglishSpanish

Sentences (segments)71721
TrainingDifferent file pairs71721

Words8732841006106

Vocabulary7639482585

Average sentence length12,214

Sentences (segments)2000
TestDifferent file pairs2000

Common pairs with training2

Words2383827544

Perplexity (Trigrams)243154




Perplexity is a measure that gives us an idea of the complexity of the task and how similar the test is to the training. The higher the perplexity, the higher the difficulty.

Results

Model training + optimization: Moses+MERT
Language models: 5-grams
# TMX files for each category
ECH: 800
MEB: 76
SOF: 80
Translation results English->Spanish
BLEU: ECH: 49.98
MEB: 24.39
SOF: 47.78
Meteor 0.8.3
ECH: 0.4312
MEB: 0.2610
SOF: 0.4377
The best scoring domain is Electronics-Computer Hardware, with almost 50% scoring in BLEU and 43 in METEOR.
Results in Software are also very high (47,78% and 43,7% respectively).
This is a new domain for our development and we have used almost exclusively TDA data plus one of our client’s.
Marketing-Business-Economics lags behind with around 25% in both. Specific, “imaginative” marketing TMs weigh a lot here, and there is less content from TDA. Marketing literature may be closer to human speech. The result also highlights the necessity to count on at least 2M for a customized development (client corpus was under 1M).
Nevertheless, the results surpass our expectations. A 50% BLEU scoring can translate in large increases in language production. Even the 25%, as an initial result for marketing leaves a lot of room for improvement once even more data is available.


Next time you think languages, think Pangeanic

Leave a Reply

Your email address will not be published. Required fields are marked *


− 1 = eight

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>