[ Skip to the content ]

Institute of Formal and Applied Linguistics

at Faculty of Mathematics and Physics, Charles University, Prague, Czech Republic


[ Back to the navigation ]

Publication


Year 2011
Type article
Status published
Language English
Author(s) Berka, Jan Černý, Martin Bojar, Ondřej
Title Quiz-Based Evaluation of Machine Translation
Czech title Vyhodnocování strojového překladu pomocí otázek
Journal The Prague Bulletin of Mathematical Linguistics
Volume 95
Pages range 77-86
Month March
URL http://ufal.mff.cuni.cz/pbml/95/art-berka-cerny-bojar.pdf
Supported by 2010-2012 GPP406/10/P259 (Hybridní frázový a hloubkově-syntaktický strojový překlad) 2011-2013 GAP406/11/1499 (Čeština ve věku strojového překladu) 2009-2012 FP7-ICT-2007-3-231720 (EuroMatrix Plus) 2009-2012 7E09003 (EuroMatrixPlus – Bringing Machine Translation for European Languages to the User)
Czech abstract Článek navrhuje novou metodu ručního vyhodnocování kvality překladu, tzv. evaluaci pomocí otázek.
English abstract This paper proposes a new method of manual evaluation for statistical machine translation, the so-called quiz-based evaluation, estimating whether people are able to extract information from machine-translated texts reliably. We apply the method to two commercial and two experimental MT systems that participated in WMT 2010 in English-to-Czech translation. We report inter-annotator agreement for the evaluation as well as the outcomes of the individual systems. The quiz-based evaluation suggests rather different ranking of the systems compared to the WMT 2010 manual and automatic metrics. We also see that overall, MT quality is becoming acceptable for obtaining information from the text: about 80% of questions can be answered correctly given only machine-translated text.
Specialization linguistics ("jazykověda")
Confidentiality default – not confidential
Open access no
DOI 10.2478/v10108-011-0006-1
ISSN* 0032-6585
Institution* Univerzita Karlova v Praze
Creator: Common Account
Created: 8/30/11 3:02 PM
Modifier: Almighty Admin
Modified: 3/5/12 4:17 PM
***

paperpublic2011-FILE-berka_cerny_bojar_2011-PUBLISHED.pdfapplication/pdf
Content, Design & Functionality: ÚFAL, 2006–2016. Page generated: Thu Apr 26 11:23:18 CEST 2018

[ Back to the navigation ] [ Back to the content ]

100% OpenAIRE compliant