[ÚFAL logo]

CzEng 1.0 (Czech-English Parallel Corpus, version 1.0)

Ondřej Bojar, Zdeněk Žabokrtský
in cooperation with
Ondřej Dušek, Petra Galuščáková, Martin Majliš, David Mareček,
Jiří Maršík, Michal Novák, Martin Popel, and Aleš Tamchyna

Introduction

CzEng 1.0 is the fourth release of a sentence-parallel Czech-English corpus compiled at the Institute of Formal and Applied Linguistics (ÚFAL) freely available for non-commercial research purposes.

CzEng 1.0 contains 15 million parallel sentences (233 million English and 206 million Czech tokens) from seven different types of sources automatically annotated at surface and deep (a- and t-) layers of syntactic representation. The number of sentences and nodes of a given layer and language per data source is given in the following table:

Czech English
Source Domain Sentences a-layer t-layer a-layer t-layer
Fiction 4,335,183 57,176,714 41,142,003 64,264,229 38,690,193
EU Legislation 3,992,551 78,022,137 56,446,093 87,488,870 52,717,871
Movie Subtitles 3,076,887 19,571,940 14,614,899 23,353,842 14,917,777
Parallel Web Pages 1,883,804 30,891,696 23,140,879 35,454,657 22,057,255
Technical Documentation 1,613,297 16,015,336 11,941,650 16,836,098 11,207,157
News 201,103 4,280,039 3,207,858 4,736,751 2,963,451
Project Navajo 33,301 484,453 363,317 556,702 343,649
Total 15,136,126 206,442,315 150,856,699 232,691,149 142,897,353

All further details about CzEng 1.0 will be available in a paper (in preparation). The core design is siminar to CzEng 0.9, see the paper cited below. The data formats were slightly updated since CzEng 0.9, see the summary below.

CzEng 1.0 is not error-free. Here is the list of known issues.

Citing CzEng 1.0

If you make use of CzEng 1.0 data, please cite the following paper:

To improve the reproducibility of your results, please indicate which sections have you used for training and/or evaluation.

Register

To download CzEng 1.0, you have to register by filling in the following form. Within a week we will send you a unique username to access the files.

Name:
E-mail:
Institution:
Country:

I certify that I will use CzEng 1.0 only for research and non-commercial purposes.

Download

After the registration, you received a unique username. The unique username and a shared password "czeng" will be requested at the following links.

To simplify the download, the 100 sections of CzEng are grouped to packs of 10 sections each (the dev and test sections are separate). CzEng 1.0 is shuffled so you may wish to use just one of the packs for your experiments as a random sample.

File Format Avg. Download Size Training Sections DevTest Section EvalTest Section
Treex Format, rich XML 4.7 GB each 0* 1* 2* 3* 4* 5* 6* 7* 8* 9[0-7] 98 (~480 MB) 99 (~480 MB)
Export Format, rich factored plaintext 900 MB each 0* 1* 2* 3* 4* 5* 6* 7* 8* 9[0-7] 98 (88 MB) 99 (88 MB)
Plaintext, untokenized 115 MB each 0* 1* 2* 3* 4* 5* 6* 7* 8* 9[0-7] 98 (12 MB) 99 (12 MB)

Tip for Linux wget tool: Use the flags --user=YOUR-USERNAME --password=czeng to pass the authorization check. Use the flag --continue to continue an interrupted transfer.

Remark for WMT12 participants: There is no intersection between CzEng 1.0 data and WMT12 dev and evaluation data. However, WMT12 participants are kindly asked to use only the Training sections of CzEng and avoid DevTest and EvalTest sections (packs 98 and 99), so that there remain some held-out data for the evaluation of future experiments. In any case, please indicate clearly how much data and from which sections of CzEng 1.0 you have eventually used.

Brief Note on File Formats

CzEng 1.0 is shuffled at the level of "blocks", sequences of not more than 15 consecutive sentences from one source. The original documents thus cannot be reconstructed but some information for cross-sentence phenomena is preserved. Specifically, the Treex format of CzEng includes Czech grammatical and textual co-reference links that do span sentence boundaries.

Individual text "blocks" are combined to numbered files, each file holds about 200 sentence pairs.

Each "block" comes from one of the domains (EU Legislation, etc.) and the domain is indicated in the sentence ID.

Treex Format

The primary format of CzEng is Treex XML, a successor of the TMT format of TectoMT used in CzEng 0.9.

Treex XML can be processed using the Treex platform implemented in Perl and available on CPAN.

The best option for manual inspection of the data is the tree editor TrEd, which can read Treex XML using ttred wrapper provided in Treex.

Export Format

To facilitate access to most of the automatic rich annotation of CzEng 1.0 without XML hassle, we provide a simple "factored" line-oriented export format. Note that e.g. named entities or co-reference links are not available in export format at all.

Column Sample Explanation
1 subtitles-b2-00train-f00001-s8 ID specifying the domain, block number, train/dev/test section, file number and sentence within the file.
2 0.99261036 Filter score indicating the quality of the sentence pair. The score of 1 is perfect pair, pairs below 0.3 are removed.
Czech
3 Zachránil|zachránit_:W|VpYS---XR-AA---|1|0|Pred mi|já|PH-S3--1-------|2|1|Obj můj|můj_^(přivlast.)|PSYS1-S1-------|3|5|Atr milovaný|milovaný_^(*2t)|AAIS1----1A----|4|5|Atr krk|krk|NNIS1-----A----|5|1|Obj .|.|Z:----... Czech a-layer (surface-syntactic tree) in factored form: word-form|lemma|morphological-tag|index-in-sentence|index-of-governor|syntactic-function.
4 zachránit|PRED|1|0|complex|v:fin|v|-|neg0|ant|ind|decl|-|cpl|-|-|disp0|-|it0|-|-|res0|-|-|1|-|- #PersPron|ADDR|2|1|complex|n:3|n.pron.def.pers|sg|-|-|-|-|-|-|-|-|-|nr|-|1|basic|-|-|-|-|-|- ... Czech t-layer (tectogrammatical tree): t-lemma|functor|index-in-tree|index-of-governor|nodetype|formeme|semantic-part-of-speech|... and many detailed t-layer attributes.
5 0-0 1-1 2-2 3-3 4-4 Correspondence between Czech a-layer and t-layer for content words. Indexed from 0.
6 Correspondence between Czech a-layer and t-layer for auxiliary words. Indexed from 0.
English
7 He|he|PRP|1|2|Sb saved|save|VBD|2|0|Pred my|my|PRP$|3|4|Atr ever-lovin|ever-lovin|NN|4|6|Atr '|'|''|5|6|AuxG neck|neck|NN|6|2|Obj .|.|.|7|0|AuxK English a-layer (surface-syntactic tree) in factored form: word-form|lemma|tag|index-in-sentence|index-of-governor|syntactic-function.
8 #PersPron|ACT|1|2|complex|n:subj|n.pron.def.pers|sg|-|-|-|-|-|-|-|-|-|inan|-|3|-|-|-|-|0|-|- save|PRED|2|0|complex|v:fin|v|-|neg0|ant|ind|decl|-|-|-|-|disp0|-|it0|-|-|res0|-|-|1|-|- #PersPron|APP|3|4|complex|n:poss|n.pron... English t-layer (tectogrammatical tree): t-lemma|functor|index-in-tree|index-of-governor|nodetype|formeme|semantic-part-of-speech|... and many detailed t-layer attributes.
9 0-0 1-1 2-2 3-3 5-4 Correspondence between English a-layer and t-layer for content words. Indexed from 0.
10 4-4 Correspondence between English a-layer and t-layer for auxiliary words. Indexed from 0.
Cross-Language Alignments Between Surface Czech and English
Always indexed from 0, Czech-English.
11 0-1 1-2 2-2 3-3 4-5 5-6 GIZA++ alignments "there" for cs2en.
12 0-0 0-1 2-2 3-3 3-4 4-5 5-6 GIZA++ alignments "back" for cs2en.
13 0-0 0-1 1-2 2-2 3-3 3-4 4-5 5-6 GIZA++ alignments symmetrized using grow-diag-final-and for cs2en.
14 0-0 0-1 1-2 2-2 3-3 3-4 4-5 5-6 GIZA++ alignments symmetrized using grow-diag-final-and for en2cs (not the inverse of column 13).
Cross-Language Alignments Between T-Layer Czech and English
Always indexed from 0, Czech-English.
15 0-1 1-2 2-2 3-3 4-4 T-alignment "there" for cs2en.
16 0-0 0-1 2-2 3-3 4-4 T-alignment "back" for cs2en.
17 Additional rule-based t-alignment linking esp. generated nodes like #Perspron;.

Plaintext Format

The plaintext format is very simple, four tab-delimited columns containing:

  1. Sentence pair ID
  2. Filter score
  3. Czech, not tokenized.
  4. English, not tokenized.

Online access via PML-TQ

A sample of the first 151 212 sentences (section 00train) is indexed via the PML-TQ service and you can query it here with username public and password public. See the documentation for PML-TQ syntax. A very simple query a-node [lemma="go"] will find all occurences of the word go. Note the gray line with filename "< - > f00086.treex.gz (96/213)" – by clicking on the arrows (< and >) you can browse to the previous/following sentence.

Sample sentence

Sample sentence annotated on the a-layer and t-layer (with named entities marked in the n-tree).

en: Shaw cursed himself under his breath for not starting here.

cs: Shaw se v duchu na sebe zlobil, že nezačal právě tady.

Image of a sample tree

Acknowledgement

CzEng 1.0 release was funded by the grants EuroMatrix Plus (FP7-ICT-2007-3-231720 of the EU and 7E09003+7E11051 of the Ministry of Education, Youth and Sports of the Czech Republic), Faust (FP7-ICT-2009-4-247762 of the EU and 7E11041 of the Ministry of Education, Youth and Sports of the Czech Republic), GAČR P406/10/P259, GAUK 116310, and GAUK 4226/2011.
Institute of Formal and Applied Linguistics (ÚFAL)
Ondřej Bojar, bojar <at> ufal.mff.cuni.cz
$Id: index.html 1219 2012-07-26 09:22:55Z bojar $