Warning: This is only a draft of a shared task that may or may not be realized in future!
Multilingual Dependency Parsing Shared Task
Introduction
Many years have elapsed since the last CoNLL shared task focused on syntactic dependency parsing, and a lot of research has been done since then. Although researchers generally care to report their results in a way that makes them comparable to previous results, full comparability is often difficult or impossible to achieve, and it is not at all clear what the current state-of-the-art approaches to parsing are.
Moreover, the CoNLL treebanks, which are typically used for parser evaluation, are becoming outdated. For many of them, there exists newer versions of the original treebanks, which may be larger, have a higher quality, and provide a richer annotation (actually, already the original conversions of treebanks for the CoNLL shared tasks were sometimes lossy). Many new treebanks have also emerged, thus significantly widening the range of language families represented by at least one treebank. And most importantly, treebank harmonization has been introduced, allowing to abstract from annotation style differences of individual treebanks; the most prominent harmonized resource are the Universal Dependencies, which currently contain 18 dependency treebanks annotated in the same way on both morphological and syntactit level. This is a great leap forward in treebank standardization (compared to CoNLL which "only" standardized the data format), enabling real multilingual approches to parsing.
And finally, new parsing tasks have emerged since the CoNLL tasks, focused on analyzing languages for which there are little or no annotated resources. Various unsupervised and semisupervised techniques are employed in such scenarios, but the comparability of such approaches is even more difficult than with the supervised one. Therefore, we feel an urging need for a shared task focused on these approaches, which would reliably compare the existing approaches, identify the state-of-the-art methods, and motivate further research in this area.
Tasks
In each of the tasks, the parsing system must be fully language-neutral: language-specific adjustments may be made, but the system must be able to parse any language (the 18 task languages as well as any other language) fully automatically, without a need for manual adjustments. Ideally, the adjustments should be made automatically based on the input data, but it is also allowed to use other resources that are available for a large number of languages: WALS, monolingual data, parallel data (with English as the other language), Wikipedia articles, etc. Dictionaries without morphological information (e.g. POS) may also be used, but dictionaries with morphological information are regarded as morphologically annotated corpora and are thus not allowed in some of the tasks, either only for the target language (semisupervised) or for all the languages (unsupervised); the dictionaries are allowed if the morphological information is not used.
It is forbidden to use other syntactically annotated resources than the training sections of the task treebanks.
Supervised parsing
Parse the test section of the target treebank using any morphologically or syntactically annotated resources in any language.
The classical parsing task, similar to the past CoNLL parsing tasks. The default approach is to train a parser on the training section of the target treebank and apply it to the test section of the target treebank. Additional resources may be used, except for syntactically annotated resources other than the training section of the target treebank.
Semisupervised parsing
Parse the test section of the target treebank without using any morphologically or syntactically annotated resources in the target language, but using any resources in other languages.
The basic approach is the delexicalized parser transfer, where a treebank for another language is stripped of word forms, a parser is trained on it, and applied to the target data. Many improvements have been introduced, often introducing relexicalization using e.g. parallel data.
Unsupervised parsing
Parse the test section of the target treebank without using any morphologically or syntactically annotated resources in any language.
Subtasks
Each of the tasks has two implicit subtasks:
- Gold morphology: The gold morphological annotation (lemma, POS tag, morpho features) of the target language treebank (both train and test section) may be used.
- Automatic morphology: The morphological annotation of the target language treebank test section must not be used; if the participant's system requires this annotation on input, the participant must induce it himself, using only the allowed resources. Morphologically annotated target language data may be used to train a tagger in the supervised task, but not in the other tasks since using such data is forbidden there. In the semisupervised task, one may use e.g. parallel data and POS tag projection. In the unsupervised parsing, one may use e.g. unsupervised POS tags.
Dataset
The dataset used for the task is the Universal Dependencies 1.1 treebanks, available at http://hdl.handle.net/11234/LRT-1478; information about the dataset format and annotation can be found at http://universaldependencies.github.io/docs/.
All 18 languages in the dataset are the shared task languages, and each submitted system must be applied to all these 18 languages: Basque (eu), Bulgarian (bg), Croatian (hr), Czech (cs), Danish (da), English (en), Finnish (fi), French (fr), German (de), Modern Greek (el), Hebrew (he), Hungarian (hu), Indonesian (id), Irish (ga), Italian (it), Persian (fa), Spanish (es), Swedish (sv).
Evaluation
The participants must submit the parsed test sections of the treebanks, parsed by their systems, by sending an e-mail to the task organizer (rosa@ufal.mff.cuni.cz), with the outputs for each subtask in a separate zip file.
The zip file must contain 18 files in the CONLL-U format, named langcode.conllu, where langcode is the ISO 639-1 two-letter code of the target language -- e.g. bg.conllu or cs.conllu (the ISO 639-1 codes of all the languages are listed in the previous section).
The name of the zip file must conform to the following regular expression:
^{sup|unsup|cross}\.{gold|auto}\.[a-zA-Z0-9_]+\.zip$
where the last part is an identifier of the submitted system, chosen by the participant to be unique (but may be identical if the same system is submitted for multiple subtasks). So a submission to the unsupervised parsing task using gold morphological annotation may be called e.g.:
unsup.gold.rosa_DMV.zip
The results will be evaluated using both UAS and LAS, both including and disregarding punctuation; thus, four scores will be reported for each result. The script that will be used for the evaluation is eval.pl.
Important dates
- 15th June 2015: submission of datasets
- 15th July 2015: submission of system description papers (4-8 pages, ACL style)
- 15th September 2015: notification of acceptance
- 15th November 2015: submission of camera-ready papers
- 15th December 2015: presentation of the papers on the Crosslingual Dependency Parsing Workshop in Prague
Contact
In case of any question, feel free to contact the shared task organizer, Rudolf Rosa, on rosa@ufal.mff.cuni.cz.