CRAC 2022 Shared Task on Multilingual Coreference Resolution


Overview

Coreference resolution is the task of clustering together multiple mentions of the same entity appearing in a textual document (e.g. Joe Biden, the U.S. President and he). This CodaLab-powered shared task deals with multilingual coreference resolution and is associated with the CRAC 2022 Workshop (the Fifth Workshop on Computational Models of Reference, Anaphora and Coreference) held at COLING 2022.

Important Dates

  • April 11, 2022 - start of the development phase:
    • training and devevelopment data available at the LINDAT/CLARIAH-CZ repository
    • evaluation tool available at GitHub
    • baseline system available at GitHub
    • registration form available (please register asap if interested in the shared task)
  • May 17, 2022 - development evaluation possible via CodaLab
  • June 1, 2022 - start of the evaluation phase (test evaluation via CodaLab)
  • June 8, 2022 (AoE) - end of evaluation phase
  • July 11, 2022 - submission of system description papers
  • see CRAC's Important Dates for the dates of acceptance notification and camera-ready papers
  • October 16-17, 2022 – the CRAC workshop at COLING

Background

Recently, inspired by the Universal Dependencies initiative (UD) [1], the coreference community has started discussions on establishing a universal annotation scheme and using it to harmonize existing corpora. The discussions at the CRAC 2020 workshop led to proposing the Universal Anaphora initiative. One of the lines of effort related to Universal Anaphora resulted in CorefUD, which is a multilingual collection of coreference data resources harmonized under a common scheme [2]. The public edition of CorefUD 1.0 contains 13 datasets for 10 languages, namely Catalan, Czech (2×), English (2×), French, German (2×), Hungarian, Lithuanian, Polish, Russian, and Spanish. The CRAC 2022 shared task deals with coreference resolution in all these languages.

The file format used in CorefUD 1.0 represents coreference using the bracketing notation inspired by the CoNLL-2011 and CoNLL-2012 shared tasks [3], and inserts it into the MISC column of the CoNLL-U, the file format used in UD. The content of the other columns is fully compatible with morphological and syntactic annotations of the UD framework in CorefUD (with, for instance, automatically parsed trees added to resources that miss manual syntactic annotations). Thus, the shared task participant can easily employ UD-style morphosyntactic features for coreference prediction for all resources in a unified way, if they want to (pilot studies of the relation between coreference and dependency syntax can be found in [4] and [5]). CorefUD tokenization is UD-compliant, too.

Task Description

The main rules of the CRAC 2022 shared task are the following:
  • Shared task participants are supposed to both (a) identify mentions in texts and (b) predict which mentions belong to the same coreference cluster (i.e., refer to the same entity or event).
  • Training and development data will be published first; evaluation data (without gold annotations) will be publicly available only after the beginning of the evaluation phase and must not be used for improving the models.
  • Participants are expected to deliver their submissions exclusively via CodaLab; a submission must have the form of a zip file containing test set files with predicted coreference, ideally for all 13 CorefUD datasets; however, participants who are unable to predict coreference for all CorefUD datasets are encouraged to submit at least a subset of test set files.
  • Technically, ‘files with predicted coreference’ means that coreference attributes using the CorefUD notation are filled into the MISC column of test set CoNLL-U files.
  • In this shared task, only identity coreference is supposed to be predicted (even if some of the CorefUD datasets contain annotation of bridging).
  • There is a single official evaluation criterion that will be used for the main ranking of all submissions within the evaluation phase. In other words, there are no subtasks delimited within this shared task. We define the criterion as the arithmetic mean (macro-average) of the CoNLL score (an average of the F1 values of MUC, B-cubed and CEAFe scores) [6, 3] across the 13 datasets.
  • Even if there are no subtasks declared, additional evaluation criteria might be evaluated for all submissions and presented by the organizers (for instance, secondary rankings of submissions according to scores reached for individual languages).
  • A deep-learning-based baseline system [7] will be available to participants, and it is up to their decision whether they start developing their system from scratch, or by incremental improvements of this baseline.
  • After the evaluation period, participants will be invited to submit their system description papers to the CRAC 2022 workshop.

References

  • [1] De Marneffe, M.-C., Manning, C. D., Nivre, J., & Zeman, D. (2021). Universal Dependencies. Computational Linguistics, 47(2), 255-308.
  • [2] Nedoluzhko, A., Novák, M., Popel, M., Žabokrtský, Z. & and Zeman, D. (2021). Coreference meets Universal Dependencies – a pilot experiment on harmonizing coreference datasets for 11 languages. Technical Report 66, ÚFAL MFF UK, Praha, Czechia.
  • [3] Pradhan, S., Moschitti, A., Xue, N., Uryupina, O., & Zhang, Y. (2012). CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL-Shared Task (pp. 1-40).
  • [4] Popel, M., Žabokrtský, Z., Nedoluzhko, A., Novák, M., & Zeman, D. (2021). Do UD Trees Match Mention Spans in Coreference Annotations? In Findings of the Association for Computational Linguistics: EMNLP 2021 (pp. 3570-3576).
  • [5] Nedoluzhko, A., Novák, M., Popel, M., Žabokrtský, Z., & Zeman, D. (2021). Is one head enough? Mention heads in coreference annotations compared with UD-style heads. In Proceedings of the Sixth International Conference on Dependency Linguistics (Depling, SyntaxFest 2021) (pp. 101-114).
  • [6] Denis, P. & Baldridge, J. (2009). Global joint models for coreference resolution and named entity classification. Procesamiento del lenguaje natural, Nº. 42, (pp. 87-96).
  • [7] Pražák, O., Konopík, M., & Sido, J. (2021). Multilingual Coreference Resolution with Harmonized Annotations. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021). 2021.

Organizers

  • Charles University (Prague, Czechia): Anna Nedoluzhko, Michal Novák, Martin Popel, Zdeněk Žabokrtský, Daniel Zeman

  • Polish Academy of Sciences (Warsaw, Poland): Maciej Ogrodniczuk

  • Georgetown University (Washington D.C., USA): Yilun Zhu

  • University of West Bohemia (Pilsen, Czechia): Miloslav Konopík, Ondřej Pražák, Jakub Sido

Registration Form and Contact to Organizers

If you are interested in participating in this shared task, please fill the registration form as soon as possible.

Technically, this registration will not be connected with participants' CodaLab accounts in any way. In other words, it will be possible to upload your CodaLab submissions without being registered here. However, we strongly recommend that at least one person from each participating team fills this registration form so that we can keep you informed about all updates regarding the shared task.

In addition, you can send any questions about the shared task to the organizers via corefud@googlegroups.com.

Acknowledgements

This shared task is supported by the Grants No. 20-16819X (LUSyD) of the Czech Science Foundation, and LM2018101 (LINDAT/CLARIAH-CZ) of the Ministry of Education, Youth, and Sports of the Czech Republic.

Data and Evaluation

CorefUD datasets

The public edition of CorefUD 1.0 data is used in this shared task, both for training and evaluation purposes. CorefUD 1.0 is a collection of previously existing datasets annotated with coreference, converted into a common annotation scheme. The datasets are enriched with automatic morphological and syntactic annotations that are fully compliant with the standards of the Universal Dependencies project. All the datasets are stored in the CoNLL-U format, with coreference-specific information captured in the MISC column.

The public edition of CorefUD 1.0 contains 13 datasets for 10 languages, labeled as follows:

  1. Catalan-AnCora, based on the Catalan part of Coreferentially annotated corpus AnCora,
  2. Czech-PCEDT, based on the Czech part of the Prague Czech-English Dependency Treebank,
  3. Czech-PDT, based on the Prague Dependency Treebank,
  4. English-GUM, based on the Georgetown University Multilayer Corpus,
  5. English-ParCorFull, based on the English part of ParCorFull,
  6. French-Democrat, based on the Democrat corpus,
  7. German-ParCorFull, based on the German part of ParCorFull,
  8. German-PotsdamCC, based on the Potsdam Commentary Corpus,
  9. Hungarian-SzegedKoref, based on the Hungarian Coreference Corpus SzegedKoref,
  10. Lithuanian-LCC, based on the Lithuanian Coreference Corpus,
  11. Polish-PCC, based on the Polish Coreference Corpus,
  12. Russian-RuCor, based on the Russian Coreference Corpus RuCor,
  13. Spanish-AnCora, based on the Spanish part of Coreferentially annotated corpus AnCora.

(There is also a non-public edition of CorefUD 1.0, containing 4 more datasets, however, they cannot be used for this shared task purposes because of their license limitations.)

File format

The full specification of the CoNLL-U format is available at the website of Universal Dependencies. In a nutshell: every token has its own line; lines starting with # are sentence-level comments, and empty lines terminate a sentence. Regular token lines start with an integer number. There are also lines starting with intervals (e.g. 4-5), which introduce what UD calls “multi-word tokens”; these lines must be preserved in the output but otherwise the participants do not have to care about them (coreference annotation does not occur on them). Finally, there are also lines starting with decimal numbers (e.g. 2.1), which correspond to empty nodes in the dependency graph; these nodes may represent zero mentions and may contain coreference annotation. Every token/node line contains 10 tab-separated fields (columns). The first column is the numeric ID of the token/node, the next column contains the word FORM; any coreference annotation, if present, will appear in the last column, which is called MISC. The file must use Linux-style line breaks, that is, a single LF character, rather than CR LF, which is common on Windows.

The MISC column is either a single underscore (_), meaning there is no extra annotation, or one or more pieces of annotation (typically in the Attribute=Value form), separated by vertical bars (|). The annotation pieces relevant for this shared task always start with Entity=; these should be learned from the training data and predicted for the test data. Any other annotation that is present in the MISC column of the input file should be preserved in the output (especially note that if you discard SpaceAfter=No, or introduce a new one, the validator may report the file as invalid).

For more information on the Entity attribute, see the PDF with the description of the CorefUD 1.0 format.

Example:

# global.Entity = eid-etype-head-minspan-infstat-link-identity
# sent_id = GUM_academic_art-3
# text = Claire Bailey-Ross xxx@port.ac.uk University of Portsmouth, United Kingdom
1	Claire	Claire	PROPN	NNP	Number=Sing	0	root	0:root	Entity=(e5-person-1-1,2,4-new-coref|Discourse=attribution:3->57:7
2	Bailey	Bailey	PROPN	NNP	Number=Sing	1	flat	1:flat	SpaceAfter=No
3	-	-	PUNCT	HYPH	_	4	punct	4:punct	SpaceAfter=No
4	Ross	Ross	PROPN	NNP	Number=Sing	2	flat	2:flat	Entity=e5)
5	xxx@port.ac.uk	xxx@port.ac.uk	PROPN	NNP	Number=Sing	1	list	1:list	Entity=(e6-abstract-1-1-new-sgl)
6	University	university	NOUN	NNP	Number=Sing	1	list	1:list	Entity=(e7-organization-1-3,5,6-new-sgl-University_of_Portsmouth
7	of	of	ADP	IN	_	8	case	8:case	_
8	Portsmouth	Portsmouth	PROPN	NNP	Number=Sing	6	nmod	6:nmod:of	Entity=(e8-place-1-3,4-new-sgl-Portsmouth|SpaceAfter=No
9	,	,	PUNCT	,	_	11	punct	11:punct	_
10	United	unite	VERB	NNP	Tense=Past|VerbForm=Part	11	amod	11:amod	Entity=(e9-place-2-1,2-new-coref-United_Kingdom
11	Kingdom	kingdom	NOUN	NNP	Number=Sing	1	list	1:list	Entity=e9)e8)e7)

Data download

Each CorefUD dataset is divided into a training section, a development section, and a test section (train/dev/test for short). Technically, each CorefUD dataset consists of three CoNLL-U files containing disjoint sets of documents; boundaries between the three sections can be placed only on document boundaries.

Training and development files containing gold coreference annotations are identical to the CoNLL-U files available in CorefUD 1.0 (the link leads to the LINDAT/CLARIAH-CZ repository where the data can be downloaded from). In addition, the development set with gold coreference annotation stripped off, which might be useful for development purposes, is available for download.

Test sets without gold coreference annotations will be made available to participants at the beginning of the evaluation phase. Test data with gold coreference annotation will be used internally in CodaLab for evaluation of submissions.


Evaluation Metric and Evaluation Tool

The official scorer for the shared task is corefud-scorer.py. Run the following command to calculate the primary score (CoNLL score) that will be used to rank the submissions (KEY_FILE is the file with gold annotations, RESPONSE_FILE is the file with your predictions):

python corefud-scorer.py KEY_FILE RESPONSE_FILE

The main evaluation metric for the task is the CoNLL score, which is an unweighted average of the F1 values of MUC, B-cubed, and CEAFe scores. To encourage the participants to develop multilingual systems, the primary ranking score will be computed by macro-averaging CoNLL F1 scores over all datasets.

For the same reason, singletons (entities with a single mention) will not be taken into account in calculation of the primary score, as many of the datasets do not have singletons annotated.

Since mention heads are annotated in CorefUD 1.0, it allows us to perform partial matching during evaluation. The response (predicted) mention matches a key (gold) mention partially if all words in the response mention belong also to the key mention and one of these words is the key mention head at the same time. The primary score will be computed using this partial matching (i.e. exact matching will not be used).

Although some of the datasets also comprise annotation of split antecedents, bridging and other anaphoric relations, these are not going to be evaluated.

Besides the primary ranking, the overview paper on the shared task will also introduce multiple secondary rankings, e.g. by CoNLL score for individual languages.


Submission Instructions

In a typical case, shared task participants should proceed as follows.

In the development phase (Phase 1):

  1. download training and development data
  2. decide whether to start from the baseline system, or from scratch
  3. use training data for developing your coreference resolution model
  4. apply the trained model to predict coreference in the development sets
  5. check validity of the development set files with predicted coreference
  6. evaluate the quality of your prediction by applying the evaluation script
  7. if you can see space for improvement of your model, improve it and go to 3
  8. create your CodaLab account
  9. practice the CodaLab submission process by uploading a zip file containing the development set files to CodaLab
The zip file uploaded to CodaLab must contain the following 13 files (without any other files or subdirectories):
ca_ancora-corefud-dev.conllu
cs_pcedt-corefud-dev.conllu
cs_pdt-corefud-dev.conllu
de_parcorfull-corefud-dev.conllu
de_potsdamcc-corefud-dev.conllu
en_gum-corefud-dev.conllu
en_parcorfull-corefud-dev.conllu
es_ancora-corefud-dev.conllu
fr_democrat-corefud-dev.conllu
hu_szegedkoref-corefud-dev.conllu
lt_lcc-corefud-dev.conllu
pl_pcc-corefud-dev.conllu
ru_rucor-corefud-dev.conllu

In the evaluation phase (Phase 2):

  1. download the test sets (a download link will be provided on June 1)
  2. apply the trained model to predict coreference in the test sets
  3. check validity of the test set files with predicted coreference
  4. submit a zip file with test set files to CodaLab, and see the evaluation results. The zip file must contain 13 files (ca_ancora-corefud-test.conllu, cs_pcedt-corefud-test.conllu etc.)
  5. if something goes wrong, fix your prediction model and go to step 2

Let us empasize that even if multiple submissions are possible within the evaluation phase, their number is limited, they are allowed rather because of resolving some unexpected situations, but definitely should not be used for systematic optimization of parameters or hyperparameters of your model towards the scores shown by CodaLab.

Pre-submission tests

A submission will receive a non-zero score only if it passes validation tests. If some test sets in the submitted zip file are valid and some are invalid, the submission may still get non-zero scores for the valid files; however, the zeros for the invalid files count towards the macro-average score of the submission. Many things can go wrong when filling the predicted coreference annotation in the CoNLL-U format (incorrect syntax in the MISC column, unmatched brackets etc.) It is highly recommended to always check validity prior to submitting the files, so that you do not run out of the maximum daily and total submission trials specified for the shared task.

There are two basic requirements for each submitted CoNLL-U file:

  • It must be accepted by the official UD validator script with the settings described below.
  • The contents of the input file must be preserved, i.e., the participating system is supposed to add coreference annotation but not to change anything else. In particular, tokenization and sentence segmentation must not change.
    • Within the submitted zip file, the CoNLL-U files output by the system must reside in the root folder and their names must be identical to the names of the corresponding input files. If this naming (and placement) convention is not observed, the scorer will not be able to pair the output with the input, and the output will not be scored.

The official UD validator will be used to check the validity of the CoNLL-U format. Anyone can obtain it by cloning the UD tools repository from Github and running the script validate.py. Python 3 is needed to run the script (depending on your system, it may be available under the command python or python3; if in doubt, try python -V to see the version).

$ git clone git@github.com:UniversalDependencies/tools.git
$ cd tools
$ python3 validate.py -h

In addition, a third-party module called regex must be installed via pip. Try this if you do not have the module already:

$ sudo apt-get install python3-pip; python3 -m pip install regex

The validation script distinguishes several levels of validity; level 2 is sufficient in the shared task, as the higher levels deal with morphosyntactic requirements on the UD-released treebanks. On the other hand, we will use the --coref option to turn on tests specific to coreference annotation. The validator also requires the option --lang xx where xx is the ISO language code of the data set.

$ python3 validate.py --level 2 --coref --lang cs cs_pdt-corefud-test.conllu
*** PASSED ***

If there are errors, the script will print messages describing the location and the nature of the error, it will print *** FAILED *** with (number of) errors, and it will return a non-zero exit value. If the file is OK, the script will print *** PASSED *** and return zero as its exit value. The script may also print warning messages that point to potential problems in the file but are not considered errors and will not make the file invalid.

Baseline system

The baseline system is based on the multilingual coreference resolution system presented in [7]. The model uses multilingual BERT in the end-to-end setting. In simple words, the model goes through all potential spans and maximizes the probability of gold antecedents for each span. The same system is used for all the languages. More details can be found in [7].

The simplified system adapted to CorefUD 1.0, is publically available on GitHub along with tagged dev data and its dev data results.

The dev files with coreference predicted by the baseline system can be downloaded directly as a zip file from here, so you do not have to run the baseline system yourself and can only try to improve its outputs. The zip file's structure is identical to what will be expected by the CodaLab submission system.

Udapi – Recommended Python API for CorefUD

Udapi is a Python API for reading, writing, querying and editing Universal Dependencies data in the CoNLL-U format (and several other formats). Newly, it has support for the coreference annotations (and it was used for producing CorefUD). Even if you decide not to build your system by extending the baseline system, you can use Udapi for accessing the CorefUD data in a comfortable way. We will provide a tutorial on that soon.

Terms and Conditions

  • Training, development, and test datasets are subject to license agreements specified individually for each dataset in the public edition of the CorefUD 1.0 collection (which, in turn, are the same as license agreements of the original resources before CorefUD harmonization). In all cases, the licenses are sufficient for using the data for the CRAC2022 shared task purposes. However, the participants must check the license agreements in case they want to use their trained models also for other purposes; for instance, usage for commercial purposes is prohibited with several CorefUD datasets as they are available under CC BY-NC-SA.
  • By submitting results to this competition, the participants consent to the public release of their scores at the CRAC2022 workshop and in the associated proceedings, at the task organizers' discretion. Participants further agree that the task organizers are under no obligation to release scores and that scores may be withheld if it is the task organizers' judgment that the submission was erroneous or deceptive.