Logo ÚFAL Logo MFF

UMR Parsing


ÚFAL
UMR Parsing

UMR Parsing Shared Task

To be held as part of the DMR 2026 workshop, collocated with LREC (Palma de Mallorca, Spain).

Interested? Register here.

Timeline

We tentatively work with the following schedule:

  • 1 February 2026: Training + development data available (together with the evaluation script)
  • 16 February 2026: Blind test data available. Test phase starts.
  • 27 February 2026: Submission of system outputs. Test phase ends.
  • 1 March 2026: Announcement of the results.
  • 15 March 2026: System description papers due.
  • 22 March 2026: Reviews of system description papers due.
  • 30 March 2026: Camera-ready papers due.
  • 11–16 May 2026: LREC + DMR Workshop, Palma de Mallorca, Spain

Data

Training data will be based on UMR 2.1, although not necessarily identical. There will be multiple languages with training data (English, Czech, Chinese). Test data may contain additional languages, leading to zero-shot scenarios.

All training/development data will be freely available, without need for registration or signing a contract. The previously unpublished test data will be published under similar conditions after the shared task.

Detailed specification of the format will be published together with the training data. Participants will be expected to submit valid system outputs in the same format. Blind data provided as system input will be tokenized and segmented to sentences; to facilitate evaluation, system outputs must preserve tokenization and segmentation.

Evaluation

The evaluation script will be made available to the participants before the test phase so that they can evaluate their output locally.

We will define the main metric for ranking of the participating systems, but we also plan on computing various additional metrics (such as a separate F1-score for concepts). While the main evaluation will be centered around the sentence graphs, the additional metrics will include document-level relations (for systems that can predict these relations, and only on data that actually have gold-standard document level graphs). Alignments between nodes and words will be also evaluated.

The shared task will not be divided into any tracks. System outputs will be submitted to the task as a whole, and every submission will be evaluated along the same set of metrics.

Participation

Individuals and teams considering participation should register via a simple Google form (https://forms.gle/pc2c7A27TxeHjRKZ7). There is no deadline for registration but the sooner the better, as we intend to send important information to registered participants by e-mail.

There are no restrictions on who can participate. (The two main organizers will not participate.)

The link to the submission form will be posted here before the test phase starts. Participants will submit system outputs (parsed data), not the systems themselves. Each submission will be automatically checked for validity, so the participants know whether their submission can be evaluated.