E2E
NLG Challenge

We are happy to announce that the interest in the E2E NLG shared task has by far outperformed our expectations. Heriot-Watt University has set out this challenge for the first time this year, and we received a total of 62 submissions by 17 institutions, with about 1/3 of these submissions coming from industry. In comparison, the well established Conference in Machine Translation WMT’17 (running since 2006) got 31 institutions submitting to a total of 8 tasks.

Participants map

A brief summary of the E2E NLG Challenge results is now available in our INLG 2018 paper, a more detailed analysis is in preparation.

Automatic Metrics

The automatic evaluation results were obtained using the metrics scripts provided with the baseline. The table is sortable – just click on the metric you want use for sorting. Click again to reverse the sort.

Human Evaluation (updated results)

The human evaluation was conducted on the 20 primary systems and the baseline using the CrowdFlower platform. We used our newly developed RankME method (Novikova et al., 2018) to obtain the ratings. Crowd workers were presented with five randomly selected outputs of different systems corresponding to a single meaning representation, and were asked to rank these systems from the best to worst, ties permitted. A single human-authored reference was provided for comparison. We collected separate ranks for quality and naturalness.

Quality is defined as an overall quality of the utterance, in terms of its grammatical correctness, fluency, adequacy and other important factors. When collecting quality ratings, system outputs were presented to crowd workers together with the corresponding meaning representation.

Naturalness is defined the extent to which the utterance could have been produced by a native speaker. When collecting naturalness ratings, system outputs were presented to crowd workers without the corresponding meaning representation.

If used in a real-life NLG system, quality would be considered the primary measure.

The final evaluation results were produced using the TrueSkill algorithm (Sakaguchi et al., 2014). For naturalness, the algorithm performed 1890 pairwise comparisons per each system (37800 comparisons in total), for quality – 1260 comparisons per system (25200 comparisons in total). In results tables, systems are ordered by their inferred system TrueSkill scores, and clustered. Systems within a cluster are considered tied. The system clusters have been created using bootstrap resampling, with a p-level of p ≤ 0.05.

Quality

Naturalness

A brief description of the challenge results was published at INLG. To cite the challenge, use:

@inproceedings{dusek2018findings,
  title={Findings of the {E2E} {NLG} {Challenge}},
  author={Du{\v{s}}ek, Ond\v{r}ej and Novikova, Jekaterina and Rieser, Verena},
  booktitle={Proceedings of the 11th International Conference 
             on Natural Language Generation},
  address={Tilburg, The Netherlands},
  year={2018},
  note={arXiv:1810.01170},
  url={https://arxiv.org/abs/1810.01170},
}

A greatly extended analysis of the challenge results is now released on arXiv (as a journal submission under review).

System outputs and human ratings can now be downloaded from here. Please use the same citation to refer for this data release.

All submitters participating in human evaluation provided a description of their primary systems as a technical paper. The papers are linked below:

Other papers using the E2E dataset

Published versions of the systems participating in the Challenge:

• Agarwal et al. (INLG 2018): Char2char Generation with Reranking for the E2E NLG Challenge
• Elder et al. (INLG 2018): E2E NLG Challenge Submission: Towards Controllable Generation of Diverse Natural Language
• Puzikov & Gurevych (INLG 2018): E2E NLG Challenge: Neural Models vs. Templates
• Smiley et al. (INLG 2018): The E2E NLG Challenge: A Tale of Two Systems
• Deriu & Cieliebak (INLG 2018): Syntactic Manipulation for Generating more Diverse and Interesting Texts
• Gehrmann et al. (INLG 2018): End-to-End Content and Plan Selection for Data-to-Text Generation
• Juraska et al. (NAACL 2018): A Deep Ensemble Model with Slot Alignment for Sequence-to-Sequence Natural Language Generation

Further works that use the E2E dataset but did not participate in the official E2E challenge:

• Guy Lapalme's realizations using jsRealB -- symbolic realizer, also produces French versions of the outputs. Includes a browser for the outputs, grouping them by slot values.
• Su & Chen (SLT 2018): Investigating Linguistic Pattern Ordering in Hierarchical Natural Language Generation
• Reed et al. (INLG 2018): Can Neural Generators for Dialogue Learn Sentence Planning and Discourse Structuring?
• Juraska & Walker (INLG 2018): Characterizing Variation in Crowd-Sourced Data for Training Neural Language Generators to Produce Stylistically Varied Outputs
• Shimorina & Gardent (INLG 2018): Handling Rare Items in Data-to-text Generation
• Jagdfeld et al. (INLG 2018): Sequence-to-Sequence Models for Data-to-Text Natural Language Generation: Word- vs. Character-based Processing and Output Diversity
• Wiseman et al. (EMNLP 2018): Learning Neural Templates for Text Generation
• Freitag & Roy (EMNLP 2018): Unsupervised Natural Language Generation with Denoising Autoencoders
• Oraby et al. (Interspeech 2018): Neural MultiVoice Models for Expressing Novel Personalities in Dialog
• Su et al. (NAACL 2018): Natural Language Generation by Hierarchical Decoding with Linguistic Patterns
• Shen et al. (NAACL 2019): Pragmatically Informative Text Generation