Submission Rules

One submission is a gzipped tar archive, named “xxx-yyy.tgz”, where xxx should be replaced by a team identifier and yyy should be replaced by a unique submission identifier. The team identifier and the submission identifier may contain lowercase English letters a-z, digits 0-9, and the underscore character “_”. If the submission identifier is identical to an identifier of a previous submission of the same team, the previous submission is forgotten and replaced by the current submission.

The most recent submission received from a team is considered the primary submission that will yield the official score of the team in the shared task. Exception: if the submission identifier starts with the prefix “secondary” and if there is at least one submission without such prefix, the submission with the “secondary” prefix is not considered primary.

The tgz archive must contain directly the parsed UMR files (no folders!) For example, if the current working folder is the one where all parser-output UMR files reside and there are no other files, you can create the archive like this (on a Unix system):

tar czf myteam-mysubmission.tgz *.umr

We intend to batch-process the submissions roughly like this:

GOLD=/path/to/gold-standard
TEAM=xxx
SUBMISSION=yyy
if [[ -e “$TEAM/$SUBMISSION” ]] ; then
  rm -rf $TEAM/$SUBMISSION
fi
mkdir -p $TEAM/$SUBMISSION
cd $TEAM/$SUBMISSION
tar xzf ../../$TEAM-$SUBMISSION.tgz
for i in $GOLD/*.umr ; do evaluate $i `basename $i` ; done

It is thus important to name the files inside the tgz archive correctly, otherwise they will not be found by the evaluation procedure. The naming convention is simple. We expect 30 files named “ll-nnnn.umr”, where ll should be replaced by the ISO code of the language and nnnn is a string of digits. The same naming scheme applies to the respective plain-text input files: “ll-nnnn.txt”, so the parser knows the code and number for each file. The language codes are:

Arapaho … arp; Chinese … zh; Czech … cs; English … en; Italian … it; Latin … la; Navajo … nv.

Download the blind test (input) data here.

Validity of the System Output

The submission must be a valid UMR file and it must be accepted by the official validating script (it is enough to pass validate.py --level 2). Participants are strongly encouraged to validate the preliminary output of their system well in advance so they can spot potential problems early.

In addition to the validity of the UMR file, it is required that tokens in the Words line in the first block of each sentence be identical to those in the input file. This is a pre-requisite for the evaluation; files with modified word forms will be rejected.

If a UMR file for a particular test document is invalid or missing, the submission is not completely discarded and can still be evaluated for the test documents for which valid output was submitted, but the scores for the missing/invalid files are set to zero.

Submission Site

Click here to proceed to the submission site.

The test phase is scheduled to conclude on February 27, 23:59 AoE. The submission site will stay open even after the deadline, but the official ranking will be based on submissions that arrived in time.