Asignment #3: PFL068 Statistical NLP

Tagging

Instructor: Jan Hajic <hajic@ufal.mff.cuni.cz>
TA: Pavel Pecina <pecina@ufal.mff.cuni.cz>

Back to syllabus.

---

Requirements

For all parts of this homework, work either alone or in a group of max. two people (identical grade will be assigned to both of you in such a case - thus please make sure you understand what your colleague is doing, and that s/he is doing it right!). On top of the results/requirements specific to a certain part of the homework, turn in all of your code, commented in such a way that it is possible to determine what, how and why you did what you did solely from the comments, and a discussion/comments of/on the results (in a plain text/html) file. Technically, follow the usual pattern (see the Syllabus): For this whole homework, use data found in

texten2.ptg

textcz2.ptg

In the following, "the data" refers to both English and Czech, as usual.

Split the data in the following way: use last 40,000 words for testing (data S), and from the remaining data, use the last 20,000 for smoothing (data H, if any). Call the rest "data T" (training).

1. Brill's Tagger & Tagger Evaluation

Download Eric Brill's supervised tagger from UFAL's course assignment space. Install it (i.e., uncompress (gunzip), untar, and make).

You might need to make some changes in his makefile of course (it's and OLD program, in this fast changing world...).

After installation, get the data, train it on as much data from T as time allows (in the package, there is an extensive documentation on how to train it on new data), and evaluate on data S. Tabulate the results.

Do cross-validation of the results: split the data into S', [H',] T' such that S' is the first 40,000 words, and T' is the last but the first 20,000 words from the rest. Train Eric Brill's tagger on T' (again, use as much data as time allows) and evaluate on S'. Again, tabulate the results.

Do three more splits of your data (using the same formula: 40k/20k/the rest) in some way or another (as different as possible), and get another three sets of results. Compute the mean (average) accuracy and the standard deviation of the accuracy. Tabulate all results.

2. Unsupervised Learning: HMM Tagging

Use the datasets T, H, and S. Estimate the parameters of an HMM tagger using supervised learning off the T data (trigram and lower models for tags). Smooth (both the trigram tag model as well as the lexical model) in the same way as in Homework No. 1 (use data H). Evaluate your tagger on S, using the Viterbi algorithm.

Now use only the first 10,000 words of T to estimate the initial (raw) parameters of the HMM tagging model. Strip off the tags from the remaining data T. Use the Baum-Welch algorithm to improve on the initial parameters. Smooth as usual. Evaluate your unsupervised HMM tagger and compare the results to the supervised HMM tagger.

Tabulate and compare the results of the HMM tagger vs. the Brill's tagger.