Selected Problems in Machine Learning

• Teachers: Zdeněk Žabokrtský, David Mareček
• Time and location: Tuesday 12:20–14.00 S7

Course focus

The course has been designed especially for PhD students with a deep interest in Machine Learning. The course is a flexible combination of lectures, discussions, exercises and literature reading, aimed at the following three topics:

1. refreshing (and deeper understanding of) basic notions of Machine Learning
2. introduction to Bayesian inference
3. practising unsupervised ML (especially methods based on sampling)

Course prerequisities

Students are expected to be familiar with basic probabilistic and ML concepts, roughly in the extent of NPFL067/068 - Statistical methods in NLP I/II, and NPFL 054 - Introduction to Machine Learning (in NLP).

Course schedule

1. "Calibration" test - let me know what you already know
   • calibration_test.pdf
   • most of the questions are covered by any modern ML book, my favourite is
     • Bishop's Pattern Recognition and Machine Learning
2. Patching the holes disclosed by the calibration test.
3. Patching the holes disclosed by the calibration test, continued.
4. Very studious excercise on Beta distribution.
   • let us admire two mighty parameters generating a broad family of different shapes
   • generalization to n-dimenzions - Dirichlet distribution
   • supplementary materials - mathematicalmonks's videos:
     • (ML 7.7.A1) Dirichlet distribution
5. Derivation of some simple Bayesian models - let's enjoy conjugacy!
   • Beta-Bernoulli model
   • Dirichlet-Categorical model
   • supplementary materials - mathematicalmonks's videos:
     • (ML 7.5) Beta-Bernoulli model (part 1)
     • (ML 7.6) Beta-Bernoulli model (part 2)
     • (ML 7.7) Dirichlet-Categorical model (part 1)
     • (ML 7.8) Dirichlet-Categorical model (part 2)
6. Assignment 1 - Word-alignment using Gibbs sampling
   • Text: PDF
   • Data: english-czech.tsv
   • Gold data: test.tsv
   • Evaluation script: eval.pl
   • Deadline: November 20, 2012
7. Reading - Bayesian Inference
   • Bayesian inference with tears by Kevin Knight
8. Kernel methods
   • slides on kernel methods by Mark Johnson
9. Assignment 2 - Segmentation of dependency trees
   • assignment
   • data
10. Gibbs sampling in NLP - two case studies
    • Sharon Goldwater and Thomas L. Griffiths: A Fully Bayesian Approach to Unsupervised Part-of-Speech Tagging. Proceedings of the ACL, 2007.
    • Jason Naradowsky and Sharon Goldwater: Improving morphology induction by learning spelling rules. In Proceedings of IJCAI. 2009.

Other useful links

• Nonparametric Bayesian Models of Lexical Acquisition. Sharon Goldwater. Ph.D. Dissertation, Brown University, 2006.
• Sharon Goldwater's Reading list on Bayesian modeling for language
• Bayesian nonparametrics and the probabilistic approach to modelling
• Bayesian Inference: In layman's terms, how does Gibbs Sampling work?

Course passing requirements

All students are required to actively participate in the classes.