Unsupervised Machine Learning in NLP

The seminar focuses on deeper understanding of selected unsupervised machine learning methods for students who have already have basic knowledge of machine learning and probability models. The first half of the semester is devoted to methods of unsupervised learning using Bayesian inference (Dirichlet-Categorical models, Mixture of Categoricals, Mixture of Gaussians, Expectation Maximization, Gibbs sampling) and implementation of these methods on selected tasks. Other lectures will be devoted to clustering methods, componet analysis and inspecting deep neural networks.

About

SIS code: NPFL097
Semester: winter
E-credits: 3
Examination: 1/1 C
Guarantor: David Mareček

Timespace Coordinates

The lectures in Czech are given on Tuesdays, 12:20 - 13:50 in S1 (fourth floor)

The lectures in English are given on Wednesdays 12:20 - 13:50 (write me an e-mail if interested)

Course prerequisities

Students are expected to be familiar with basic probabilistic and ML concepts, roughly in the extent of:

  • NPFL067 - Statistical methods in NLP I,
  • NPFL054 - Introduction to Machine Learning (in NLP).

In the second half of the course, you should be familiar with the basics of deep-learning methods. I recommend to attend

Course passing requirements

  • There are three programming assignments during the term. For each one, you can obtain 10 points. When submitted after the deadline, you can obtain at most half of the points.
  • You can obtain 10 point for individual 30-minutes presentation about selected machine learning method or task.
  • You pass the course if you obtain at least 20 points.

Lectures

1. Introduction Slides Warm-Up test

2. Beta-Bernoulli probabilistic model Beta-Bernoulli Beta distribution

3. Dirichlet-Categorical probabilistic model Dirichlet-Categorical Document collections Categorial Mixture Models

4. Latent Dirichlet Allocation Beta and Dirichlet distributions Topic Models - Introduction Topic Models - Evaluation Topic Models - Gibbs Sampling Latent Dirichlet Allocation

5. Gibbs Sampling Bayessian inference with Tears

6. Chinese Segmentation Bayessian inference with Tears Chinese Restaurant Process Chinese Segmentation

7. Clustering Clustering - Basics Clustering - Hierarchical Clustering - K-means Clustering - Gaussian Mixture Models K-Means and Gaussian Mixture Models Gaussians Mixture Models

8. Principal Component Analysis Principal Component Analysis, SVD

1. Introduction

 Feb 25

  • Course overview Slides
  • revision of the basics of probability and machine learning theory Warm-Up test

2. Beta-Bernoulli probabilistic model

 Mar 3

3. Dirichlet-Categorical probabilistic model

 Mar 10

4. Latent Dirichlet Allocation

 Mar 24

  • Watch the following videos by Jordan Boyd-Graber (University of Maryland):

Beta and Dirichlet distributions Topic Models - Introduction Topic Models - Evaluation Topic Models - Gibbs Sampling

  • Based on the given lectures, complete the following assignment:

Latent Dirichlet Allocation

5. Gibbs Sampling

 Apr 7

  • Read the following tutorial by Kevin Knight and learn about Bayessian inference from another point of view with a nice examples:

Bayessian inference with Tears

6. Chinese Segmentation

 Apr 14

  • Unsupervised segmentation of texts in languages which does not use spaces between words.

  • All the necessary information the second assignment is covered by the tutorial from the last lecture:

Bayessian inference with Tears

  • The unsupervised segmentation is decribed in sections 17 and 29, however, you will need many other hints from the whole text, so please, read it all.

  • You can also go through the following slides:

Chinese Restaurant Process

  • Complete the following assignment. If you have any questions or do not understand something, let me know.

Chinese Segmentation

7. Clustering

 May 12

  • Watch the following videos by Alexander Ihler:

Clustering - Basics Clustering - Hierarchical Clustering - K-means Clustering - Gaussian Mixture Models

  • The algorithms of K-Means and Gaussian Mixture Models are aslo well explained in the following presentation by David Rosenberg from University of New York:

K-Means and Gaussian Mixture Models

  • Complete the following assignment:

Gaussians Mixture Models

8. Principal Component Analysis

 May 19

  • Watch the following video about PCA:

Principal Component Analysis, SVD

Latent Dirichlet Allocation

 Deadline: Apr 14 23:59  10 points

Chinese Segmentation

 Deadline: May 5 23:59  10 points

Gaussians Mixture Models

 Deadline: Sep 15, 23:59

Points 10 points

  • Christopher Bishop: Pattern Recognition and Machine Learning, Springer-Verlag New York, 2006 (read here)

  • Kevin P. Murphy: Machine Learning: A Probabilistic Perspective, The MIT Press, Cambridge, Massachusetts, 2012 (read here)