1. Introduction

 Oct 5

• Course overview, Examples of Unsupervised learning methods Introduction

2. Beta-Bernoulli probabilistic model

 Oct 12

• Binary models: Bernoulli, Binomial, and Beta distributions, making predictions Beta-Bernoulli
• How to compute expected value of the Beta distribution: Expected value of Beta distribution
• Web application showing the Beta-Bernouli distribution from the Coin Experiment: Beta Coin Experiment.
• Beta distribution online demo: https://mathlets.org/mathlets/beta-distribution/.

3. Dirichlet-Categorical probabilistic model, Modeling document collections

 Oct 19

• generalization of Beta-Bernoulli to Dirichlet-Categorical distribution Dirichlet-Categorical
• introduction to modeling document collections Document collections

4. Mixture of Categoricals, Expectation-Maximization, Bayesian Mixture Models, Latent Dirichlet Allocation

 Oct 26

• Expectation-Maximization for Categorical Mixture Models Mixture of Categoricals
• A very simple code using EM algorithm: em.py.
• Bayessian Mixture Models, Latent Dirichlet Allocation Latent Dirichlet Allocation

5. Gibbs Sampling in Latent Dirichlet Allocation, Entropy, Assignment 1

 Nov 9

• Gibbs Sampling in LDA Gibbs Sampling for LDA
• Example video of Gibbs sampling from the bivariate normal distribution: Gibbs Sampling from bivariate Gaussian
• Assignment 1: Latent Dirichlet Allocation

6. Chinese Restaurant Process

 Nov 16

• Slides for Chinese Restaurant Process and Unsupervised segmentation of texts Chinese Restaurant Process
• Chinese Restaurant Process demo: https://topicmodels.info/ckling/tmt/crp.html.
• tutorial (2009) Bayessian inference with Tears (by K.Knight)

7. Unsupervised Text Segmentation

 Nov 23

• Rediscovering spaces between words Chinese Restaurant Process
• Assignment 2: Text Segmentation
• Gibbs Sampling method for solving traditional NLP Tasks Traditional_NLP_Tasks

8. K-Means clustering, Mixture of Gaussians

 Nov 30

• Slides K-Means and Gaussian Mixture Models

9. Aglomerative Clustering, Evaluation methods

 Dec 7

• Aglomerative clustering, evaluation methods Aglomerative Clustering and Clustering Evaluation

10. Dimesionality Reduction

 Dec 14

• t-SNE, Principal Component Analysis, Independent Component Analysis
• Slides Dimensionality Reduction
• Demo: t-SNE and PCA demo
• Assignment 3: Clustering and Component Analysis on Word Vectors

11. CANCELLED

 Dec 21

12. In-context Learning, Interpretation of Neural Language Models

 Jan 4 In-Context Learning Interpretation of Neural Language Models

13. Final test

 Jan 11

Latent Dirichlet Allocation

 Deadline: Nov 30, 23:59  10 points

• Instructions: assignment1.pdf
• Preprocessing procedures: lda-data.py

Text Segmentation

 Deadline: Dec 14 23:59  10 points

• Instructions: assignment2.pdf
• Input data: data_small.txt
• Evaluation script eval.py
• Evaluation data: data_small_gold.txt

Clustering and Component Analysis on Word Vectors

 Deadline: Feb 18 23:59  10 points

• Instructions: assignment3.pdf
• Input data: nmt-en-dec-512.txt

List of questions for the final test

1. Define Beta distribution, describe its parameters. Plot (roughly) the following distributions: Beta(1,1), Beta(0.1,0.1), Beta(10, 10).

2. Derive the posterior distribution from the prior (Beta distribution) and likelihood (Binomial distribution). Derive the predictive distribution for the Beta-Bernoulli posterior.

3. Explain Dirichlet distribution, describe its parameters. Plot (roughly) the following distributions: Dir(1,1,1), Dir(0.1,0.1,0.1), Dir(10, 10, 10).

4. Derive the posterior distribution from the prior (Dirichlet distribution) and likelihood (Multinomial distribution). Derive the predictive distribution for the Dirichlet-Categorical posterior.

5. Explain the "Mixture of Categoricals" model (a topic is assigned to each document) for Modeling document collections. Describe all its parameters and hyperparameters. From what distributions are they drawn? Describe the Expectation-Maximization algorithm for training such model.

6. Explain the Latent Dirichlet Allocation model (a topic is asigned to each word in each document). Describe all its parameters and hyperparameters. From what distributions are they drawn? What are the latent variables? Describe the learning algorithm based on Gibbs sampling.

7. Explain Collapsed Gibbs sampling. Choose one unsupervised task from the lectures (word alignment, tagging, segmentation) and describe the basic algorithm. What is annealing?

8. Explain Chinese Restaurant Process. What distributions does it generate? What is exchangeability? Explain its generalization to the Pitman-Yor process.

9. Explain the Gaussian Mixture model for clustering. What are the advantages of Gaussian Mixture model compared to K-means? Provide an example of clusters in 2D where K-means fails and where Gaussian Mixture model works well.

10. Explain Hierarchical Agglomerative clustering methods. What are their advantages over K-means? What linkage criteria do you know? Provide examples of clusters in 2D where these criteria fail.

11. What is t-SNE? What properties does it have? What is it used for? How does it work?

12. What is Principal Component Analysis? What properties does it have? What is it used for? How does it work? Explain it in a 2D example.

• Christopher Bishop: Pattern Recognition and Machine Learning, Springer-Verlag New York, 2006 (read here)

• Kevin P. Murphy: Machine Learning: A Probabilistic Perspective, The MIT Press, Cambridge, Massachusetts, 2012 (read here)

• David Mareček, Jindřich Libovický, Tomáš Musil, Rudolf Rosa, Tomasz Limisiewicz: HIDDEN IN THE LAYERS: Interpretation of Neural Networks for Natural Language Processing. Institute of Formal and Applied Linguistics, 2020 (read_here)