1. Introductory notes and discussion on large language models

 Feb 19 Slides

Instructor: Zdeněk Kasner

Covered topics: aims of the course, passing requirements. We informally discussed what are (large) language models, what are they for, what are their benefits and downsides. We also gathered ideas on how to train the models, how to use them, and how to evaluate them.

2. The Transformer architecture

 Feb 26 Slides Notes Recording

Instructor: Jindřich Libovický

Learning objectives. After the lecture you should be able to...

• Explain the building blocks of the Transformer architecture to a non-technical person.

• Describe the Transformer architecture using equations, especially the self-attention block.

• Implement the Transformer architecture (in PyTorch or another framework that does automated differentiation).

Additional materials.

• Transformers explained by AI Coffee Break with Letitia (20 min)

• Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy (2 hours)

• The Illustrated Transformer by Jay Alammar

• MicroGPT Project: Complete Transformer training and inference code in 200 lines of Python

3. Data & Evaluation

 Mar 5 Slides

Instructors: Ondřej Dušek & Patrícia Schmidtová

Learning objectives:

• Understand what kinds of data LLMs need for training
• Know about viable tasks for evaluating LLMs
• Know where to find data for evaluation tasks
• Have an idea on how to evaluate -- what approaches there are and what are their pros and cons

Additional materials:

• A survey of datasets for LLMs

• A blog post on evaluation -- a little different take than what we showed at the lecture, but very well written

4. LLM Inference

 Mar 12 Slides Code Recording

Instructor: Zdeněk Kasner

Learning objectives. After the class you should be able to...

• Understand how to generate text with a Transformer-based language model.
• Explain differences between decoding algorithms and the role of decoding parameters.
• Choose a suitable LLM for your task.
• Run a LLM locally on your computer or computational cluster.

Additional materials.

• AnimatedLLM, series of LLM visualizations
• LLM Visualization, a 3D visualization of transformer inference
• Deep Dive into LLMs like ChatGPT by Andrej Karpathy (3.5 hours)
• HuggingFace Models, the repository of open (L)LMs
• Awesome LLM, a curated list of resources for LLMs
• How to generate text, the Huggingface decoding algorithms overview
• Generation with LLMs, common pitfalls when generating text with LLMs

5. Hands-on session

 Mar 19 Exercise Shared Report

Instructor: Zdeněk Kasner

Learning objectives. After the class, you should be able to...

• Query open local models via an API.
• Describe the influence of different decoding parameters on the nature of the outputs.
• Choose an appropriate prompting technique for different models.

Additional materials.

• OpenWeather API docs
• Prompting guide
• Best prompting practices from Huggingface

6. LLM Training

 Mar 26 Slides Recording

Instructor: Ondřej Dušek

Learning objectives:

• Have a basic idea about training neural networks (gradient descent)
• Understand the phases of LLM training (pretraining, instruction tuning, RLHF)
• Distinguish between prompting and finetuning

7. Tokenization

 Apr 2 Slides

Instructor: Jindřich Libovický

Learning objectives. After the class, you should be able to...

• Explain how the current LLM tokenizers (BPE and Unigram) are trained and used at inference time
• When seeing a tokenized text, reason why it got tokenized the way it did
• Discuss emerging alternatives to BPE and subword tokenization in general

8. Multilinguality

 Apr 9

Instructor: Jindřich Libovický

9. Mini-conference: Research for and with LLMs

 Apr 16

Researchers from ÚFAL will briefly present their recent projects that involve LLMs.

Project work

You will work on a team project during the semester. Teams of 4-6 students will work on the following topics.

Project Timeline

• 2 March: project assignment
• Week of 16–20 March, Kick-off meetings with supervisor
• 31 March: Experiment plan submitted
• 21 April: Self-assessment form
• Project presentations
  • Early bird: 14 May
  • Standard: 21 May
• Project reports by the end of the semester (Hard deadline: 5 working days before you need the credit)

In addition, write a log every week from the start until the end (the earlier you submit the less logs to do).

Project Reports

Each team will submit a report, consisting of:

• Brief method overview
• Summary of related work
• Experimental design
• Results
• Conclusions
• Overview of individual contributions of team members
• References

The length of the report should be maximum 4 pages plus references and contributions. You might want to use the ACL paper template.

Reading assignments

You will be asked at least once to read a paper before the class.

Final written test

You need to take part in a final written test that will not be graded.