1. Introductory notes and discussion on large language models

 Feb 19 Slides

Instructor: Zdeněk Kasner

Covered topics: aims of the course, passing requirements. We informally discussed what are (large) language models, what are they for, what are their benefits and downsides. We also gathered ideas on how to train the models, how to use them, and how to evaluate them.

2. The Transformer architecture

 Feb 26 Slides Notes Recording

Instructor: Jindřich Libovický

Learning objectives. After the lecture you should be able to...

• Explain the building blocks of the Transformer architecture to a non-technical person.

• Describe the Transformer architecture using equations, especially the self-attention block.

• Implement the Transformer architecture (in PyTorch or another framework that does automated differentiation).

Additional materials.

• Transformers explained by AI Coffee Break with Letitia (20 min)

• Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy (2 hours)

• The Illustrated Transformer by Jay Alammar

• MicroGPT Project: Complete Transformer training and inference code in 200 lines of Python

3. Data & Evaluation

 Mar 5 Slides

Instructors: Ondřej Dušek & Patrícia Schmidtová

Learning objectives:

• Understand what kinds of data LLMs need for training
• Know about viable tasks for evaluating LLMs
• Know where to find data for evaluation tasks
• Have an idea on how to evaluate -- what approaches there are and what are their pros and cons

Additional materials:

• A survey of datasets for LLMs

• A blog post on evaluation -- a little different take than what we showed at the lecture, but very well written

4. LLM Inference

 Mar 12 Slides Code Recording

Instructor: Zdeněk Kasner

Learning objectives. After the class you should be able to...

• Understand how to generate text with a Transformer-based language model.
• Explain differences between decoding algorithms and the role of decoding parameters.
• Choose a suitable LLM for your task.
• Run a LLM locally on your computer or computational cluster.

Additional materials.

• AnimatedLLM, series of LLM visualizations
• LLM Visualization, a 3D visualization of transformer inference
• Deep Dive into LLMs like ChatGPT by Andrej Karpathy (3.5 hours)
• HuggingFace Models, the repository of open (L)LMs
• Awesome LLM, a curated list of resources for LLMs
• How to generate text, the Huggingface decoding algorithms overview
• Generation with LLMs, common pitfalls when generating text with LLMs

5. Hands-on session

 Mar 19

Instructor: Zdeněk Kasner

6. LLM Training

 Mar 26

Instructor: Ondřej Dušek

Project work

You will work on a team project during the semester. Teams of 4-6 students will work on the following topics.

Project Timeline

• 2 March: project assignment
• Week of 16–20 March, Kick-off meetings with supervisor
• 31 March: Experiment plan submitted
• 21 April: Self-assessment form
• Project presentations
  • Early bird: 14 May
  • Standard: 21 May
• Project reports by the end of the semester (Hard deadline: 5 working days before you need the credit)

In addition, write a log every week from the start until the end (the earlier you submit the less logs to do).

Project Reports

Each team will submit a report, consisting of:

• Brief method overview
• Summary of related work
• Experimental design
• Results
• Conclusions
• Overview of individual contributions of team members
• References

The length of the report should be maximum 4 pages plus references and contributions. You might want to use the ACL paper template.

Reading assignments

You will be asked at least once to read a paper before the class.

Final written test

You need to take part in a final written test that will not be graded.