Lectures

1. Introduction

 20 February Slides Domain selection Questions

• What are dialogue systems
• Common usage areas
• Task-oriented vs. non-task oriented systems
• Closed domain, multi-domain, open domain
• System vs. user initiative in dialogue
• Standard dialogue systems components

2. What happens in a dialogue?

 27 February Slides Dataset exploration Questions

• Dialogue turns
• Utterances as acts, pragmatics
• Grounding, grounding signals
• Deixis
• Conversational maxims
• Prediction and adaptation

3. Data & Evaluation

 6 March Slides Questions

• How to get data for building dialogue systems
• Available corpora/datasets
• Annotation
• Data splits
• Evaluation metrics -- subjective & objective, intrinsic & extrinsic
• Significance checks

4. Natural Language Understanding

 13 March Slides Questions Rule-based Natural Language Understanding

• What needs to be handled to understand the user
• How to represent meaning: grammars, frames, graphs, dialogue acts (“shallow parsing”)
• Rule-based NLU
• Basics of machine learning, discriminative & generative classifiers
• Classification-based NLU (features, logistic regression, SVM)
• Sequence tagging (HMM, MEMM, CRF)

5. Neural NLU + State Tracking

 20 March Slides Questions Statistical Natural Language Understanding

• Some basics about neural networks
• How to use neural networks for NLU: neural classifiers and sequence taggers
• Handling speech recognition noise in NLU
• What is dialogue state, what is belief state, and what they're good for
• Dialogue as a Markov decision process (MDP)
• Dialogue trackers: generative and discriminative
• Static and dynamic trackers
• Neural trackers, integrated NLU+tracker

Homework Assignments

There will be 12 homework assignments, each for a maximum of 10 points. Please see details on grading and deadlines on a separate tab. Note that there's a 50% minimum requirement to pass the course.

Assignments should be submitted via Git – see instructions on a separate tab. Please take special care about naming your Git branches and files the way they're given in the assignments. If our automatic checks don't find your files, you'll lose points!

You should run automatic checks before submitting -- have a look at TESTS.md. Code that crashes during the automatic checks will not get any points. You may fail the checks and still get full points, or ace the checks and get no points (especially if your code games the checks). Note that you should update your checkout since the code for the assignments might be changed during the semester.

All deadlines are 23:59:59 CET/CEST.

Index

1. Domain selection

2. Dataset exploration

3. Rule-based Natural Language Understanding

4. Statistical Natural Language Understanding

1. Domain selection

 Presented: 20 February, Deadline: 10 March

You will be building a task-oriented dialogue system in (some of) the homeworks for this course. Your first task is to choose a domain and imagine how your system will look like and work like. Since you might later find that you don't like the domain, you are now required to pick two, so you have more/better ideas later and can choose only one of them for building the system.

Requirements

The required steps for this homework are:

1. Pick two domains of your liking that are suitable for building a task-oriented dialogue system. Think of a reasonable backend (see below).

2. Write 5 example system-user dialogues for both domains, which are at least 5+5 turns long (5 sentences for both user and system). This will make sure that your domain is interesting enough. You do not necessarily have to use English here (but it's easier if we understand the language you're using -- ask us if unsure, Czech & Slovak are perfectly fine).

3. Create a flowchart for your two domains, with labels such as “ask about phone number”, “reply with phone number”, “something else” etc. It should cover all of your example dialogues. You can use e.g. Mermaid to do this, but the format is not important. Feel free to draw this by hand and take a photo, as long as it's legible.

   • It's OK (even better) if your example dialogues don't go all in a straight line (e.g. some of them might loop or go back to the start).

Files to commit

Please stick to the file naming conventions -- you will lose points if you don't!

• hw1/README.md with short commentary on both domains (ca. 10-15 sentences) -- what they are, what features you'd like to include, what will be the backend.

• hw1/examples-<domain1>.txt, hw1/examples-<domain2>.txt -- 5 example dialogues for each of the domains (as described above). Use a short domain name, best with just letters and underscores.

  • Please use UTF-8 encoding for these files.

• hw1/flowchart-<domain1>.{pdf,jpg,png}, hw1/flowchart-<domain2>.{pdf,jpg,png} -- the flowcharts for each of the domains, as described above.

See the instructions on submission via Git -- create a branch and a merge request with your changes. Make sure to name your branch hw1 so we can find it easily.

Inspiration

You may choose any domain you like, be it tourist information, information about culture events/traffic, news, scheduling/agenda, task completion etc. You can take inspiration from stuff presented in the first lecture, or you may choose your own topic.

Since your domain will likely need to be connected to some backend database, you might want to make use of some external public APIs -- feel free to choose under one of these links:

• List of Free APIs
• Another list of free APIs
• An informal listing of various Czech APIs

You can of course choose anything else you like as your backend, e.g. portions of Wikidata/DBPedia or other world knowledge DBs, or even a handwritten “toy” database of a meaningful size, which you'll need to write to be able to test your system.

2. Dataset exploration

 Presented: 27 February, Deadline: 24 March (extended!)

The task in this lab is to explore dialogue datasets and find out more about them. Your job will thus be to write a script that computes some basic statistics about datasets, and then try to interpret the script's results.

Requirements

1. Take a look at the Dialog bAbI Tasks Data 1-6 dataset. Read the description of the data format in the readme.txt file. You'll be working with Tasks 5 and 6 (containing full generated dialogues and DSTC2 data). Use the training sets for Task 5 and Task 6.

2. Write a script that will read all turns in the data and separate the user and system utterances in the training set.

   • Make the script ignore any search results lines in the data (they don't contain a tab character).
   • If the script finds a turn where the user is silent (the user turn contains only <SILENCE>), it should concatenate the system response from this turn to the previous turn. Note that this may happen on multiple consecutive turns, and the script should join all of these together into one system response.
     • If <SILENCE> is the first word in the dialogue, just delete it.
   • For tokenization, you should use the word_tokenize function from the nltk package.

3. Implement a routine that will compute the following statistics for both bAbI tasks for system and user turns (separately, i.e., 4 sets of statistics altogether):

   • data length (total number of dialogues, turns, words)
   • mean and standard deviations for individual dialogue lengths (number of turns in a dialogue, number of words in a turn)
   • vocabulary size
   • Shannon entropy and bigram conditional entropy, i.e. entropy conditioned on 1 preceding word (see lecture 2 slides)

   Commit this file as hw2/stats.py.

4. Along with your script, submit also a dump of the results. The results should be formatted as JSON file with the following structure:

{
    "task5_user":
    {
        "dialogues_total": XXX,
        "turns_total": XXX,
        "words_total": XXX,
        "mean_dialogue_turns": XXX,
        "stddev_dialogue_turns": XXX,
        "mean_dialogue_words_per_turn": XXX,
        "stddev_dialogue_words_per_turn": XXX,
        "vocab_size": XXX,
        "entropy": XXX
        "cond_entropy": XXX
    },
    "task5_system": ...
    "task6_user": ...
    "task6_system": ...
}

(Create a dict and use json.dump for this.) Commit the json file as hw2/results.json.

5. Add your own comments, comparing the results between the two bAbI Tasks. 3-5 sentences is enough, but try to explain why you think the vocabulary and entropy numbers are different.

   Put your comments in Markdown as hw2/stats.md.

Files to commit

There are empty files ready for you in the repo the right places, you just need to fill them with information.

Just to sum up, the files are:

• hw2/stats.py -- your data analysis script
• hw2/results.json -- JSON results of the analysis
• hw2/stats.md -- your comments

Create a branch and a merge request containing (changes to) all requested files. Please keep the filenames and directory structure.

Notes

• Don't worry too much about the exact numbers you get, slight variations in implementation may cause them to change. We won't penalize it if you don't get the exact same numbers as us, the main point is that your implementation should be reasonable (and you shouldn't be off by orders of magnitude).

• Don't worry about system api_calls, just treat them as a normal system turn.

3. Rule-based Natural Language Understanding

 Presented: 13 March, Deadline: 31 March

In this assignment, you will design a dialogue system component for Natural Language Understanding in a specific domain. For completing it, you will use our prepared Dialmonkey dialogue framework (which is the base of your Gitlab repository), so you can test the outcome directly.

Language understanding means converting user utterances (such as “I'm looking for a Chinese restaurant in the city center”) into some formal representation used by the dialogue manager. We'll use dialogue acts as our representation – so the example sentence would convert to something like inform(food=Chinese,area=centre) or find_restaurant(food=Chinese,area="city center"), depending on how you define the intents, slots and values within the dialogue acts for your own domain.

Note: We're not thinking about how to reply just yet! The only thing we're concerned with is representing user inputs in our domain with reasonable intents, slots, and values.

Requirements

1. Make yourself familiar with the Dialmonkey-npfl123 repository you cloned for the homeworks. Read the README and look around a bit to get familiar with the code. Have a look at the 101 Jupyter notebook to see some examples.

2. Recall the domains you picked in the first homework assignment and choose one of them. If you've changed your mind in the meantime, you can even pick a different domain.

3. Think of the set of dialogue acts suitable to describe this domain, i.e., list all intents, slots and values that will be needed (some slots may have open sets of values, e.g. “restaurant name”, “artist name”, “address” etc.). List them, give a description and examples in Markdown under hw3/README.md.

4. Create a component in the dialogue system (as Python code) that:

   • inherits from dialmonkey.component.Component
   • is placed under dialmonkey.nlu.rule_<your-domain>
   • implements a rule-based NLU for your domain -- i.e., given user utterance, finds its intent, slots and values
   • yields Dialogue Act items you designed in step 3 (as DA objects).

   Please only use the core Python libraries and those listed in requirements.txt. If you have a good reason to use a different library from PyPi, let us know and we can discuss adding it into the requirements (but this will be global for everyone).

5. Create a YAML config file for your domain in the conf directory. You can use the sample_conf.yaml file or nlu_test.yaml as a starting point. These files are almost identical, just have a look at the I/O setup if you're interested. Note that instead of a policy and NLG components, a system with this config will simply reply with the NLU result.

6. Write at least 15 distinct utterances that demonstrate the functionality of your class (a tab-separated file with input + corresponding NLU result, one-per-line). Make sure your NLU gives you the same results. The test utterances can (but don't have to) be taken over from the example dialogues you wrote earlier for your domain. Save them as hw3/examples.tsv.

Files to include in your merge request

There will be empty files ready for you in the right place, just rename them according to your domain (e.g. change <your_domain> to restaurant, bus etc.) and fill them with the required content.

• Lists of intents, slots and values in hw3/README.md
• Your NLU component in dialmonkey/nlu/rule_<your_domain>.py
• Your configuration file in conf/nlu_<your_domain>.yaml
• Example test utterances & outputs in hw3/examples.tsv

Create a branch and a merge request containing (changes to) all requested files. Please keep the filenames and directory structure.

Hints

Use regular expressions or keyword matching to find the intents and slot values (based on the value, you'll know which slot it belongs to).

If you haven't ever used regular expressions, have a look some tutorials:

• Unofficial, but easy to read
• Official, though harder to read

Note that you might later need to improve your NLU to handle contextual requests, but you don't need to worry about this now. For instance, the system may ask What time do you want to leave? and the user replies just 7pm. From just 7pm (without the preceding question), you don't know if that's a departure or arrival time. Once you have your dialogue policy ready and know how the system questions look like (which will be the 6th homework), you'll be able to look at the last system question and disambiguate. For now, you can keep these queries ambiguous (e.g. just mark the slot as “time”).

4. Statistical Natural Language Understanding

 Presented: 20 March, Deadline: 7 April

In this assignment, you will build and evaluate a statistical Natural Language Understanding component on the DSTC2 restaurant information data. For completing it, you will use the Dialmonkey framework in your code checkout so you can test the outcome directly.

Requirements

1. Locate the data in your Dialmonkey-NPFL123 repository in data/hw4/. This is what we'll work with for this assignment.

2. Implement a script that trains statistical models to predict DAs. We'll use classifiers here. By default, your approach shouldn't be to predict the whole DA as a single classifier, rather you should classify the correct value for each intent-slot pair where applicable (e.g. inform(food) has multiple possible values) and classify a binary 0-1 for each intent-slot pair that can't have different values (e.g. request(price) or bye() ).

   Don't forget that for the multi-value slots (including 2-valued slots with Y/N), you'll need a “null” value too. Have a look at examples here to get a better idea.

   You can use any kind of statistical classifier you like (e.g. logistic regression, SVM, neural network, including pretrained models or LLMs), with any library of your choice (e.g. Scikit-Learn, Tensorflow, Pytorch ).

   Using binary classifiers for everything may be an option too (especially if you use neural networks with shared layers), but the number of outputs will be rather high, hence the recommendation for multi-class classifiers. Note that we can't do slot tagging here, as the individual words in the texts aren't tagged with slot values.

   In case you want to use an LLM, we prefer using a (small!) model locally to prompting ChatGPT or the like. In this case, you don't have to stick to individual classifications and may want to have the model predict everything -- just make sure the output gets the whole DA from the inputs you have available.

3. Train this model on the training set. You can use the development set for parameter tuning. Using dialmonkey.DA.parse_cambridge_da() should help you get the desired DA values out of the textual representation. Do not look at the test set at this point!

   In case you're using LLMs, this would mean prompt tuning -- try out at least 3 different variations of the prompt.

4. Evaluate your model on the test set and report the overall precision, recall and F1 over dialogue act items (triples of intent-slot-value).

   Use the script provided in dialmonkey.evaluation.eval_nlu. You can run it directly from the console like this:

   ./dialmonkey/evaluation/eval_nlu.py -r data/hw4/dstc2-nlu-test.json -p hw4/predicted.txt

   The script expects reference JSON in the same format as your data here, and a system output with one DA per line. You can have a look at conf/nlu_test.yaml to see how to get one-per-line DA output.

   For the purpose of our evaluation script F1 computation, non-null values count as positives, null values count as negatives. Whether they're true or false depends on whether they're correctly predicted.

5. Implement a module in Dialmonkey that will load your NLU model and work with inputs in the restaurant domain. Create a copy of the nlu_test.yaml config file to work with your new NLU.

Files to include in your merge request

• Your statistical NLU module as dialmonkey/nlu/stat_dstc.py and your trained model. The filename for the model will depend on your implementation, it just needs to be loaded automatically for your model to work. Preferrably store in the same directory, but it could be anywhere else if needed. If the file is too big for Git (>10MB), share it using a cloud service (e.g. CESNET OwnCloud) and make your NLU module download it automatically in __init__().
• Your YAML config file under conf/nlu_dstc.yaml.
• Your training script as hw4/train_nlu.py.
• Your predicted output as hw4/predicted.txt and a short evaluation report under hw4/README.md (including your F1 scores).

Important implementation notes

Note 1: There are templates for you to get you started quickly with the training script and the prediction Dialmonkey module. Feel free to change them in any way you like or need, they're not binding (just make sure your code does what's expected).

Note 2: Please do not use any Python libraries other than the ones in requirements.txt, plus the following ones: torch (Pytorch), tensorflow, pytorch-lightning, torchtext, transformers (Huggingface). Note that scikit-learn is included already. If you need any others, please let us know beforehand.

Note 3: Please make sure that your code doesn't take more than a minute to load + classify the first 20 entries in the test data. For the sake of model storage, it's better to choose a smaller one, especially if you choose to play with pretrained language models.

Note 4: And this one is for all further assignments -- Do not use absolute file paths in your solution! You may want to try out os.path.dirname(__file__), which gets you the directory of the current source file. If you need to use slashes in a (relative!) path, use either os.path.join instead, or use forward slashes and a pathlib.Path object (simply create it using filename = Path(filename)). This will handle the slashes properly both on Windows and Linux. Note that we're mainly checking on Linux, so any backslashes in file paths will break our workflow and we may deduce points.

Hints

• Start playing with the classifier separately, only integrate it into Dialmonkey after you've trained a model and can load it.

• If you have never used a machine learning tool, have a look at the Scikit-Learn tutorial. It contains most of what you'll need to finish this exercise.

• You'll need to convert your texts into something your classifier understands (i.e., some input numerical features). You can probably do very well with just “bag-of-words” as input features to the classifier -- that means that you'll have a binary indicator for each word from the training data (e.g. word “restaurant”). The feature for the word “restaurant” will be 1 if the word “restaurant” appears in the sentence, 0 if it doesn't. You can also try using the same type of features for bigrams. Have a look at the DictVectorizer class in Scikit-Learn. You may also want to consider CountVectorizer, which could speed up things even more.

• For Scikit-Learn, you can use pickle to store your trained models. If you want to pickle classes, use the dill library instead (since pickle can't do that).

• To easily load JSON files, you can use the SimpleJSONInput class.

• You better don't use the naive Bayes classifier, it doesn't work well on this data – basically anything else works better (you won't lose any points if you use naive Bayes, just don't expect good performance).

Homework Submission Instructions

All homework assignments will be submitted using a Git repository on MFF GitLab.

We provide an easy recipe to set up your repository below:

Creating the repository

1. Log into your MFF gitlab account. Your username and password should be the same as in the CAS, see this.
2. You'll have a project repository created for you under the teaching/NPFL123/2025 group. The project name will be the same as your CAS username. If you don't see any repository, it might be the first time you've ever logged on to Gitlab. In that case, Ondřej first needs to run a script that creates the repository for you (please let him know on Slack). In any case, you can explore everything in the base repository. Your own repo will be derived from this one.
3. Clone your repository.
4. Change into the cloned directory and run

git remote show origin

You should see these two lines:

* remote origin
  Fetch URL: git@gitlab.mff.cuni.cz:teaching/NPFL123/2025/your_username.git
  Push  URL: git@gitlab.mff.cuni.cz:teaching/NPFL123/2025/your_username.git

5. Add the base repository (with our code, for everyone) as your upstream:

git remote add upstream https://gitlab.mff.cuni.cz/teaching/NPFL123/base.git

6. You're all set!

Submitting the homework assignment

1. Make sure you're on your master branch

git checkout master

2. Checkout a new branch -- make sure to name it hwX (e.g. “hw4” or “hw11”) so our automatic checks can find it later!

git checkout -b hwX

3. Solve the assignment :)

4. Add new files and commit your changes -- make sure to name your files as required, or you won't pass our automatic checks!

git add hwX/solution.py
git commit -am "commit message"

5. Push to your origin remote repository:

git push origin hwX

6. Create a Merge request in the web interface. Make sure you create the merge request into the master branch in your own forked repository (not into the upstream).

    Merge requests -> New merge request
   

7. Wait a bit till we check your solution, then enjoy your points :)!
8. Once approved, merge your changes into your master branch – you might need them for further homeworks.

Updating from the base repository

You might need to update from the upstream base repository every once in a while (most probably before you start implementing each assignment). We'll let you know when we make changes to the base repo.

To upgrade from upstream, do the following:

1. Make sure you're on your master branch

git checkout master

2. Fetch the changes

git fetch upstream master

3. Apply the diff

git merge upstream/master master

Exam Question Pool

The exam will have 10 questions, mostly from this pool. Each counts for 10 points. We reserve the right to make slight alterations or use variants of the same questions. Note that all of them are covered by the lectures, and they cover most of the lecture content. In general, none of them requires you to memorize formulas, but you should know the main ideas and principles. See the Grading tab for details on grading.

Introduction

• What's the difference between task-oriented and non-task-oriented systems?
• Describe the difference between closed-domain, multi-domain, and open-doman systems.
• Describe the difference between user-initiative, mixed-initiative, and system-initiative systems.

Linguistics of Dialogue

• What are turn taking cues/hints in a dialogue? Name a few examples.
• Explain the main idea of the speech acts theory.
• What is grounding in dialogue?
• Give some examples of grounding signals in dialogue.
• What is deixis? Give some examples of deictic expressions.
• What is coreference and how is it used in dialogue?
• What does Shannon entropy and conditional entropy measure? No need to give the formula, just the principle.
• What is entrainment/adaptation/alignment in dialogue?

Data & Evaluation

• What are the typical options for collecting dialogue data?
• How does Wizard-of-Oz data collection work?
• What is corpus annotation, what is inter-annotator agreement?
• What is the difference between intrinsic and extrinsic evaluation?
• What is the difference between subjective and objective evaluation?
• What are the main extrinsic evaluation techniques for task-oriented dialogue systems?
• What are some evaluation metrics for non-task-oriented systems (chatbots)?
• What's the main metric for evaluating ASR systems?
• What's the main metric for NLU (both slots and intents)?
• Explain an NLG evaluation metric of your choice.
• Why do you need to check for statistical significance (when evaluating an NLP experiment and comparing systems)?
• Why do you need to evaluate on a separate test set?

Natural Language Understanding

• What are some alternative semantic representations of utterances, in addition to dialogue acts?
• Describe language understanding as classification and language understanding as sequence tagging.
• How do you deal with conflicting slots or intents in classification-based NLU?
• What is delexicalization and why is it helpful in NLU?
• Describe one of the approaches to slot tagging as sequence tagging.
• What is the IOB/BIO format for slot tagging?
• What is the label bias problem?
• How can an NLU system deal with noisy ASR output? Propose an example solution.

Neural NLU & Dialogue State Tracking

• Describe a neural architecture for NLU.
• What is the dialogue state and what does it contain?
• What is an ontology in task-oriented dialogue systems?
• Describe the task of a dialogue state tracker.
• What's a partially observable Markov decision process?
• Describe a viable architecture for a belief state tracker.
• What is the difference between dialogue state and belief state?
• What's the difference between a static and a dynamic state tracker?

Dialogue Policies

• What are the non-statistical approaches to dialogue management/action selection?
• Why is reinforcement learning preferred over supervised learning for training dialogue managers?
• Describe the main idea of reinforcement learning (agent, environment, states, rewards).
• What are deterministic and stochastic policies in dialogue management?
• What's a value function in a reinforcement learning scenario?
• What's the difference between actor and critic methods in reinforcement learning?
• What's the difference between model-based and model-free approaches in RL?
• What are the main optimization approaches in reinforcement learning (what measures can you optimize and how)?
• Why do you typically need a user simulator to train a reinforcement learning dialogue policy?

Neural Policies & Natural Language Generation

• How do you involve neural networks in reinforcement learning (describe a Q network or a policy network)?
• What are the main steps of a traditional NLG pipeline – describe at least 2.
• Describe one approach to NLG of your choice.
• Describe how template-based NLG works.
• What are some problems you need to deal with in template-based NLG?
• Describe a possible neural networks based NLG architecture.

Dialogue Tooling

• What is a dialogue flow/tree?
• What are intents and entities/slots?
• How can you improve a chatbot in production?
• What is the containment rate (in the context of using dialogue systems in call centers)?
• What is retrieval-augmented generation?

Voice assistants & Question Answering

• What is a smart speaker made of and how does it work?
• Briefly describe a viable approach to question answering.
• What is document retrieval and how is it used in question answering?
• What is dense retrieval?
• How can you use neural models in answer extraction?
• How can you use retrieval-augmented generation in question answering?
• What is a knowledge graph?

Automatic Speech Recognition

• What is a speech activity detector?
• Describe the main components of an ASR pipeline system.
• How do input features for an ASR model look like?
• What is the function of the acoustic model in a pipeline ASR system?
• What's the function of a decoder/language model in a pipeline ASR system?
• Describe the architecture of an end-to-end neural ASR system.

Text-to-speech Synthesis

• How do humans produce sounds of speech?
• What's the difference between a vowel and a consonant?
• What is F0 and what are formants?
• What is a spectrogram?
• What are main distinguishing characteristics of consonants?
• What is a phoneme?
• What are the main distinguishing characteristics of different vowel phonemes (both how they're produced and perceived)?
• What are the main approaches to grapheme-to-phoneme conversion in TTS?
• Describe the main idea of concatenative speech synthesis.
• Describe the main ideas of statistical parametric speech synthesis.
• How can you use neural networks in speech synthesis?

Chatbots

• What are the three main approaches to building chitchat/non-task-oriented open-domain chatbots?
• How does the Turing test work? Does it have any weaknesses?
• What are some techniques rule-based chitchat chatbots use to convince their users that they're human-like?
• Describe how a retrieval-based chitchat chatbot works.
• How can you use neural networks for chatbots (non-task-oriented, open-domain systems)? Does that have any problems?
• Describe a possible architecture of an ensemble non-task-oriented chatbot.
• What do you need to train a large language model?
• What are some issues you may encounter when chatting to LLMs?

Course Grading

To pass this course, you will need to:

1. Take an exam (a written test covering important lecture content).
2. Do lab homeworks (various dialogue system implementation tasks).

Exam test

• There will be a written exam test at the end of the semester.
• There will be 10 questions, we expect 2-3 sentences as an answer, with a maximum of 10 points per question.
• To pass the course, you need to get at least 50% of the total points from the test.
• We plan to publish a list of possible questions beforehand.

In case the pandemic does not get better by the exam period, there will be a remote alternative for the exam (an essay with a discussion).

Homework assignments

• There will be 12 homework assignments, introduced every week, starting on the 2nd week of the semester.
• You will submit the homework assignments into a private Gitlab repository (where we will be given access).
• For each assignment, you will get a maximum of 10 points.
• All assignments will have a fixed deadline.
• If you submit the assignment after the deadline, you will get:
  • up to 50% of the maximum points if it is less than 2 weeks after the deadline;
  • 0 points if it is more than 2 weeks after the deadline.
• Once we check the submitted assignments, you will see the points you got and the comments from us as comments on your merge requests on Gitlab.
• You need to get at least 50% of the total assignments points to pass the course.
  • You can take the exam even if you don't have 50% yet (esp. due to potential delays in grading), but you'll need to get the required points eventually.

Grading

The final grade for the course will be a combination of your exam score and your homework assignment score, weighted 3:1 (i.e. the exam accounts for 75% of the grade, the assignments for 25%).

Grading:

• Grade 1: >=87% of the weighted combination
• Grade 2: >=74% of the weighted combination
• Grade 3: >=60% of the weighted combination
• An overall score of less than 60% means you did not pass.

In any case, you need >50% of points from the test and >50% of points from the homeworks to pass. If you get less than 50% from either, even if you get more than 60% overall, you will not pass.

No cheating

• Cheating is strictly prohibited and any student found cheating will be punished. The punishment can involve failing the whole course, or, in grave cases, being expelled from the faculty.
• Discussing homework assignments with your classmates is OK. Sharing code is not OK (unless explicitly allowed); by default, you must complete the assignments yourself.
• All students involved in cheating will be punished. E.g. if you share your assignment with a friend, both you and your friend will be punished.

Recommended Reading

You should pass the course just by following the lectures, but here are some hints on further reading. There's nothing ideal on the topic as this is a very active research area, but some of these should give you a broader overview.

Basic (good and up-to-date, but very brief, available online):

• Jurafsky & Martin: Speech & Language processing. 3rd ed. draft (chapters 14-16).

More detailed (very good but slightly outdated, available as e-book from our library):

• McTear: Conversational AI: Dialogue Systems, Conversational Agents, and Chatbots. Morgan & Claypool 2021.

Further reading (mostly outdated but still informative):

• Janarthanam: Hands-On Chatbots and Conversational UI Development. Packt 2017.
  • practical guide on developing dialogue systems for voice bot platforms, virtually no theory
• Gao et al.: Neural Approaches to Conversational AI. arXiv:1809.08267
  • an advanced, good overview of basic neural approaches in dialogue systems
• McTear et al.: The Conversational Interface: Talking to Smart Devices. Springer 2016.
  • practical, more advanced and more theory than Janarthanam
• Jokinen & McTear: Spoken dialogue systems. Morgan & Claypool 2010.
  • good but outdated, some systems very specific to particular research projects
• Rieser & Lemon: Reinforcement learning for adaptive dialogue systems. Springer 2011.
  • advanced, outdated, project-specific
• Lemon & Pietquin: Data-Driven Methods for Adaptive Spoken Dialogue Systems. Springer 2012.
  • ditto
• Skantze: Error Handling in Spoken Dialogue Systems. PhD Thesis 2007, Chap. 2.
  • good introduction into dialogue systems in general, albeit dated
• McTear: Spoken Dialogue Technology. Springer 2004.
  • good but dated
• Psutka et al.: Mluvíme s počítačem česky. Academia 2006.
  • virtually the only book in Czech, good for ASR but dated, not a lot about other parts of dialogue systems