SIS code: 
Semester: 
summer
E-credits: 
5
Examination: 
C+Ex
Instructor: 
Kristýna Klesnilová, Jan Cuřín

NPFL123 – Dialogue Systems

About

This course is a detailed introduction into the architecture of spoken dialogue systems, voice assistants and conversational systems (chatbots). We will introduce the main components of dialogue systems (speech recognition, language understanding, dialogue management, language generation and speech synthesis) and show alternative approaches to their implementation.

The lab sessions will be dedicated to implementing a simple dialogue system and selected components (via weekly homework assignments). We will use Python and a version of our Dialmonkey framework for this.

Logistics (spring 2024)

Schedule change

Since Ondrej is away on March 20, the lecture on March 20 is CANCELLED.

And since we need to go through 12 lectures and have only 12 weeks (due to 1 and 8 May falling on Wednesdays), we'll rearrange the schedule this way:

  • 13 March: 2 lectures -- 4. Language understanding & 5. Dialogue state tracking
    • we’ll finish around 6:50pm, so it'll be a bit longer
  • 20 March: CANCELLED
  • 27 March: 1 lecture (6. Dialogue policies) + HW4 & HW5
    • this might take slightly longer
    • HW deadlines will be spaced apart
  • 3 April: 1 lecture (7. Neural policies & NLG) + HW6 & HW7
    • usual length, it’s actually best to do these two HW together

Language

The course will be taught in English, but we're happy to explain in Czech, too.

Time & Place

In-person lectures and labs take place in the Malá Strana building.

  • Lectures: Wed 15:40, room S5 (2nd floor)
  • Labs: Wed 17:20, room S5 (2nd floor)

In addition, we plan to stream both lectures and lab instruction over Zoom and make the recordings available on Youtube (under a private link, on request). We'll do our best to provide a useful experience.

  • Zoom meeting ID: 981 1966 7454
  • Password is the SIS code of this course (capitalized)

There's also a Slack workspace you can use to discuss assignments and get news about the course. Invite links will be sent out to all enrolled students by the start of the semester. Please contact us by email if you want to join and haven't got an invite yet.

Passing the course

To pass this course, you will need to take an exam and do lab homeworks. There's a 60% points minimum for the exam and 50% for the homeworks to pass the course. See more details here.

Topics covered

Dialogue systems schema
  • Dialogue system types & formats (open/closed domain, task/chat-oriented)
  • Data for dialogue systems
  • Dialogue systems evaluation
  • What happens in a dialogue (linguistic background)
  • Dialogue system components
    • speech recognition
    • language understanding, dialogue state tracking
    • dialogue management
    • language generation
    • speech synthesis
  • Voice assistants & question answering
  • Dialogue authoring tools (IBM Watson Assistant/Google Assistant/Amazon Alexa)
  • Open-domain/chitchat chatbots

Lectures

PDFs with lecture slides will appear here shortly before each lecture (more details on each lecture are on a separate tab). You can also check out last year's lecture slides.

1. Introduction Slides Domain selection Questions

2. What happens in a dialogue? Slides Dataset exploration Questions

3. Data & Evaluation Slides Rule-based Natural Language Understanding Questions

4. Natural Language Understanding Slides Questions Statistical Natural Language Understanding

5. Neural NLU + State Tracking Slides Questions Belief State Tracking

6. Dialogue Policy (non-neural) Slides Questions


Literature

A list of recommended literature is on a separate tab.

Lectures

1. Introduction

 21 February Slides Domain selection Questions

  • What are dialogue systems
  • Common usage areas
  • Task-oriented vs. non-task oriented systems
  • Closed domain, multi-domain, open domain
  • System vs. user initiative in dialogue
  • Standard dialogue systems components

2. What happens in a dialogue?

 28 February Slides Dataset exploration Questions

  • Dialogue turns
  • Utterances as acts, pragmatics
  • Grounding, grounding signals
  • Deixis
  • Conversational maxims
  • Prediction and adaptation

3. Data & Evaluation

 6 March Slides Rule-based Natural Language Understanding Questions

  • How to get data for building dialogue systems
  • Available corpora/datasets
  • Annotation
  • Data splits
  • Evaluation metrics -- subjective & objective, intrinsic & extrinsic
  • Significance checks

4. Natural Language Understanding

 13 March Slides Questions Statistical Natural Language Understanding

  • What needs to be handled to understand the user
  • How to represent meaning: grammars, frames, graphs, dialogue acts (“shallow parsing”)
  • Rule-based NLU
  • Basics of machine learning, discriminative & generative classifiers
  • Classification-based NLU (features, logistic regression, SVM)
  • Sequence tagging (HMM, MEMM, CRF)

5. Neural NLU + State Tracking

 13 March Slides Questions Belief State Tracking

  • Some basics about neural networks
  • How to use neural networks for NLU: neural classifiers and sequence taggers
  • Handling speech recognition noise
  • What is dialogue state, what is belief state, and what they're good for
  • Dialogue as a Markov decision process (MDP)
  • Dialogue trackers: generative and discriminative
  • Static and dynamic trackers

6. Dialogue Policy (non-neural)

 27 March Slides

Questions

  • What's a dialogue policy -- how to choose the next action
  • Finite-state, frame-based, and rule-based policies
  • Reinforcement learning basics
  • Value and policy optimization (SARSA, Q-learning, REINFORCE)
  • Mapping to POMPDs (partially observable MDPs)
  • Summary space (making it tractable)
  • User simulation

Homework Assignments

There will be 12 homework assignments, each for a maximum of 10 points. Please see details on grading and deadlines on a separate tab. Note that there's a 50% minimum requirement to pass the course.

Assignments should be submitted via Git – see instructions on a separate tab. Please take special care about naming your Git branches and files the way they're given in the assignments. If our automatic checks don't find your files, you'll lose points!

You should run automatic checks before submitting -- have a look at TESTS.md. Code that crashes during the automatic checks will not get any points. You may fail the checks and still get full points, or ace the checks and get no points (especially if your code games the checks). Note that you should update your checkout since the code for the assignments might be changed during the semester.

All deadlines are 23:59:59 CET/CEST.

Index

1. Domain selection

2. Dataset exploration

3. Rule-based Natural Language Understanding

4. Statistical Natural Language Understanding

5. Belief State Tracking

1. Domain selection

 Presented: 21 February, Deadline: 8 March

You will be building a task-oriented dialogue system in (some of) the homeworks for this course. Your first task is to choose a domain and imagine how your system will look like and work like. Since you might later find that you don't like the domain, you are now required to pick two, so you have more/better ideas later and can choose only one of them for building the system.

Requirements

The required steps for this homework are:

  1. Pick two domains of your liking that are suitable for building a task-oriented dialogue system. Think of a reasonable backend (see below).

  2. Write 5 example system-user dialogues for both domains, which are at least 5+5 turns long (5 sentences for both user and system). This will make sure that your domain is interesting enough. You do not necessarily have to use English here (but it's easier if we understand the language you're using -- ask us if unsure, Czech & Slovak are perfectly fine).

  3. Create a flowchart for your two domains, with labels such as “ask about phone number”, “reply with phone number”, “something else” etc. It should cover all of your example dialogues. You can use e.g. Mermaid to do this, but the format is not important. Feel free to draw this by hand and take a photo, as long as it's legible.

    • It's OK (even better) if your example dialogues don't go all in a straight line (e.g. some of them might loop or go back to the start).

Files to commit

Please stick to the file naming conventions -- you will lose points if you don't!

  • hw1/README.md with short commentary on both domains (ca. 10-15 sentences) -- what they are, what features you'd like to include, what will be the backend.

  • hw1/examples-<domain1>.txt, hw1/examples-<domain2>.txt -- 5 example dialogues for each of the domains (as described above). Use a short domain name, best with just letters and underscores.

    • Please use UTF-8 encoding for these files.
  • hw1/flowchart-<domain1>.{pdf,jpg,png}, hw1/flowchart-<domain2>.{pdf,jpg,png} -- the flowcharts for each of the domains, as described above.

See the instructions on submission via Git -- create a branch and a merge request with your changes. Make sure to name your branch hw1 so we can find it easily.

Inspiration

You may choose any domain you like, be it tourist information, information about culture events/traffic, news, scheduling/agenda, task completion etc. You can take inspiration from stuff presented in the first lecture, or you may choose your own topic.

Since your domain will likely need to be connected to some backend database, you might want to make use of some external public APIs -- feel free to choose under one of these links:

You can of course choose anything else you like as your backend, e.g. portions of Wikidata/DBPedia or other world knowledge DBs, or even a handwritten “toy” database of a meaningful size, which you'll need to write to be able to test your system.

2. Dataset exploration

 Presented: 28 February, Deadline: 15 March

The task in this lab is to explore dialogue datasets and find out more about them. Your job will thus be to write a script that computes some basic statistics about datasets, and then try to interpret the script's results.

Requirements

  1. Take a look at the Dialog bAbI Tasks Data 1-6 dataset. Read the description of the data format in the readme.txt file. You'll be working with Tasks 5 and 6 (containing full generated dialogues and DSTC2 data). Use the training sets for Task 5 and Task 6.

  2. Write a script that will read all turns in the data and separate the user and system utterances in the training set.

    • Make the script ignore any search results lines in the data (they don't contain a tab character).
    • If the script finds a turn where the user is silent (the user turn contains only <SILENCE>), it should concatenate the system response from this turn to the previous turn. Note that this may happen on multiple consecutive turns, and the script should join all of these together into one system response.
      • If <SILENCE> is the first word in the dialogue, just delete it.
    • For tokenization, you should use the word_tokenize function from the nltk package.
  3. Implement a routine that will compute the following statistics for both bAbI tasks for system and user turns (separately, i.e., 4 sets of statistics altogether):

    • data length (total number of dialogues, turns, words)
    • mean and standard deviations for individual dialogue lengths (number of turns in a dialogue, number of words in a turn)
    • vocabulary size
    • Shannon entropy and bigram conditional entropy, i.e. entropy conditioned on 1 preceding word (see lecture 2 slides)

    Commit this file as hw2/stats.py.

  4. Along with your script, submit also a dump of the results. The results should be formatted as JSON file with the following structure:

{
    "task5_user":
    {
        "dialogues_total": XXX,
        "turns_total": XXX,
        "words_total": XXX,
        "mean_dialogue_turns": XXX,
        "stddev_dialogue_turns": XXX,
        "mean_dialogue_words_per_turn": XXX,
        "stddev_dialogue_words_per_turn": XXX,
        "vocab_size": XXX,
        "entropy": XXX
        "cond_entropy": XXX
    },
    "task5_system": ...
    "task6_user": ...
    "task6_system": ...
}

(Create a dict and use json.dump for this.) Commit the json file as hw2/results.json.

  1. Add your own comments, comparing the results between the two bAbI Tasks. 3-5 sentences is enough, but try to explain why you think the vocabulary and entropy numbers are different.

    Put your comments in Markdown as hw2/stats.md.

Files to commit

There are empty files ready for you in the repo the right places, you just need to fill them with information.

Just to sum up, the files are:

  • hw2/stats.py -- your data analysis script
  • hw2/results.json -- JSON results of the analysis
  • hw2/stats.md -- your comments

Create a branch and a merge request containing (changes to) all requested files. Please keep the filenames and directory structure.

Notes

  • Don't worry too much about the exact numbers you get, slight variations in implementation may cause them to change. We won't penalize it if you don't get the exact same numbers as us, the main point is that your implementation should be reasonable (and you shouldn't be off by orders of magnitude).

  • Don't worry about system api_calls, just treat them as a normal system turn.

3. Rule-based Natural Language Understanding

 Presented: 6 March, Deadline: 28 March (extended!)

In this assignment, you will design a dialogue system component for Natural Language Understanding in a specific domain. For completing it, you will use our prepared Dialmonkey dialogue framework (which is the base of your Gitlab repository), so you can test the outcome directly.

Language understanding means converting user utterances (such as “I'm looking for a Chinese restaurant in the city center”) into some formal representation used by the dialogue manager. We'll use dialogue acts as our representation – so the example sentence would convert to something like inform(food=Chinese,area=centre) or find_restaurant(food=Chinese,area="city center"), depending on how you define the intents, slots and values within the dialogue acts for your own domain.

Note: We're not thinking about how to reply just yet! The only thing we're concerned with is representing user inputs in our domain with reasonable intents, slots, and values.

Requirements

  1. Make yourself familiar with the Dialmonkey-npfl123 repository you cloned for the homeworks. Read the README and look around a bit to get familiar with the code. Have a look at the 101 Jupyter notebook to see some examples.

  2. Recall the domains you picked in the first homework assignment and choose one of them. If you've changed your mind in the meantime, you can even pick a different domain.

  3. Think of the set of dialogue acts suitable to describe this domain, i.e., list all intents, slots and values that will be needed (some slots may have open sets of values, e.g. “restaurant name”, “artist name”, “address” etc.). List them, give a description and examples in Markdown under hw3/README.md.

  4. Create a component in the dialogue system (as Python code) that:

    • inherits from dialmonkey.component.Component
    • is placed under dialmonkey.nlu.rule_<your-domain>
    • implements a rule-based NLU for your domain -- i.e., given user utterance, finds its intent, slots and values
    • yields Dialogue Act items you designed in step 3 (as DA objects).

    Please only use the core Python libraries and those listed in requirements.txt. If you have a good reason to use a different library from PyPi, let us know and we can discuss adding it into the requirements (but this will be global for everyone).

  5. Create a YAML config file for your domain in the conf directory. You can use the sample_conf.yaml file or nlu_test.yaml as a starting point. These files are almost identical, just have a look at the I/O setup if you're interested. Note that instead of a policy and NLG components, a system with this config will simply reply with the NLU result.

  6. Write at least 15 distinct utterances that demonstrate the functionality of your class (a tab-separated file with input + corresponding NLU result, one-per-line). Make sure your NLU gives you the same results. The test utterances can (but don't have to) be taken over from the example dialogues you wrote earlier for your domain. Save them as hw3/examples.tsv.

Files to include in your merge request

There will be empty files ready for you in the right place, just rename them according to your domain (e.g. change <your_domain> to restaurant, bus etc.) and fill them with the required content.

  • Lists of intents, slots and values in hw3/README.md
  • Your NLU component in dialmonkey/nlu/rule_<your_domain>.py
  • Your configuration file in conf/nlu_<your_domain>.yaml
  • Example test utterances & outputs in hw3/examples.tsv

Create a branch and a merge request containing (changes to) all requested files. Please keep the filenames and directory structure.

Hints

Use regular expressions or keyword matching to find the intents and slot values (based on the value, you'll know which slot it belongs to).

If you haven't ever used regular expressions, have a look some tutorials:

Note that you might later need to improve your NLU to handle contextual requests, but you don't need to worry about this now. For instance, the system may ask What time do you want to leave? and the user replies just 7pm. From just 7pm (without the preceding question), you don't know if that's a departure or arrival time. Once you have your dialogue policy ready and know how the system questions look like (which will be the 6th homework), you'll be able to look at the last system question and disambiguate. For now, you can keep these queries ambiguous (e.g. just mark the slot as “time”).

4. Statistical Natural Language Understanding

 Presented: 27 March, Deadline: 12 April

In this assignment, you will build and evaluate a statistical Natural Language Understanding component on the DSTC2 restaurant information data. For completing it, you will use the Dialmonkey framework in your code checkout so you can test the outcome directly.

Requirements

  1. Locate the data in your Dialmonkey-NPFL123 repository in data/hw4/. This is what we'll work with for this assignment.

  2. Implement a script that trains statistical models to predict DAs. We'll use classifiers here. Your approach shouldn't be to predict the whole DA as a single classifier, rather you should classify the correct value for each intent-slot pair where applicable (e.g. inform(food) has multiple possible values) and classify a binary 0-1 for each intent-slot pair that can't have different values (e.g. request(price) or bye() ).

    Don't forget that for the multi-value slots, you'll need a “null” value too. Have a look at examples here to get a better idea.

    You can use any kind of statistical classifier you like (e.g. logistic regression, SVM, neural network), with any library of your choice (e.g. Scikit-Learn, Tensorflow, Pytorch ).

    Using binary classifiers for everything may be an option too (especially if you use neural networks with shared layers), but the number of outputs will be rather high, hence the recommendation for multi-class classifiers. Note that we can't do slot tagging here, as the individual words in the texts aren't tagged with slot values.

  3. Train this model on the training set. You can use the development set for parameter tuning. Using dialmonkey.DA.parse_cambridge_da() should help you get the desired DA values out of the textual representation. Do not look at the test set at this point!

  4. Evaluate your model on the test set and report the overall precision, recall and F1 over dialogue act items (triples of intent-slot-value).

    Use the script provided in dialmonkey.evaluation.eval_nlu. You can run it directly from the console like this:

    ./dialmonkey/evaluation/eval_nlu.py -r data/hw4/dstc2-nlu-test.json -p hw4/predicted.txt

    The script expects reference JSON in the same format as your data here, and a system output with one DA per line. You can have a look at conf/nlu_test.yaml to see how to get one-per-line DA output.

    For the purpose of our evaluation script F1 computation, non-null values count as positives, null values count as negatives. Whether they're true or false depends on whether they're correctly predicted.

  5. Implement a module in Dialmonkey that will load your NLU model and work with inputs in the restaurant domain. Create a copy of the nlu_test.yaml config file to work with your new NLU.

Files to include in your merge request

  • Your statistical NLU module as dialmonkey/nlu/stat_dstc.py and your trained model. The filename for the model will depend on your implementation, it just needs to be loaded automatically for your model to work. Preferrably store in the same directory, but it could be anywhere else if needed. If the file is too big for Git (>10MB), share it using a cloud service (e.g. CESNET OwnCloud) and make your NLU module download it automatically in __init__().
  • Your YAML config file under conf/nlu_dstc.yaml.
  • Your training script as hw4/train_nlu.py.
  • Your predicted output as hw4/predicted.txt and a short evaluation report under hw4/README.md (including your F1 scores).

Important implementation notes

Note 1: Please do not use any Python libraries other than the ones in requirements.txt, plus the following ones: torch (Pytorch), tensorflow, pytorch-lightning, torchtext, transformers (Huggingface). Note that scikit-learn is included already. If you need any others, please let us know beforehand.

Note 2: Please make sure that your code doesn't take more than a minute to load + classify the first 20 entries in the test data. For the sake of model storage, it's better to choose a smaller one, especially if you choose to play with pretrained language models.

Note 3: And this one is for all further assignments -- Do not use absolute file paths in your solution! You may want to try out os.path.dirname(__file__), which gets you the directory of the current source file. If you need to use slashes in a (relative!) path, use either os.path.join instead, or use forward slashes and a pathlib.Path object (simply create it using filename = Path(filename)). This will handle the slashes properly both on Windows and Linux. Note that we're mainly checking on Linux, so any backslashes in file paths will break our workflow and we may deduce points.

Hints

  • Start playing with the classifier separately, only integrate it into Dialmonkey after you've trained a model and can load it.

  • If you have never used a machine learning tool, have a look at the Scikit-Learn tutorial. It contains most of what you'll need to finish this exercise.

  • You'll need to convert your texts into something your classifier understands (i.e., some input numerical features). You can probably do very well with just “bag-of-words” as input features to the classifier -- that means that you'll have a binary indicator for each word from the training data (e.g. word “restaurant”). The feature for the word “restaurant” will be 1 if the word “restaurant” appears in the sentence, 0 if it doesn't. You can also try using the same type of features for bigrams. Have a look at the DictVectorizer class in Scikit-Learn. You may also want to consider CountVectorizer, which could speed up things even more.

  • For Scikit-Learn, you can use pickle to store your trained models. If you want to pickle classes, use the dill library instead (since pickle can't do that).

  • To easily load JSON files, you can use the SimpleJSONInput class.

  • You better don't use the naive Bayes classifier, it doesn't work well on this data – basically anything else works better (you won't lose any points if you use naive Bayes, just don't expect good performance).

5. Belief State Tracking

 Presented: 27 March, Deadline: 19 April

This week, you will build a simple probabilistic dialogue/belief state tracker to work with NLU for both your own domain of choice (3rd assignment) and DSTC2 (4th assignment).

Requirements

  1. Implement a dialogue state tracker that works with the dial.state structure (it's a dict) and fills it with a probability distribution of values over each slot (assume slots are independent), updated during the dialogue after each turn. Don't forget None is a valid value, meaning “we don't know/user didn't say anything about this”.

    At the beginning of the dialogue, each slot should be initialized with the distribution {None: 1.0}.

    The update rule for a slot, say food, should go like this:

    • Take all mentions of food in the current NLU, with their probabilities. Say you got Chinese with a probability of 0.7 and Italian with a probability of 0.2. This means None has a probability of 0.1.
    • Use the probability of None to multiply current values with it (e.g. if the distribution was {'Chinese': 0.2, None: 0.8}, it should be changed to {'Chinese': 0.02, None: 0.08}.
    • Now add the non-null values with their respective probabilities from the NLU. This should result in {'Chinese': 0.72, 'Italian': 0.2, None: 0.08}.

    To make sure your tracker works with either NLU system, simply use whatever is in dial.nlu as the input. Make sure both your NLU systems fill in the dial.nlu structure -- feel free to edit your NLU systems' code.

    Note that you should be using probability estimates in your statistical NLU from the 4th assignment now (for your rule-based NLU on own domain from the 3rd assignment, just assume a probability of 1 if your patterns match).

    Set a threshold for low-probability NLU outputs that you'll ignore in the tracker (good choice is 0.01-0.05), so that the outputs are not too messy.

  2. Create new configuration files both for your own rule-based domain and for DSTC2, which include NLU and the tracker. In addition, use dialmonkey.policy.dummy.ReplyWithState as the output (so the system's response is now the current dialogue state).

    You can use conf/dst_test.yaml as a starting point (just put in your NLU and tracker).

  3. Run your NLU + tracker over your NLU examples from 3rd assignment and the first 20 lines from the DSTC2 development data (file data/hw4/dstc2-nlu-dev.json ) from the 4th assignment data and save the outputs to a text file for both. You can use Dialmonkey's file input settings for this.

    Treat the examples as a single dialogue, even though such a “dialogue” doesn't make any logical sense. Your tracker should be able to handle it.

Hints

  • You can have a look at an example dialogue with commentary for the update rule in a separate file (it's basically the same stuff as above, just more detailed).

  • Remember that all this is just about slots, not intents – do not consider intents as part of the state. You generally don't need to track past intents, just the current intent from NLU is enough to guide the system.

  • If your statistical NLU is built as classifiers for intent-slot pairs (as foreseen by default in HW4), you can just use the probabilities coming from the inform intent for the given slot. The best way would be to add up "positive" intents such as inform or confirm and discount negative ones, such as deny, but we don't require you to do this as part of the assignment.

Files to include in your merge request

  • Your tracker code under dialmonkey/dst/rule.py.
  • Your updated configuration files for both domains under conf/dst_<my-domain>.yaml and conf/dst_dstc.yaml.
  • Your text files with the outputs into hw5/outputs_<my-domain>.txt and hw5/outputs_dstc.txt.

Homework Submission Instructions

All homework assignments will be submitted using a Git repository on MFF GitLab.

We provide an easy recipe to set up your repository below:

Creating the repository

  1. Log into your MFF gitlab account. Your username and password should be the same as in the CAS, see this.
  2. You'll have a project repository created for you under the teaching/NPFL123/2024 group. The project name will be the same as your CAS username. If you don't see any repository, it might be the first time you've ever logged on to Gitlab. In that case, Ondřej first needs to run a script that creates the repository for you (please let him know on Slack). In any case, you can explore everything in the base repository. Your own repo will be derived from this one.
  3. Clone your repository.
  4. Change into the cloned directory and run
git remote show origin

You should see these two lines:

* remote origin
  Fetch URL: git@gitlab.mff.cuni.cz:teaching/NPFL123/2024/your_username.git
  Push  URL: git@gitlab.mff.cuni.cz:teaching/NPFL123/2024/your_username.git

  1. Add the base repository (with our code, for everyone) as your upstream:
git remote add upstream https://gitlab.mff.cuni.cz/teaching/NPFL123/base.git
  1. You're all set!

Submitting the homework assignment

  1. Make sure you're on your master branch
git checkout master
  1. Checkout a new branch -- make sure to name it hwX (e.g. “hw4” or “hw11”) so our automatic checks can find it later!
git checkout -b hwX
  1. Solve the assignment :)

  2. Add new files and commit your changes -- make sure to name your files as required, or you won't pass our automatic checks!

git add hwX/solution.py
git commit -am "commit message"
  1. Push to your origin remote repository:
git push origin hwX
  1. Create a Merge request in the web interface. Make sure you create the merge request into the master branch in your own forked repository (not into the upstream).

     Merge requests -> New merge request
    
Merge request
  1. Wait a bit till we check your solution, then enjoy your points :)!
  2. Once approved, merge your changes into your master branch – you might need them for further homeworks.

Updating from the base repository

You might need to update from the upstream base repository every once in a while (most probably before you start implementing each assignment). We'll let you know when we make changes to the base repo.

To upgrade from upstream, do the following:

  1. Make sure you're on your master branch
git checkout master
  1. Fetch the changes
git fetch upstream master
  1. Apply the diff
git merge upstream/master master

Exam Question Pool

The exam will have 10 questions, mostly from this pool. Each counts for 10 points. We reserve the right to make slight alterations or use variants of the same questions. Note that all of them are covered by the lectures, and they cover most of the lecture content. In general, none of them requires you to memorize formulas, but you should know the main ideas and principles. See the Grading tab for details on grading.

Introduction

  • What's the difference between task-oriented and non-task-oriented systems?
  • Describe the difference between closed-domain, multi-domain, and open-doman systems.
  • Describe the difference between user-initiative, mixed-initiative, and system-initiative systems.

Linguistics of Dialogue

  • What are turn taking cues/hints in a dialogue? Name a few examples.
  • Explain the main idea of the speech acts theory.
  • What is grounding in dialogue?
  • Give some examples of grounding signals in dialogue.
  • What is deixis? Give some examples of deictic expressions.
  • What is coreference and how is it used in dialogue?
  • What does Shannon entropy and conditional entropy measure? No need to give the formula, just the principle.
  • What is entrainment/adaptation/alignment in dialogue?

Data & Evaluation

  • What are the typical options for collecting dialogue data?
  • How does Wizard-of-Oz data collection work?
  • What is corpus annotation, what is inter-annotator agreement?
  • What is the difference between intrinsic and extrinsic evaluation?
  • What is the difference between subjective and objective evaluation?
  • What are the main extrinsic evaluation techniques for task-oriented dialogue systems?
  • What are some evaluation metrics for non-task-oriented systems (chatbots)?
  • What's the main metric for evaluating ASR systems?
  • What's the main metric for NLU (both slots and intents)?
  • Explain an NLG evaluation metric of your choice.
  • Why do you need to check for statistical significance (when evaluating an NLP experiment and comparing systems)?
  • Why do you need to evaluate on a separate test set?

Natural Language Understanding

  • What are some alternative semantic representations of utterances, in addition to dialogue acts?
  • Describe language understanding as classification and language understanding as sequence tagging.
  • How do you deal with conflicting slots or intents in classification-based NLU?
  • What is delexicalization and why is it helpful in NLU?
  • Describe one of the approaches to slot tagging as sequence tagging.
  • What is the IOB/BIO format for slot tagging?
  • What is the label bias problem?
  • How can an NLU system deal with noisy ASR output? Propose an example solution.

Neural NLU & Dialogue State Tracking

  • Describe a neural architecture for NLU.
  • What is the dialogue state and what does it contain?
  • What is an ontology in task-oriented dialogue systems?
  • Describe the task of a dialogue state tracker.
  • What's a partially observable Markov decision process?
  • Describe a viable architecture for a belief tracker.
  • What is the difference between dialogue state and belief state?
  • What's the difference between a static and a dynamic state tracker?

Dialogue Policies

  • What are the non-statistical approaches to dialogue management/action selection?
  • Why is reinforcement learning preferred over supervised learning for training dialogue managers?
  • Describe the main idea of reinforcement learning (agent, environment, states, rewards).
  • What are deterministic and stochastic policies in dialogue management?
  • What's a value function in a reinforcement learning scenario?
  • What's the difference between actor and critic methods in reinforcement learning?
  • What's the difference between model-based and model-free approaches in RL?
  • What are the main optimization approaches in reinforcement learning?
  • Why do you typically need a user simulator to train a reinforcement learning dialogue policy?

Neural Policies & Natural Language Generation

  • How do you involve neural networks in reinforcement learning (describe a Q network or a policy network)?
  • What are the main steps of a traditional NLG pipeline – describe at least 2.
  • Describe one approach to NLG of your choice.
  • Describe how template-based NLG works.
  • What are some problems you need to deal with in template-based NLG?
  • Describe a possible neural networks based NLG architecture.

Dialogue Tooling

  • What is a dialogue flow?
  • What are intents and entities/slots?
  • How can you improve a chatbot in production?

Voice assistants & Question Answering

  • What is a smart speaker made of and how does it work?
  • Briefly describe a viable approach to question answering.
  • What is document retrieval and how is it used in question answering?
  • What is a knowledge graph?

Automatic Speech Recognition

  • What is a speech activity detector?
  • Describe the main components of an ASR pipeline system.
  • How do input features for an ASR model look like?
  • What is the function of the acoustic model in a pipeline ASR system?
  • What's the function of a decoder/language model in a pipeline ASR system?
  • Describe the architecture of an end-to-end neural ASR system.

Text-to-speech Synthesis

  • How do humans produce sounds of speech?
  • What's the difference between a vowel and a consonant?
  • What is F0 and what are formants?
  • What is a spectrogram?
  • What are main distinguishing characteristics of consonants?
  • What is a phoneme?
  • What are the main distinguishing characteristics of different vowel phonemes (both how they're produced and perceived)?
  • What are the main approaches to grapheme-to-phoneme conversion in TTS?
  • Describe the main idea of concatenative speech synthesis.
  • Describe the main ideas of statistical parametric speech synthesis.
  • How can you use neural networks in speech synthesis?

Chatbots

  • What are the three main approaches to building chatbots?
  • How does the Turing test work? Does it have any weaknesses?
  • What are some techniques rule-based chatbots use to convince their users that they're human-like?
  • Describe how a retrieval-based chatbot works.
  • How can you use neural networks for chatbots? Does that have any problems?
  • Describe a possible architecture of an ensemble chatbot.

Course Grading

To pass this course, you will need to:

  1. Take an exam (a written test covering important lecture content).
  2. Do lab homeworks (various dialogue system implementation tasks).

Exam test

  • There will be a written exam test at the end of the semester.
  • There will be 10 questions, we expect 2-3 sentences as an answer, with a maximum of 10 points per question.
  • To pass the course, you need to get at least 50% of the total points from the test.
  • We plan to publish a list of possible questions beforehand.

In case the pandemic does not get better by the exam period, there will be a remote alternative for the exam (an essay with a discussion).

Homework assignments

  • There will be 12 homework assignments, introduced every week, starting on the 2nd week of the semester.
  • You will submit the homework assignments into a private Gitlab repository (where we will be given access).
  • For each assignment, you will get a maximum of 10 points.
  • All assignments will have a fixed deadline.
  • If you submit the assignment after the deadline, you will get:
    • up to 50% of the maximum points if it is less than 2 weeks after the deadline;
    • 0 points if it is more than 2 weeks after the deadline.
  • Once we check the submitted assignments, you will see the points you got and the comments from us as comments on your merge requests on Gitlab.
  • You need to get at least 50% of the total assignments points to pass the course.
    • You can take the exam even if you don't have 50% yet (esp. due to potential delays in grading), but you'll need to get the required points eventually.

Grading

The final grade for the course will be a combination of your exam score and your homework assignment score, weighted 3:1 (i.e. the exam accounts for 75% of the grade, the assignments for 25%).

Grading:

  • Grade 1: >=87% of the weighted combination
  • Grade 2: >=74% of the weighted combination
  • Grade 3: >=60% of the weighted combination
  • An overall score of less than 60% means you did not pass.

In any case, you need >50% of points from the test and >50% of points from the homeworks to pass. If you get less than 50% from either, even if you get more than 60% overall, you will not pass.

No cheating

  • Cheating is strictly prohibited and any student found cheating will be punished. The punishment can involve failing the whole course, or, in grave cases, being expelled from the faculty.
  • Discussing homework assignments with your classmates is OK. Sharing code is not OK (unless explicitly allowed); by default, you must complete the assignments yourself.
  • All students involved in cheating will be punished. E.g. if you share your assignment with a friend, both you and your friend will be punished.

Recommended Reading

You should pass the course just by following the lectures, but here are some hints on further reading. There's nothing ideal on the topic as this is a very active research area, but some of these should give you a broader overview.

Basic (good but very brief, available online):

More detailed (very good, available as e-book from our library):

Further reading:

  • Janarthanam: Hands-On Chatbots and Conversational UI Development. Packt 2017.
    • practical guide on developing dialogue systems for current platforms, virtually no theory
  • Gao et al.: Neural Approaches to Conversational AI. arXiv:1809.08267
    • an advanced, good overview of the latest neural approaches in dialogue systems
  • McTear et al.: The Conversational Interface: Talking to Smart Devices. Springer 2016.
    • practical, for current platforms, more advanced and more theory than Janarthanam
  • Jokinen & McTear: Spoken dialogue systems. Morgan & Claypool 2010.
    • good but slightly outdated, some systems very specific to particular research projects
  • Rieser & Lemon: Reinforcement learning for adaptive dialogue systems. Springer 2011.
    • advanced, slightly outdated, project-specific
  • Lemon & Pietquin: Data-Driven Methods for Adaptive Spoken Dialogue Systems. Springer 2012.
    • ditto
  • Skantze: Error Handling in Spoken Dialogue Systems. PhD Thesis 2007, Chap. 2.
    • good introduction into dialogue systems in general, albeit slightly dated
  • McTear: Spoken Dialogue Technology. Springer 2004.
    • good but dated
  • Psutka et al.: Mluvíme s počítačem česky. Academia 2006.
    • virtually the only book in Czech, good for ASR but dated, not a lot about other parts of dialogue systems