Logo ÚFAL Logo MFF

Institute of Formal and Applied Linguistics


Charles University, Czech Republic
Faculty of Mathematics and Physics

ÚFAL
NPFL114

Deep Learning – Summer 2020/21

In recent years, deep neural networks have been used to solve complex machine-learning problems. They have achieved significant state-of-the-art results in many areas.

The goal of the course is to introduce deep neural networks, from the basics to the latest advances. The course will focus both on theory as well as on practical aspects (students will implement and train several deep neural networks capable of achieving state-of-the-art results, for example in image classification, object detection, lemmatization, speech recognition or 3D object recognition). No previous knowledge of artificial neural networks is required, but basic understanding of machine learning is advisable.

About

SIS code: NPFL114
Semester: summer
E-credits: 7
Examination: 3/2 C+Ex
Guarantor: Milan Straka

Timespace Coordinates

  • lectures: Czech lecture is held on Monday 9:50 in S5, English lecture on Monday 13:10 in S5; first lecture is on Mar 1
  • practicals: there are two parallel practicals, a Czech one on Tuesday 10:40 in S9, and an English ones on Tuesday 9:00 in S9; first practicals are on Mar 2
  • consultations: voluntary consultations regarding the assignments or other issues are held regularly on Tuesday 14:00 in SU1

All lectures and practicals will be recorded and available on this website.

Given the pandemic situation, all lectures and practicals are currently held online.

Lectures

1. Introduction to Deep Learning Slides PDF Slides

The lecture content, including references to study materials. The main study material is the Deep Learning Book by Ian Goodfellow, Yoshua Bengio and Aaron Courville, (referred to as DLB).

References to study materials cover all theory required at the exam, and sometimes even more – the references in italics cover topics not required for the exam.

1. Introduction to Deep Learning

 Mar 01 Slides PDF Slides

Requirements

To pass the practicals, you need to obtain at least 80 points, excluding the bonus points. Note that up to 40 points above 80 (including the bonus points) will be transfered to the exam. In total, assignments for at least 120 points (not including the bonus points) will be available.

Environment

The tasks are evaluated automatically using the ReCodEx Code Examiner.

The evaluation is performed using Python 3.8, TensorFlow 2.4.1, TensorFlow Addons 0.12.1, TensorFlow Probability 0.12.1, TensorFlow Hub 0.11.0 and OpenAI Gym 0.18.0. You should install the exact version of these packages yourselves.

Teamwork

Solving assignments in teams of size 2 or 3 is encouraged, but everyone has to participate (it is forbidden not to work on an assignment and then submit a solution created by other team members). All members of the team must submit in ReCodEx individually, but can have exactly the same sources/models/results. Each such solution must explicitly list all members of the team to allow plagiarism detection using this template.

In the competitions, your goal is to train a model and then predict target values on the given unannotated test set.

Submitting to ReCodEx

When submitting a competition solution to ReCodEx, you can include any number of files of any kind, and either submit them individually or compess them in a .zip file. However, there should be exactly one text file with the test set annotation (.txt) and at least one Python source (.py) containing the model training and prediction. The Python sources are not executed, but must be included for inspection.

Evaluation in ReCodEx

  • For every submission, ReCodEx checks the above conditions (exactly one .txt, at least one .py) and whether the given annotations can be evaluated without error. If not, it will report a corresponding error in the logs.

  • Before the deadline, ReCodEx prints the exact achieved score, but only if it is worse than the baseline.

    If you surpass the baseline, the assignment is marked as solved in ReCodEx and you immediately get regular points for the assignment. However, ReCodEx does not print the reached score.

  • After the competition deadline, the latest submission of every user surpassing the required baseline participates in a competition. Additional bonus points are then awarded according to the ordering of the performance of the participating submissions.

  • After the competition results announcement, ReCodEx starts to show the exact performance for all the already submitted solutions and also for the solutions submitted later.

What Is Allowed

  • You can use the given annotated training data in any way.
  • You can use the given annotated development data for evaluation or hyperparameter tuning, but not for the training itself.
  • Additionally, you can use any unannotated or manually created data for training and evaluation.
  • The test set annotations must be the result of your system (so you cannot manually correct them; but your system can contain other parts than just trained models, like hand-written rules).
  • Do not use test set annotations in any way, if you somehow get access to them.
  • Unless stated otherwise, you can use any algorithm to solve the competition task at hand. The implementation should be either created by you or you should understand it fully.
  • If you utilize an already trained model, it must be trained only on the allowed training data, unless stated otherwise.

Install

  • Installing to central user packages repository

    You can install all required packages to central user packages repository using pip3 install --user --upgrade pip setuptools followed by pip3 install --user tensorflow==2.4.1 tensorflow-addons==0.12.1 tensorflow-probability==0.12.1 tensorflow-hub==0.11.0 gym==0.18.0.

  • Installing to a virtual environment

    Python supports virtual environments, which are directories containing independent sets of installed packages. You can create a virtual environment by running python3 -m venv VENV_DIR and then install the required packages with VENV_DIR/bin/pip3 install --upgrade pip setuptools followed by VENV_DIR/bin/pip3 install tensorflow==2.4.1 tensorflow-addons==0.12.1 tensorflow-probability==0.12.1 tensorflow-hub==0.11.0 gym==0.18.0.

  • Windows TensorFlow fails with ImportError: DLL load failed

    If your Windows TensorFlow fails with ImportError: DLL load failed, you are probably missing Visual C++ 2019 Redistributable.

  • Cannot start TensorBoard after installation

    If tensorboard cannot be found, make sure the directory with pip installed packages is in your PATH (that directory is either in your virtual environment if you use a virtual environment, or it should be ~/.local/bin on Linux and %UserProfile%\AppData\Roaming\Python\Python3[5-7] and %UserProfile%\AppData\Roaming\Python\Python3[5-7]\Scripts on Windows).

Requirements

To pass the practicals, you need to obtain at least 80 points, excluding the bonus points. Note that up to 40 points above 80 (including the bonus points) will be transfered to the exam. In total, assignments for at least 120 points (not including the bonus points) will be available.

To pass the exam, you need to obtain at least 60, 75 and 90 out of 100-point exam, to obtain grades 3, 2 and 1, respectively. (PhD students with binary grades require 75 points.) The exam consists of 100-point-worth questions from the list below (the questions are randomly generated, but in such a way that there is at least one question from every lecture). In addition, you can get at most 40 surplus points from the practicals and at most 10 points for community work (i.e., fixing slides or reporting issues) – but only the points you already have at the time of the exam count.

Exam Questions