The lecture content, including references to study materials. The main study material is the Deep Learning Book by Ian Goodfellow, Yoshua Bengio and Aaron Courville, (referred to as DLB).

References to study materials cover all theory required at the exam, and sometimes even more – the references in italics cover topics not required for the exam.

1. Introduction to Deep Learning

 Feb 14 Slides PDF Slides CZ Lecture EN Lecture Questions numpy_entropy pca_first mnist_layers_activations

• Random variables, probability distributions, expectation, variance, Bernoulli distribution, Categorical distribution [Sections 3.2, 3.3, 3.8, 3.9.1 and 3.9.2 of DLB]
• Self-information, entropy, cross-entropy, KL-divergence [Section 3.13 of DBL]
• Gaussian distribution [Section 3.9.3 of DLB]
• Machine Learning Basics [Section 5.1-5.1.3 of DLB]
• History of Deep Learning [Section 1.2 of DLB]
• Linear regression [Section 5.1.4 of DLB]
• Challenges Motivating Deep Learning [Section 5.11 of DLB]
• Neural network basics
  • Neural networks as graphs [Chapter 6 before Section 6.1 of DLB]
  • Output activation functions [Section 6.2.2 of DLB, excluding Section 6.2.2.4]
  • Hidden activation functions [Section 6.3 of DLB, excluding Section 6.3.3]
  • Basic network architectures [Section 6.4 of DLB, excluding Section 6.4.2]
• Universal approximation theorem

2. Training Neural Networks

 Feb 21 Slides PDF Slides CZ Lecture CZ Adam EN Lecture Questions sgd_backpropagation sgd_manual gym_cartpole

• Capacity, overfitting, underfitting, regularization [Section 5.2 of DLB]
• Hyperparameters and validation sets [Section 5.3 of DLB]
• Maximum Likelihood Estimation [Section 5.5 of DLB]
• Neural network training
  • Gradient Descent and Stochastic Gradient Descent [Sections 4.3 and 5.9 of DLB]
  • Backpropagation algorithm [Section 6.5 to 6.5.3 of DLB, especially Algorithms 6.1 and 6.2; note that Algorithms 6.5 and 6.6 are used in practice]
  • SGD algorithm [Section 8.3.1 and Algorithm 8.1 of DLB]
  • SGD with Momentum algorithm [Section 8.3.2 and Algorithm 8.2 of DLB]
  • SGD with Nestorov Momentum algorithm [Section 8.3.3 and Algorithm 8.3 of DLB]
  • Optimization algorithms with adaptive gradients
    • AdaGrad algorithm [Section 8.5.1 and Algorithm 8.4 of DLB]
    • RMSProp algorithm [Section 8.5.2 and Algorithm 8.5 of DLB]
    • Adam algorithm [Section 8.5.3 and Algorithm 8.7 of DLB]

3. Training Neural Networks II

 Feb 28 Slides PDF Slides CZ Lecture EN Lecture Questions mnist_training mnist_regularization mnist_ensemble uppercase

• Softmax with NLL (negative log likelihood) as a loss function [Section 6.2.2.3 of DLB, notably equation (6.30); plus slides 10-12]
• Regularization [Chapter 7 until Section 7.1 of DLB]
  • Early stopping [Section 7.8 of DLB, without the How early stopping acts as a regularizer part]
  • L2 and L1 regularization [Sections 7.1 and 5.6.1 of DLB; plus slides 17-18]
  • Dataset augmentation [Section 7.4 of DLB]
  • Ensembling [Section 7.11 of DLB]
  • Dropout [Section 7.12 of DLB]
  • Label smoothing [Section 7.5.1 of DLB]
• Saturating non-linearities [Section 6.3.2 and second half of Section 6.2.2.2 of DLB]
• Parameter initialization strategies [Section 8.4 of DLB]
• Gradient clipping [Section 10.11.1 of DLB]

4. Convolutional Neural Networks

 Mar 07 Slides PDF Slides CZ Lecture EN Lecture Questions mnist_cnn image_augmentation tf_dataset mnist_multiple cifar_competition

• Introduction to convolutional networks [Chapter 9 and Sections 9.1-9.3 of DLB]
• Convolution as operation on 4D tensors [Section 9.5 of DLB, notably Equations (9.7) and (9.8)]
• Max pooling and average pooling [Section 9.3 of DLB]
• Stride and Padding schemes [Section 9.5 of DLB]
• AlexNet [ImageNet Classification with Deep Convolutional Neural Networks]
• VGG [Very Deep Convolutional Networks for Large-Scale Image Recognition]
• GoogLeNet (aka Inception) [Going Deeper with Convolutions]
• Batch normalization [Section 8.7.1 of DLB, optionally the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift]
• Inception v2 and v3 [Rethinking the Inception Architecture for Computer Vision]
• ResNet [Deep Residual Learning for Image Recognition]

5. Convolutional Neural Networks II

 Mar 14 Slides PDF Slides CZ Lecture EN Lecture Questions cnn_manual cags_classification cags_segmentation

• Residual CNN Networks
  • ResNet [Deep Residual Learning for Image Recognition]
  • WideNet [Wide Residual Network]
  • DenseNet [Densely Connected Convolutional Networks]
  • PyramidNet [Deep Pyramidal Residual Networks]
  • ResNeXt [Aggregated Residual Transformations for Deep Neural Networks]
• Regularizing CNN Networks
  • DropLayer [Deep Networks with Stochastic Depth]
  • CutOut [Improved Regularization of Convolutional Neural Networks with Cutout]
  • DropBlock [DropBlock: A regularization method for convolutional networks]
• SENet [Squeeze-and-Excitation Networks]
• EfficientNet [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks]
• EfficientNetV2 [EfficientNetV2: Smaller Models and Faster Training]
• Transposed convolution
• U-Net [U-Net: Convolutional Networks for Biomedical Image Segmentation]

6. Object Detection

 Mar 21 Slides PDF Slides CZ Lecture EN Lecture Questions bboxes_utils svhn_competition

• Fast R-CNN [Fast R-CNN]
• Proposing RoIs using Faster R-CNN [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks]
• Mask R-CNN [Mask R-CNN]
• Feature Pyramid Networks [Feature Pyramid Networks for Object Detection]
• Focal Loss, RetinaNet [Focal Loss for Dense Object Detection]
• EfficientDet [EfficientDet: Scalable and Efficient Object Detection]
• Group Normalization [Group Normalization]

7. Recurrent Neural Networks

 Mar 28 Slides PDF Slides CZ Lecture EN Lecture Questions sequence_classification tagger_we tagger_cle tagger_competition

• Sequence modelling using Recurrent Neural Networks (RNN) [Chapter 10 until Section 10.2.1 (excluding) of DLB]
• The challenge of long-term dependencies [Section 10.7 of DLB]
• Long Short-Term Memory (LSTM) [Section 10.10.1 of DLB, Sepp Hochreiter, Jürgen Schmidhuber (1997): Long short-term memory, Felix A. Gers, Jürgen Schmidhuber, Fred Cummins (2000): Learning to Forget: Continual Prediction with LSTM]
• Gated Recurrent Unit (GRU) [Section 10.10.2 of DLB, Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, Yoshua Bengio: Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation]
• Highway Networks [Training Very Deep Networks]
• RNN Regularization
  • Variational Dropout [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks]
  • Layer Normalization [Layer Normalization]
• Bidirectional RNN [Section 10.3 of DLB]
• Word Embeddings [Section 14.2.4 of DLB]
• Character-level embeddings using Recurrent neural networks [C2W model from Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation]
• Character-level embeddings using Convolutional neural networks [CharCNN from Character-Aware Neural Language Models]

8. CRF, CTC, Word2Vec

 Apr 04 Slides PDF Slides CZ Lecture EN Lecture Questions tensorboard_projector tagger_crf tagger_crf_manual speech_recognition

• Conditional Random Fields (CRF) loss [Sections 3.4.2 and A.7 of Natural Language Processing (Almost) from Scratch]
• Connectionist Temporal Classification (CTC) loss [Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks]
• Word2vec word embeddings, notably the CBOW and Skip-gram architectures [Efficient Estimation of Word Representations in Vector Space]
  • Hierarchical softmax [Section 12.4.3.2 of DLB or Distributed Representations of Words and Phrases and their Compositionality]
  • Negative sampling Distributed Representations of Words and Phrases and their Compositionality]
• Character-level embeddings using character n-grams [Described simultaneously in several papers as Charagram (Charagram: Embedding Words and Sentences via Character n-grams), Subword Information (Enriching Word Vectors with Subword Information or SubGram (SubGram: Extending Skip-Gram Word Representation with Substrings)]

9. Seq2seq, NMT, Transformer

 Apr 11 Slides PDF Slides CZ Lecture EN Lecture Questions lemmatizer_noattn lemmatizer_attn lemmatizer_competition

• Neural Machine Translation using Encoder-Decoder or Sequence-to-Sequence architecture [Section 12.5.4 of DLB, Ilya Sutskever, Oriol Vinyals, Quoc V. Le: Sequence to Sequence Learning with Neural Networks and Kyunghyun Cho et al.: Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation]
• Using Attention mechanism in Neural Machine Translation [Section 12.4.5.1 of DLB, Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio: Neural Machine Translation by Jointly Learning to Align and Translate]
• Translating Subword Units [Rico Sennrich, Barry Haddow, Alexandra Birch: Neural Machine Translation of Rare Words with Subword Units]
• Google NMT [Yonghui Wu et al.: Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation]
• Transformer architecture [Attention Is All You Need]

10. Easter Monday

 Apr 18 3d_recognition homr_competition

11. Transformer, BERT

 Apr 25 Slides PDF Slides CZ Lecture EN Lecture Questions tagger_transformer sentiment_analysis reading_comprehension

• Transformer architecture [Attention Is All You Need]
• BERT [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding]
• RoBERTa [RoBERTa: A Robustly Optimized BERT Pretraining Approach]

12. Deep Generative Models

 May 02 Slides PDF Slides CZ Lecture EN Lecture Questions vae gan dcgan crac2022

• Autoencoders (undercomplete, sparse, denoising) [Chapter 14, Sections 14-14.2.3 of DLB]
• Deep Generative Models using Differentiable Generator Nets [Section 20.10.2 of DLB]
• Variational Autoencoders [Section 20.10.3 plus Reparametrization trick from Section 20.9 (but not Section 20.9.1) of DLB, Auto-Encoding Variational Bayes]
  • VQ-VAE [Neural Discrete Representation Learning]
  • VQ-VAE-2 [Generating Diverse High-Fidelity Images with VQ-VAE-2]
  • VQGAN [Taming Transformers for High-Resolution Image Synthesis]
  • DALL-E [Zero-Shot Text-to-Image Generation]
• Generative Adversarial Networks
  • GAN [Section 20.10.4 of DLB, Generative Adversarial Networks]
  • CGAN [Conditional Generative Adversarial Nets]
  • DCGAN [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks]
  • WGAN [Wasserstein GAN]
  • BigGAN [Large Scale Gan Training for High Fidelity Natural Image Synthesis]

13. Introduction to Deep Reinforcement Learning

 May 09 Slides PDF Slides CZ Lecture EN Lecture Questions monte_carlo reinforce reinforce_baseline reinforce_pixels

Study material for Reinforcement Learning is the Reinforcement Learning: An Introduction; second edition by Richard S. Sutton and Andrew G. Barto (reffered to as RLB), available online.

• Multi-armed bandits [Sections 2-2.4 of RLB]
• Markov Decision Process [Sections 3-3.3 of RLB]
• Policies and Value Functions [Sections 3.5 of RLB]
• Monte Carlo Methods [Sections 5-5.4 of RLB]
• Policy Gradient Methods [Sections 13-13.1 of RLB]
• Policy Gradient Theorem [Section 13.2 of RLB]
• REINFORCE algorithm [Section 13.3 of RLB]
• REINFORCE with baseline algorithm [Section 13.4 of RLB]

14. NASNet, Speech Synthesis, External Memory Networks

 May 16 Slides PDF Slides CZ Lecture EN Lecture Questions learning_to_learn

• NasNet [Learning Transferable Architectures for Scalable Image Recognition]
• EfficientNet [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks]
• WaveNet [WaveNet: A Generative Model for Raw Audio]
• Parallel WaveNet [Parallel WaveNet: Fast High-Fidelity Speech Synthesis]
• Full speech synthesis pipeline Tacotron 2 [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions]
• Neural Turing Machine [Neural Turing Machines]
• Differenciable Neural Computer [Hybrid computing using a neural network with dynamic external memory]
• Memory Augmented Neural Networks [One-shot learning with Memory-Augmented Neural Networks]

Requirements

To pass the practicals, you need to obtain at least 80 points, excluding the bonus points. Note that all surplus points (both bonus and non-bonus) will be transfered to the exam. In total, assignments for at least 120 points (not including the bonus points) will be available, and if you solve all the assignments (any non-zero amount of points counts as solved), you automatically pass the exam with grade 1.

Environment

The tasks are evaluated automatically using the ReCodEx Code Examiner.

The evaluation is performed using Python 3.9, TensorFlow 2.8.0, TensorFlow Addons 0.16, TensorFlow Probability 0.12.1, TensorFlow Hub 0.11.0, and OpenAI Gym 0.20.0. You should install the exact version of these packages yourselves.

Teamwork

Solving assignments in teams (of size at most 3) is encouraged, but everyone has to participate (it is forbidden not to work on an assignment and then submit a solution created by other team members). All members of the team must submit in ReCodEx individually, but can have exactly the same sources/models/results. Each such solution must explicitly list all members of the team to allow plagiarism detection using this template.

No Cheating

Cheating is strictly prohibited and any student found cheating will be punished. The punishment can involve failing the whole course, or, in grave cases, being expelled from the faculty. While discussing assignments with any classmate is fine, each team must complete the assignments themselves, without using code they did not write (unless explicitly allowed). Of course, inside a team you are expected to share code and submit identical solutions.

numpy_entropy

 Deadline: Feb 28, 7:59 a.m.  3 points  Examples  Tests

The goal of this exercise is to familiarize with Python, NumPy and ReCodEx submission system. Start with the numpy_entropy.py.

Load a file specified in args.data_path, whose lines consist of data points of our dataset, and load a file specified in args.model_path, which describes a model probability distribution, with each line being a tab-separated pair of (data point, probability).

Then compute the following quantities using NumPy, and print them each on a separate line rounded on two decimal places (or inf for positive infinity, which happens when an element of data distribution has zero probability under the model distribution):

• entropy H(data distribution)
• cross-entropy H(data distribution, model distribution)
• KL-divergence D_KL(data distribution, model distribution)

Use natural logarithms to compute the entropies and the divergence.

Entropy: 0.96 nats
Crossentropy: 1.07 nats
KL divergence: 0.11 nats

Entropy: 0.96 nats
Crossentropy: inf nats
KL divergence: inf nats

Entropy: 4.15 nats
Crossentropy: 4.23 nats
KL divergence: 0.08 nats

Entropy: 4.99 nats
Crossentropy: 5.03 nats
KL divergence: 0.04 nats

Entropy: 0.96 nats
Crossentropy: 1.07 nats
KL divergence: 0.11 nats

Entropy: 0.96 nats
Crossentropy: inf nats
KL divergence: inf nats

pca_first

 Deadline: Feb 28, 7:59 a.m.  2 points  Tests

The goal of this exercise is to familiarize with TensorFlow tf.Tensors, shapes and basic tensor manipulation methods. Start with the pca_first.py (and you will also need the mnist.py module).

In this assignment, you will compute the covariance matrix of several examples from the MNIST dataset, compute the first principal component and quantify the explained variance of it.

It is fine if you are not familiar with terms like covariance matrix or principal component – the template contains a detailed description of what you have to do.

Finally, it is a good idea to read the TensorFlow guide about tf.Tensors.

Total variance: 53.12
Explained variance: 9.64%

Total variance: 53.05
Explained variance: 9.89%

Total variance: 52.74
Explained variance: 9.71%

mnist_layers_activations

 Deadline: Mar 07, 7:59 a.m.  2 points  Examples  Tests

Before solving the assignment, start by playing with example_keras_tensorboard.py, in order to familiarize with TensorFlow and TensorBoard. Run it, and when it finishes, run TensorBoard using tensorboard --logdir logs. Then open http://localhost:6006 in a browser and explore the active tabs.

Your goal is to modify the mnist_layers_activations.py template and implement the following:

• A number of hidden layers (including zero) can be specified on the command line using parameter hidden_layers.
• Activation function of these hidden layers can be also specified as a command line parameter activation, with supported values of none, relu, tanh and sigmoid.
• Print the final accuracy on the test set.

Epoch  1/10 loss: 0.5383 - accuracy: 0.8613 - val_loss: 0.2755 - val_accuracy: 0.9308
Epoch  5/10 loss: 0.2783 - accuracy: 0.9220 - val_loss: 0.2202 - val_accuracy: 0.9430
Epoch 10/10 loss: 0.2595 - accuracy: 0.9273 - val_loss: 0.2146 - val_accuracy: 0.9434
loss: 0.2637 - accuracy: 0.9259

Epoch  1/10 loss: 0.3828 - accuracy: 0.8914 - val_loss: 0.2438 - val_accuracy: 0.9350
Epoch  5/10 loss: 0.2754 - accuracy: 0.9222 - val_loss: 0.2341 - val_accuracy: 0.9370
Epoch 10/10 loss: 0.2640 - accuracy: 0.9260 - val_loss: 0.2318 - val_accuracy: 0.9400
loss: 0.2795 - accuracy: 0.9241

Epoch  1/10 loss: 0.3195 - accuracy: 0.9109 - val_loss: 0.1459 - val_accuracy: 0.9612
Epoch  5/10 loss: 0.0629 - accuracy: 0.9811 - val_loss: 0.0820 - val_accuracy: 0.9776
Epoch 10/10 loss: 0.0237 - accuracy: 0.9937 - val_loss: 0.0801 - val_accuracy: 0.9776
loss: 0.0829 - accuracy: 0.9769

Epoch  1/10 loss: 0.3414 - accuracy: 0.9039 - val_loss: 0.1668 - val_accuracy: 0.9570
Epoch  5/10 loss: 0.0750 - accuracy: 0.9783 - val_loss: 0.0813 - val_accuracy: 0.9774
Epoch 10/10 loss: 0.0268 - accuracy: 0.9937 - val_loss: 0.0788 - val_accuracy: 0.9744
loss: 0.0822 - accuracy: 0.9751

Epoch  1/10 loss: 0.4969 - accuracy: 0.8751 - val_loss: 0.2150 - val_accuracy: 0.9400
Epoch  5/10 loss: 0.1222 - accuracy: 0.9649 - val_loss: 0.1041 - val_accuracy: 0.9718
Epoch 10/10 loss: 0.0594 - accuracy: 0.9842 - val_loss: 0.0805 - val_accuracy: 0.9772
loss: 0.0862 - accuracy: 0.9741

Epoch  1/10 loss: 0.2753 - accuracy: 0.9173 - val_loss: 0.1128 - val_accuracy: 0.9672
Epoch  5/10 loss: 0.0489 - accuracy: 0.9843 - val_loss: 0.0878 - val_accuracy: 0.9778
Epoch 10/10 loss: 0.0226 - accuracy: 0.9923 - val_loss: 0.0892 - val_accuracy: 0.9788
loss: 0.0770 - accuracy: 0.9793

Epoch  1/10 loss: 0.3598 - accuracy: 0.8881 - val_loss: 0.1457 - val_accuracy: 0.9586
Epoch  5/10 loss: 0.0822 - accuracy: 0.9775 - val_loss: 0.1135 - val_accuracy: 0.9766
Epoch 10/10 loss: 0.0525 - accuracy: 0.9859 - val_loss: 0.1108 - val_accuracy: 0.9768
loss: 0.1342 - accuracy: 0.9715

Epoch  1/10 loss: 2.2830 - accuracy: 0.1088 - val_loss: 1.9021 - val_accuracy: 0.2120
Epoch  5/10 loss: 0.9505 - accuracy: 0.6286 - val_loss: 0.7622 - val_accuracy: 0.7214
Epoch 10/10 loss: 0.4468 - accuracy: 0.8919 - val_loss: 0.3524 - val_accuracy: 0.9212
loss: 0.4232 - accuracy: 0.8993

Epoch  1/1 loss: 0.5383 - accuracy: 0.8613 - val_loss: 0.2755 - val_accuracy: 0.9308
loss: 0.3304 - accuracy: 0.9110

Epoch 1/1 loss: 0.3828 - accuracy: 0.8914 - val_loss: 0.2438 - val_accuracy: 0.9350
loss: 0.2956 - accuracy: 0.9198

Epoch 1/1 loss: 0.3195 - accuracy: 0.9109 - val_loss: 0.1459 - val_accuracy: 0.9612
loss: 0.1738 - accuracy: 0.9517

Epoch 1/1 loss: 0.3414 - accuracy: 0.9039 - val_loss: 0.1668 - val_accuracy: 0.9570
loss: 0.2039 - accuracy: 0.9422

Epoch 1/1 loss: 0.4969 - accuracy: 0.8751 - val_loss: 0.2150 - val_accuracy: 0.9400
loss: 0.2627 - accuracy: 0.9268

Epoch 1/1 loss: 0.2753 - accuracy: 0.9173 - val_loss: 0.1128 - val_accuracy: 0.9672
loss: 0.1309 - accuracy: 0.9601

Epoch 1/1 loss: 0.3598 - accuracy: 0.8881 - val_loss: 0.1457 - val_accuracy: 0.9586
loss: 0.1806 - accuracy: 0.9474

Epoch 1/1 loss: 2.2830 - accuracy: 0.1088 - val_loss: 1.9021 - val_accuracy: 0.2120
loss: 1.9469 - accuracy: 0.2065

sgd_backpropagation

 Deadline: Mar 07, 7:59 a.m.  3 points  Examples  Tests

In this exercise you will learn how to compute gradients using the so-called automatic differentiation, which is implemented by an automated backpropagation algorithm in TensorFlow. You will then perform training by running manually implemented minibatch stochastic gradient descent.

Starting with the sgd_backpropagation.py template, you should:

• implement a neural network with a single tanh hidden layer and categorical output layer;
• compute the crossentropy loss;
• use tf.GradientTape to automatically compute the gradient of the loss with respect to all variables;
• perform the SGD update.

Dev accuracy after epoch 1 is 92.84
Dev accuracy after epoch 2 is 93.86
Dev accuracy after epoch 3 is 94.64
Dev accuracy after epoch 4 is 95.24
Dev accuracy after epoch 5 is 95.26
Dev accuracy after epoch 6 is 95.66
Dev accuracy after epoch 7 is 95.58
Dev accuracy after epoch 8 is 95.86
Dev accuracy after epoch 9 is 96.18
Dev accuracy after epoch 10 is 96.08
Test accuracy after epoch 10 is 95.53

Dev accuracy after epoch 1 is 93.66
Dev accuracy after epoch 2 is 95.00
Dev accuracy after epoch 3 is 95.72
Dev accuracy after epoch 4 is 95.80
Dev accuracy after epoch 5 is 96.34
Dev accuracy after epoch 6 is 96.16
Dev accuracy after epoch 7 is 96.42
Dev accuracy after epoch 8 is 96.36
Dev accuracy after epoch 9 is 96.60
Dev accuracy after epoch 10 is 96.58
Test accuracy after epoch 10 is 96.18

Dev accuracy after epoch 1 is 92.84
Dev accuracy after epoch 2 is 93.86
Test accuracy after epoch 2 is 93.21

Dev accuracy after epoch 1 is 93.66
Dev accuracy after epoch 2 is 95.00
Test accuracy after epoch 2 is 93.93

sgd_manual

 Deadline: Mar 07, 7:59 a.m.  2 points  Examples  Tests

The goal in this exercise is to extend your solution to the sgd_backpropagation assignment by manually computing the gradient.

While in this assignment we compute the gradient manually, we will nearly always use the automatic differentiation. Therefore, the assignment is more of a mathematical exercise than a real-world application. Furthermore, we will compute the derivatives together on the Feb 28 practicals.

Start with the sgd_manual.py template, which is based on sgd_backpropagation.py one. Be aware that these templates generates each a different output file.

In order to check that you do not use automatic differentiation, ReCodEx checks that you do not use tf.GradientTape in your solution.

Dev accuracy after epoch 1 is 92.84
Dev accuracy after epoch 2 is 93.86
Dev accuracy after epoch 3 is 94.64
Dev accuracy after epoch 4 is 95.24
Dev accuracy after epoch 5 is 95.26
Dev accuracy after epoch 6 is 95.66
Dev accuracy after epoch 7 is 95.58
Dev accuracy after epoch 8 is 95.86
Dev accuracy after epoch 9 is 96.18
Dev accuracy after epoch 10 is 96.08
Test accuracy after epoch 10 is 95.53

Dev accuracy after epoch 1 is 93.66
Dev accuracy after epoch 2 is 95.00
Dev accuracy after epoch 3 is 95.72
Dev accuracy after epoch 4 is 95.80
Dev accuracy after epoch 5 is 96.34
Dev accuracy after epoch 6 is 96.16
Dev accuracy after epoch 7 is 96.42
Dev accuracy after epoch 8 is 96.36
Dev accuracy after epoch 9 is 96.60
Dev accuracy after epoch 10 is 96.58
Test accuracy after epoch 10 is 96.18

Dev accuracy after epoch 1 is 92.84
Dev accuracy after epoch 2 is 93.86
Test accuracy after epoch 2 is 93.21

Dev accuracy after epoch 1 is 93.66
Dev accuracy after epoch 2 is 95.00
Test accuracy after epoch 2 is 93.93

gym_cartpole

 Deadline: Mar 07, 7:59 a.m.  3 points

Solve the CartPole-v1 environment from the Gym library, utilizing only provided supervised training data. The data is available in gym_cartpole_data.txt file, each line containing one observation (four space separated floats) and a corresponding action (the last space separated integer). Start with the gym_cartpole.py.

The solution to this task should be a model which passes evaluation on random inputs. This evaluation can be performed by running the gym_cartpole.py with --evaluate argument (optionally rendering if --render option is provided), or directly calling the evaluate_model method. In order to pass, you must achieve an average reward of at least 475 on 100 episodes. Your model should have either one or two outputs (i.e., using either sigmoid or softmax output function).

When designing the model, you should consider that the size of the training data is very small and the data is quite noisy.

When submitting to ReCodEx, do not forget to also submit the trained model.

mnist_training

 Deadline: Mar 14, 7:59 a.m.  2 points  Examples  Tests

This exercise should teach you using different optimizers, learning rates, and learning rate decays. Your goal is to modify the mnist_training.py template and implement the following:

• Using specified optimizer (either SGD or Adam).
• Optionally using momentum for the SGD optimizer.
• Using specified learning rate for the optimizer.
• Optionally use a given learning rate schedule. The schedule can be either exponential or linear (with degree 1, so linear time decay). Additionally, the final learning rate is given and the decay should gradually decrease the learning rate to reach the final learning rate just after the training.

Epoch  1/10 loss: 0.7989 - accuracy: 0.8098 - val_loss: 0.3662 - val_accuracy: 0.9146
Epoch  2/10 loss: 0.3991 - accuracy: 0.8919 - val_loss: 0.2848 - val_accuracy: 0.9258
Epoch  3/10 loss: 0.3382 - accuracy: 0.9054 - val_loss: 0.2496 - val_accuracy: 0.9350
Epoch  4/10 loss: 0.3049 - accuracy: 0.9144 - val_loss: 0.2292 - val_accuracy: 0.9390
Epoch  5/10 loss: 0.2811 - accuracy: 0.9216 - val_loss: 0.2131 - val_accuracy: 0.9426
Epoch  6/10 loss: 0.2623 - accuracy: 0.9268 - val_loss: 0.2003 - val_accuracy: 0.9464
Epoch  7/10 loss: 0.2461 - accuracy: 0.9315 - val_loss: 0.1882 - val_accuracy: 0.9500
Epoch  8/10 loss: 0.2323 - accuracy: 0.9353 - val_loss: 0.1821 - val_accuracy: 0.9522
Epoch  9/10 loss: 0.2204 - accuracy: 0.9386 - val_loss: 0.1715 - val_accuracy: 0.9560
Epoch 10/10 loss: 0.2094 - accuracy: 0.9413 - val_loss: 0.1650 - val_accuracy: 0.9572 - val_test_loss: 0.1978 - val_test_accuracy: 0.9441

Epoch  1/10 loss: 0.3617 - accuracy: 0.8973 - val_loss: 0.1684 - val_accuracy: 0.9560
Epoch  2/10 loss: 0.1803 - accuracy: 0.9490 - val_loss: 0.1274 - val_accuracy: 0.9644
Epoch  3/10 loss: 0.1319 - accuracy: 0.9625 - val_loss: 0.1051 - val_accuracy: 0.9706
Epoch  4/10 loss: 0.1048 - accuracy: 0.9709 - val_loss: 0.0922 - val_accuracy: 0.9746
Epoch  5/10 loss: 0.0864 - accuracy: 0.9756 - val_loss: 0.0844 - val_accuracy: 0.9782
Epoch  6/10 loss: 0.0731 - accuracy: 0.9794 - val_loss: 0.0791 - val_accuracy: 0.9784
Epoch  7/10 loss: 0.0633 - accuracy: 0.9825 - val_loss: 0.0738 - val_accuracy: 0.9818
Epoch  8/10 loss: 0.0550 - accuracy: 0.9848 - val_loss: 0.0746 - val_accuracy: 0.9796
Epoch  9/10 loss: 0.0485 - accuracy: 0.9866 - val_loss: 0.0758 - val_accuracy: 0.9796
Epoch 10/10 loss: 0.0429 - accuracy: 0.9888 - val_loss: 0.0704 - val_accuracy: 0.9806 - val_test_loss: 0.0677 - val_test_accuracy: 0.9789

Epoch  1/10 loss: 0.3502 - accuracy: 0.9021 - val_loss: 0.1679 - val_accuracy: 0.9576
Epoch  2/10 loss: 0.1784 - accuracy: 0.9492 - val_loss: 0.1265 - val_accuracy: 0.9646
Epoch  3/10 loss: 0.1303 - accuracy: 0.9629 - val_loss: 0.0994 - val_accuracy: 0.9724
Epoch  4/10 loss: 0.1033 - accuracy: 0.9714 - val_loss: 0.0891 - val_accuracy: 0.9754
Epoch  5/10 loss: 0.0848 - accuracy: 0.9757 - val_loss: 0.0847 - val_accuracy: 0.9776
Epoch  6/10 loss: 0.0721 - accuracy: 0.9794 - val_loss: 0.0802 - val_accuracy: 0.9778
Epoch  7/10 loss: 0.0620 - accuracy: 0.9829 - val_loss: 0.0724 - val_accuracy: 0.9818
Epoch  8/10 loss: 0.0541 - accuracy: 0.9853 - val_loss: 0.0724 - val_accuracy: 0.9808
Epoch  9/10 loss: 0.0480 - accuracy: 0.9868 - val_loss: 0.0745 - val_accuracy: 0.9796
Epoch 10/10 loss: 0.0421 - accuracy: 0.9890 - val_loss: 0.0665 - val_accuracy: 0.9824 - val_test_loss: 0.0658 - val_test_accuracy: 0.9800

Epoch  1/10 loss: 0.2699 - accuracy: 0.9231 - val_loss: 0.1166 - val_accuracy: 0.9686
Epoch  2/10 loss: 0.1139 - accuracy: 0.9665 - val_loss: 0.0921 - val_accuracy: 0.9748
Epoch  3/10 loss: 0.0769 - accuracy: 0.9773 - val_loss: 0.0831 - val_accuracy: 0.9774
Epoch  4/10 loss: 0.0561 - accuracy: 0.9833 - val_loss: 0.0758 - val_accuracy: 0.9780
Epoch  5/10 loss: 0.0425 - accuracy: 0.9872 - val_loss: 0.0732 - val_accuracy: 0.9800
Epoch  6/10 loss: 0.0312 - accuracy: 0.9910 - val_loss: 0.0838 - val_accuracy: 0.9804
Epoch  7/10 loss: 0.0268 - accuracy: 0.9918 - val_loss: 0.0776 - val_accuracy: 0.9812
Epoch  8/10 loss: 0.0194 - accuracy: 0.9941 - val_loss: 0.0739 - val_accuracy: 0.9818
Epoch  9/10 loss: 0.0154 - accuracy: 0.9957 - val_loss: 0.0796 - val_accuracy: 0.9816
Epoch 10/10 loss: 0.0128 - accuracy: 0.9962 - val_loss: 0.0828 - val_accuracy: 0.9778 - val_test_loss: 0.0762 - val_test_accuracy: 0.9786

Epoch  1/10 loss: 0.2354 - accuracy: 0.9290 - val_loss: 0.1425 - val_accuracy: 0.9576
Epoch  2/10 loss: 0.1450 - accuracy: 0.9590 - val_loss: 0.1551 - val_accuracy: 0.9584
Epoch  3/10 loss: 0.1240 - accuracy: 0.9647 - val_loss: 0.1432 - val_accuracy: 0.9682
Epoch  4/10 loss: 0.1161 - accuracy: 0.9697 - val_loss: 0.1400 - val_accuracy: 0.9626
Epoch  5/10 loss: 0.1081 - accuracy: 0.9718 - val_loss: 0.1329 - val_accuracy: 0.9688
Epoch  6/10 loss: 0.0908 - accuracy: 0.9771 - val_loss: 0.1663 - val_accuracy: 0.9688
Epoch  7/10 loss: 0.0936 - accuracy: 0.9767 - val_loss: 0.1644 - val_accuracy: 0.9670
Epoch  8/10 loss: 0.0872 - accuracy: 0.9784 - val_loss: 0.1550 - val_accuracy: 0.9686
Epoch  9/10 loss: 0.0817 - accuracy: 0.9798 - val_loss: 0.2147 - val_accuracy: 0.9642
Epoch 10/10 loss: 0.0779 - accuracy: 0.9807 - val_loss: 0.1981 - val_accuracy: 0.9718 - val_test_loss: 0.1910 - val_test_accuracy: 0.9726

Epoch  1/10 loss: 0.2235 - accuracy: 0.9319 - val_loss: 0.1255 - val_accuracy: 0.9652
Epoch  2/10 loss: 0.1145 - accuracy: 0.9659 - val_loss: 0.1273 - val_accuracy: 0.9666
Epoch  3/10 loss: 0.0761 - accuracy: 0.9762 - val_loss: 0.0905 - val_accuracy: 0.9778
Epoch  4/10 loss: 0.0514 - accuracy: 0.9842 - val_loss: 0.1031 - val_accuracy: 0.9736
Epoch  5/10 loss: 0.0323 - accuracy: 0.9893 - val_loss: 0.1046 - val_accuracy: 0.9772
Epoch  6/10 loss: 0.0189 - accuracy: 0.9938 - val_loss: 0.1010 - val_accuracy: 0.9794
Epoch  7/10 loss: 0.0127 - accuracy: 0.9959 - val_loss: 0.1019 - val_accuracy: 0.9790
Epoch  8/10 loss: 0.0073 - accuracy: 0.9977 - val_loss: 0.1066 - val_accuracy: 0.9792
Epoch  9/10 loss: 0.0039 - accuracy: 0.9990 - val_loss: 0.1049 - val_accuracy: 0.9806
Epoch 10/10 loss: 0.0021 - accuracy: 0.9997 - val_loss: 0.1057 - val_accuracy: 0.9798 - val_test_loss: 0.0868 - val_test_accuracy: 0.9809

Epoch  1/10 loss: 0.2292 - accuracy: 0.9303 - val_loss: 0.1176 - val_accuracy: 0.9634
Epoch  2/10 loss: 0.1291 - accuracy: 0.9621 - val_loss: 0.1193 - val_accuracy: 0.9658
Epoch  3/10 loss: 0.0973 - accuracy: 0.9719 - val_loss: 0.1094 - val_accuracy: 0.9712
Epoch  4/10 loss: 0.0694 - accuracy: 0.9796 - val_loss: 0.1408 - val_accuracy: 0.9656
Epoch  5/10 loss: 0.0523 - accuracy: 0.9840 - val_loss: 0.1234 - val_accuracy: 0.9704
Epoch  6/10 loss: 0.0346 - accuracy: 0.9889 - val_loss: 0.1381 - val_accuracy: 0.9740
Epoch  7/10 loss: 0.0249 - accuracy: 0.9922 - val_loss: 0.1105 - val_accuracy: 0.9776
Epoch  8/10 loss: 0.0105 - accuracy: 0.9968 - val_loss: 0.1115 - val_accuracy: 0.9780
Epoch  9/10 loss: 0.0050 - accuracy: 0.9985 - val_loss: 0.1144 - val_accuracy: 0.9800
Epoch 10/10 loss: 0.0023 - accuracy: 0.9995 - val_loss: 0.1127 - val_accuracy: 0.9788 - val_test_loss: 0.0975 - val_test_accuracy: 0.9812

loss: 0.7989 - accuracy: 0.8098 - val_loss: 0.3662 - val_accuracy: 0.9146 - val_test_loss: 0.4247 - val_test_accuracy: 0.8926

loss: 0.3617 - accuracy: 0.8973 - val_loss: 0.1684 - val_accuracy: 0.9560 - val_test_loss: 0.2011 - val_test_accuracy: 0.9456

loss: 0.3502 - accuracy: 0.9021 - val_loss: 0.1679 - val_accuracy: 0.9576 - val_test_loss: 0.1996 - val_test_accuracy: 0.9454

loss: 0.2699 - accuracy: 0.9231 - val_loss: 0.1166 - val_accuracy: 0.9686 - val_test_loss: 0.1385 - val_test_accuracy: 0.9605

loss: 0.2354 - accuracy: 0.9290 - val_loss: 0.1425 - val_accuracy: 0.9576 - val_test_loss: 0.1692 - val_test_accuracy: 0.9469

Epoch 1/2 loss: 0.1961 - accuracy: 0.9400 - val_loss: 0.0890 - val_accuracy: 0.9728
Epoch 2/2 loss: 0.0663 - accuracy: 0.9792 - val_loss: 0.0675 - val_accuracy: 0.9790 - val_test_loss: 0.0721 - val_test_accuracy: 0.9773
Final learning rate: 0.001

Epoch 1/2 loss: 0.2111 - accuracy: 0.9356 - val_loss: 0.1017 - val_accuracy: 0.9690
Epoch 2/2 loss: 0.0701 - accuracy: 0.9781 - val_loss: 0.0708 - val_accuracy: 0.9790 - val_test_loss: 0.0693 - val_test_accuracy: 0.9779
Final learning rate: 0.0001

mnist_regularization

 Deadline: Mar 14, 7:59 a.m.  3 points  Tests

You will learn how to implement three regularization methods in this assignment. Start with the mnist_regularization.py template and implement the following:

• Allow using dropout with rate args.dropout. Add a dropout layer after the first Flatten and also after all Dense hidden layers (but not after the output layer).
• Allow using L2 regularization with weight args.l2. Use tf.keras.regularizers.L2 as a regularizer for all kernels (but not biases) of all Dense layers (including the last one).
• Allow using label smoothing with weight args.label_smoothing. Instead of SparseCategoricalCrossentropy, you will need to use CategoricalCrossentropy which offers label_smoothing argument.

In ReCodEx, there will be six tests (two for each regularization methods) and you will get half a point for passing each one.

In addition to submitting the task in ReCodEx, also run the following variations and observe the results in TensorBoard (or online here), notably the training, development and test set accuracy and loss:

• dropout rate 0, 0.3, 0.5, 0.6, 0.8;
• l2 regularization 0, 0.001, 0.0001, 0.00001;
• label smoothing 0, 0.1, 0.3, 0.5.

loss: 0.7987 - accuracy: 0.7616 - val_loss: 0.3175 - val_accuracy: 0.9110 - val_test_loss: 0.3825 - val_test_accuracy: 0.8882

loss: 1.4363 - accuracy: 0.5090 - val_loss: 0.4447 - val_accuracy: 0.8862 - val_test_loss: 0.5256 - val_test_accuracy: 0.8537

loss: 0.9748 - accuracy: 0.8374 - val_loss: 0.5730 - val_accuracy: 0.9188 - val_test_loss: 0.6294 - val_test_accuracy: 0.9049

loss: 0.6501 - accuracy: 0.8396 - val_loss: 0.3136 - val_accuracy: 0.9210 - val_test_loss: 0.3704 - val_test_accuracy: 0.9075

loss: 0.9918 - accuracy: 0.8436 - val_loss: 0.7645 - val_accuracy: 0.9254 - val_test_loss: 0.8047 - val_test_accuracy: 0.9095

loss: 1.5068 - accuracy: 0.8428 - val_loss: 1.3686 - val_accuracy: 0.9332 - val_test_loss: 1.3936 - val_test_accuracy: 0.9125

mnist_ensemble

 Deadline: Mar 14, 7:59 a.m.  2 points  Examples  Tests

Your goal in this assignment is to implement model ensembling. The mnist_ensemble.py template trains args.models individual models, and your goal is to perform an ensemble of the first model, first two models, first three models, …, all models, and evaluate their accuracy on the test set.

Model 1, individual accuracy 97.69, ensemble accuracy 97.69
Model 2, individual accuracy 97.75, ensemble accuracy 98.03
Model 3, individual accuracy 97.90, ensemble accuracy 98.08
Model 4, individual accuracy 97.52, ensemble accuracy 98.05
Model 5, individual accuracy 97.59, ensemble accuracy 98.14

Model 1, individual accuracy 97.86, ensemble accuracy 97.86
Model 2, individual accuracy 98.09, ensemble accuracy 98.27
Model 3, individual accuracy 98.15, ensemble accuracy 98.41
Model 4, individual accuracy 98.13, ensemble accuracy 98.45
Model 5, individual accuracy 97.79, ensemble accuracy 98.39

Model 1, individual accuracy 95.17, ensemble accuracy 95.17
Model 2, individual accuracy 94.75, ensemble accuracy 95.10
Model 3, individual accuracy 95.19, ensemble accuracy 95.11
Model 4, individual accuracy 95.11, ensemble accuracy 95.13
Model 5, individual accuracy 95.20, ensemble accuracy 95.24

Model 1, individual accuracy 96.05, ensemble accuracy 96.05
Model 2, individual accuracy 96.11, ensemble accuracy 96.21
Model 3, individual accuracy 95.76, ensemble accuracy 96.16
Model 4, individual accuracy 95.85, ensemble accuracy 96.08
Model 5, individual accuracy 95.94, ensemble accuracy 96.10

uppercase

 Deadline: Mar 14, 7:59 a.m.  4 points+5 bonus

This assignment introduces first NLP task. Your goal is to implement a model which is given Czech lowercased text and tries to uppercase appropriate letters. To load the dataset, use uppercase_data.py module which loads (and if required also downloads) the data. While the training and the development sets are in correct case, the test set is lowercased.

This is an open-data task, where you submit only the uppercased test set together with the training script (which will not be executed, it will be only used to understand the approach you took, and to indicate teams). Explicitly, submit exactly one .txt file and at least one .py/ipynb file.

The task is also a competition. Everyone who submits a solution which achieves at least 98.5% accuracy will get 4 basic points; the 5 bonus points will be distributed depending on relative ordering of your solutions. The accuracy is computed per-character and can be evaluated by running uppercase_data.py with --evaluate argument, or using its evaluate_file method.

You may want to start with the uppercase.py template, which uses the uppercase_data.py to load the data, generate an alphabet of given size containing most frequent characters, and generate sliding window view on the data. The template also comments on possibilities of character representation.

Do not use RNNs, CNNs or Transformer in this task (if you have doubts, contact me).

mnist_cnn

 Deadline: Mar 21, 7:59 a.m.  3 points  Tests

To pass this assignment, you will learn to construct basic convolutional neural network layers. Start with the mnist_cnn.py template and assume the requested architecture is described by the cnn argument, which contains comma-separated specifications of the following layers:

• C-filters-kernel_size-stride-padding: Add a convolutional layer with ReLU activation and specified number of filters, kernel size, stride and padding. Example: C-10-3-1-same
• CB-filters-kernel_size-stride-padding: Same as C-filters-kernel_size-stride-padding, but use batch normalization. In detail, start with a convolutional layer without bias and activation, then add batch normalization layer, and finally ReLU activation. Example: CB-10-3-1-same
• M-pool_size-stride: Add max pooling with specified size and stride, using the default "valid" padding. Example: M-3-2
• R-[layers]: Add a residual connection. The layers contain a specification of at least one convolutional layer (but not a recursive residual connection R). The input to the R layer should be processed sequentially by layers, and the produced output (after the ReLU nonlinearty of the last layer) should be added to the input (of this R layer). Example: R-[C-16-3-1-same,C-16-3-1-same]
• F: Flatten inputs. Must appear exactly once in the architecture.
• H-hidden_layer_size: Add a dense layer with ReLU activation and specified size. Example: H-100
• D-dropout_rate: Apply dropout with the given dropout rate. Example: D-0.5

An example architecture might be --cnn=CB-16-5-2-same,M-3-2,F,H-100,D-0.5. You can assume the resulting network is valid; it is fine to crash if it is not.

After a successful ReCodEx submission, you can try obtaining the best accuracy on MNIST and then advance to cifar_competition.

loss: 0.3195 - accuracy: 0.9109 - val_loss: 0.1459 - val_accuracy: 0.9612

loss: 0.4794 - accuracy: 0.8575 - val_loss: 0.1617 - val_accuracy: 0.9596

loss: 0.7223 - accuracy: 0.7766 - val_loss: 0.3934 - val_accuracy: 0.8818

loss: 0.7543 - accuracy: 0.7567 - val_loss: 0.3445 - val_accuracy: 0.9004

loss: 0.5990 - accuracy: 0.8108 - val_loss: 0.2519 - val_accuracy: 0.9230

loss: 0.4530 - accuracy: 0.8567 - val_loss: 0.1787 - val_accuracy: 0.9464

image_augmentation

 Deadline: Mar 21, 7:59 a.m.  1 points  Tests

The template image_augmentation.py creates a simple convolutional network for classifying CIFAR-10. Your goal is to perform image data augmentation operations using ImageDataGenerator and to utilize these data during training.

loss: 2.1985 - accuracy: 0.1670 - val_loss: 1.9781 - val_accuracy: 0.2620

loss: 2.1988 - accuracy: 0.1678 - val_loss: 1.9996 - val_accuracy: 0.2680

tf_dataset

 Deadline: Mar 21, 7:59 a.m.  2 points  Tests

In this assignment you will familiarize yourselves with tf.data, which is TensorFlow high-level API for constructing input pipelines. If you want, you can read an official TensorFlow tf.data guide or reference API manual.

The goal of this assignment is to implement image augmentation preprocessing similar to image_augmentation, but with tf.data. Start with the tf_dataset.py template and implement the input pipelines employing the tf.data.Dataset.

loss: 2.1262 - accuracy: 0.1998 - val_loss: 1.8775 - val_accuracy: 0.3040

loss: 2.2113 - accuracy: 0.1618 - val_loss: 2.0246 - val_accuracy: 0.2640

mnist_multiple

 Deadline: Mar 21, 7:59 a.m.  3 points  Tests

In this assignment you will implement a model with multiple inputs and outputs. Start with the mnist_multiple.py template and:

• The goal is to create a model, which given two input MNIST images predicts, if the digit on the first one is larger than on the second one.
• The model has four outputs:
  • direct prediction whether the first digit is larger than the second one,
  • digit classification for the first image,
  • digit classification for the second image,
  • indirect prediction comparing the digits predicted by the above two outputs.
• You need to implement:
  • the model, using multiple inputs, outputs, losses and metrics;
  • construction of two-image dataset examples using regular MNIST data via the tf.data API.

loss: 0.9233 - digit_1_loss: 0.3067 - digit_2_loss: 0.3131 - direct_prediction_loss: 0.3036 - direct_prediction_accuracy: 0.8617 - indirect_prediction_accuracy: 0.9424 - val_loss: 0.3590 - val_digit_1_loss: 0.1264 - val_digit_2_loss: 0.0725 - val_direct_prediction_loss: 0.1601 - val_direct_prediction_accuracy: 0.9400 - val_indirect_prediction_accuracy: 0.9796

loss: 1.2151 - digit_1_loss: 0.4227 - digit_2_loss: 0.4280 - direct_prediction_loss: 0.3645 - direct_prediction_accuracy: 0.8301 - indirect_prediction_accuracy: 0.9257 - val_loss: 0.4846 - val_digit_1_loss: 0.1704 - val_digit_2_loss: 0.0990 - val_direct_prediction_loss: 0.2153 - val_direct_prediction_accuracy: 0.9164 - val_indirect_prediction_accuracy: 0.9700

cifar_competition

 Deadline: Mar 21, 7:59 a.m.  5 points+5 bonus

The goal of this assignment is to devise the best possible model for CIFAR-10. You can load the data using the cifar10.py module. Note that the test set is different than that of official CIFAR-10.

The task is a competition. Everyone who submits a solution which achieves at least 65% test set accuracy will get 5 points; the rest 5 points will be distributed depending on relative ordering of your solutions. Note that my solutions usually need to achieve around ~80% on the development set to score 65% on the test set.

You may want to start with the cifar_competition.py template which generates the test set annotation in the required format.

cnn_manual

 Deadline: Mar 28, 7:59 a.m.  3 points  Tests

To pass this assignment, you need to manually implement the forward and backward pass through a 2D convolutional layer. Start with the cnn_manual.py template, which constructs a series of 2D convolutional layers with ReLU activation and valid padding, specified in the args.cnn option. The args.cnn contains comma-separated layer specifications in the format filters-kernel_size-stride.

Of course, you cannot use any TensorFlow convolutional operation (instead, implement the forward and backward pass using matrix multiplication and other operations), nor the tf.GradientTape for gradient computation.

To make debugging easier, the template supports a --verify option, which allows comparing the forward pass and the three gradients you compute in the backward pass to correct values.

Finally, it is a good idea to read the TensorFlow guide about tensor slicing.

Dev accuracy after epoch 1 is 91.06
Test accuracy after epoch 1 is 89.51

Dev accuracy after epoch 1 is 94.08
Test accuracy after epoch 1 is 92.65

Dev accuracy after epoch 1 is 91.82
Test accuracy after epoch 1 is 90.00

Dev accuracy after epoch 1 is 93.22
Test accuracy after epoch 1 is 91.31

cags_classification

 Deadline: Mar 28, 7:59 a.m.  4 points+5 bonus

The goal of this assignment is to use pretrained EfficientNet-B0 model to achieve best accuracy in CAGS classification.

The CAGS dataset consists of images of cats and dogs of size $224×224$, each classified in one of the 34 breeds and each containing a mask indicating the presence of the animal. To load the dataset, use the cags_dataset.py module. The dataset is stored in a TFRecord file and each element is encoded as a tf.train.Example, which is decoded using the CAGS.parse method.

To load the EfficientNet-B0, use the provided efficient_net.py module. Its method pretrained_efficientnet_b0(include_top, dynamic_input_shape=False):

• downloads the pretrained weights if they are not found;
• it returns a tf.keras.Model processing image of shape $(224, 224, 3)$ with float values in range $[0, 1]$ and producing a list of results:
  • the first value is the final network output:
    • if include_top == True, the network will include the final classification layer and produce a distribution on 1000 classes (whose names are in imagenet_classes.py);
    • if include_top == False, the network will return image features (the result of the last global average pooling);
  • the rest of outputs are the intermediate results of the network just before a convolution with $\textit{stride} > 1$ is performed (denoted $C_5, C_4, C_3, C_2, C_1$ in the Object Detection lecture).

An example performing classification of given images is available in image_classification.py.

A note on finetuning: each tf.keras.layers.Layer has a mutable trainable property indicating whether its variables should be updated – however, after changing it, you need to call .compile again (or otherwise make sure the list of trainable variables for the optimizer is updated). Furthermore, training argument passed to the invocation call decides whether the layer is executed in training regime (neurons gets dropped in dropout, batch normalization computes estimates on the batch) or in inference regime. There is one exception though – if trainable == False on a batch normalization layer, it runs in the inference regime even when training == True.

The task is a competition. Everyone who submits a solution which achieves at least 90% test set accuracy will get 4 points; the rest 5 points will be distributed depending on relative ordering of your solutions.

You may want to start with the cags_classification.py template which generates the test set annotation in the required format.

cags_segmentation

 Deadline: Mar 28, 7:59 a.m.  4 points+5 bonus

The goal of this assignment is to use pretrained EfficientNet-B0 model to achieve best image segmentation IoU score on the CAGS dataset. The dataset and the EfficientNet-B0 is described in the cags_classification assignment.

A mask is evaluated using intersection over union (IoU) metric, which is the intersection of the gold and predicted mask divided by their union, and the whole test set score is the average of its masks' IoU. A TensorFlow compatible metric is implemented by the class MaskIoUMetric of the cags_dataset.py module, which can also evaluate your predictions (either by running with --task=segmentation --evaluate=path arguments, or using its evaluate_segmentation_file method).

The task is a competition. Everyone who submits a solution which achieves at least 87% test set IoU gets 4 points; the rest 5 points will be distributed depending on relative ordering of your solutions.

You may want to start with the cags_segmentation.py template, which generates the test set annotation in the required format – each mask should be encoded on a single line as a space separated sequence of integers indicating the length of alternating runs of zeros and ones.

bboxes_utils

 Deadline: Apr 4, 7:59 a.m.  2 points

This is a preparatory assignment for svhn_competition. The goal is to implement several bounding box manipulation routines in the bboxes_utils.py module. Notably, you need to implement the following methods:

• bboxes_to_fast_rcnn: convert given bounding boxes to a Fast R-CNN-like representation relative to the given anchors;
• bboxes_from_fast_rcnn: convert Fast R-CNN-like representations relative to given anchors back to bounding boxes;
• bboxes_training: given a list of anchors and gold objects, assign gold objects to anchors and generate suitable training data (the exact algorithm is described in the template).

The bboxes_utils.py contains simple unit tests, which are evaluated when executing the module, which you can use to check the validity of your implementation. Note that the template does not contain type annotations because Python typing system is not flexible enough to describe the tensor shape changes.

When submitting to ReCodEx, the method main is executed, returning the implemented bboxes_to_fast_rcnn, bboxes_from_fast_rcnn and bboxes_training methods. These methods are then executed and compared to the reference implementation.

svhn_competition

 Deadline: Apr 4, 7:59 a.m.  5 points+5 bonus

The goal of this assignment is to implement a system performing object recognition, optionally utilizing pretrained EfficientNet-B0 backbone.

The Street View House Numbers (SVHN) dataset annotates for every photo all digits appearing on it, including their bounding boxes. The dataset can be loaded using the svhn_dataset.py module. Similarly to the CAGS dataset, it is stored in a TFRecord file with tf.train.Example elements. Every element is a dictionary with the following keys:

• "image": a square 3-channel image,
• "classes": a 1D tensor with all digit labels appearing in the image,
• "bboxes": a [num_digits, 4] 2D tensor with bounding boxes of every digit in the image.

Given that the dataset elements are each of possibly different size and you want to preprocess them using bboxes_training, it might be more comfortable to convert the dataset to NumPy. Alternatively, you can implement bboxes_training using TensorFlow operations or call Numpy implementation of bboxes_training directly in tf.data.Dataset.map by using tf.numpy_function, see FAQ.

Similarly to the cags_classification, you can load the EfficientNet-B0 using the provided efficient_net.py module. Note that the dynamic_input_shape=True argument creates a model capable of processing an input image of any size.

Each test set image annotation consists of a sequence of space separated five-tuples label top left bottom right, and the annotation is considered correct, if exactly the gold digits are predicted, each with IoU at least 0.5. The whole test set score is then the prediction accuracy of individual images. You can again evaluate your predictions using the svhn_dataset.py module, either by running with --evaluate=path arguments, or using its evaluate_file method.

The task is a competition. Everyone who submits a solution which achieves at least 20% test set IoU gets 5 points; the rest 5 points will be distributed depending on relative ordering of your solutions. Note that I usually need at least 35% development set accuracy to achieve the required test set performance.

You should start with the svhn_competition.py template, which generates the test set annotation in the required format.

A baseline solution can use RetinaNet-like single stage detector, using only a single level of convolutional features (no FPN) with single-scale and single-aspect anchors. Focal loss is available as tf.losses.BinaryFocalCrossentropy and non-maximum suppression as tf.image.non_max_suppression or tf.image.combined_non_max_suppression.

sequence_classification

 Deadline: Apr 11, 7:59 a.m.  2 points  Examples  Tests

The goal of this assignment is to introduce recurrent neural networks. Considering recurrent neural network, the assignment shows convergence speed and illustrates exploding gradient issue. The network should process sequences of 50 small integers and compute parity for each prefix of the sequence. The inputs are either 0/1, or vectors with one-hot representation of small integer.

Your goal is to modify the sequence_classification.py template and implement the following:

• Use specified RNN type (SimpleRNN, GRU and LSTM) and dimensionality.
• Process the sequence using the required RNN.
• Use additional hidden layer on the RNN outputs if requested.
• Implement gradient clipping if requested.

In addition to submitting the task in ReCodEx, please also run the following variations and observe the results in TensorBoard (or online here). Concentrate on the way how the RNNs converge, convergence speed, exploding gradient issues and how gradient clipping helps:

• --rnn_cell=SimpleRNN --sequence_dim=1, --rnn_cell=GRU --sequence_dim=1, --rnn_cell=LSTM --sequence_dim=1
• the same as above but with --sequence_dim=2
• the same as above but with --sequence_dim=10
• --rnn_cell=SimpleRNN --hidden_layer=70 --rnn_cell_dim=30 --sequence_dim=30 and the same with --clip_gradient=1
• the same as above but with --rnn_cell=GRU --hidden_layer=75 with and without --clip_gradient=0.1
• the same as above but with --rnn_cell=LSTM --hidden_layer=85 with and without --clip_gradient=1

Epoch 1/5 loss: 0.6951 - accuracy: 0.5095 - val_loss: 0.6926 - val_accuracy: 0.5176
Epoch 2/5 loss: 0.6924 - accuracy: 0.5158 - val_loss: 0.6921 - val_accuracy: 0.5217
Epoch 3/5 loss: 0.6918 - accuracy: 0.5165 - val_loss: 0.6913 - val_accuracy: 0.5114
Epoch 4/5 loss: 0.6901 - accuracy: 0.5196 - val_loss: 0.6881 - val_accuracy: 0.5157
Epoch 5/5 loss: 0.6842 - accuracy: 0.5220 - val_loss: 0.6793 - val_accuracy: 0.5231

Epoch 1/5 loss: 0.6926 - accuracy: 0.5126 - val_loss: 0.6917 - val_accuracy: 0.5157
Epoch 2/5 loss: 0.6885 - accuracy: 0.5170 - val_loss: 0.6823 - val_accuracy: 0.5143
Epoch 3/5 loss: 0.4987 - accuracy: 0.7328 - val_loss: 0.1574 - val_accuracy: 0.9795
Epoch 4/5 loss: 0.0684 - accuracy: 0.9935 - val_loss: 0.0305 - val_accuracy: 0.9975
Epoch 5/5 loss: 0.0219 - accuracy: 0.9991 - val_loss: 0.0121 - val_accuracy: 0.9998

Epoch 1/5 loss: 0.6929 - accuracy: 0.5130 - val_loss: 0.6927 - val_accuracy: 0.5153
Epoch 2/5 loss: 0.6919 - accuracy: 0.5155 - val_loss: 0.6902 - val_accuracy: 0.5156
Epoch 3/5 loss: 0.6837 - accuracy: 0.5192 - val_loss: 0.6748 - val_accuracy: 0.5285
Epoch 4/5 loss: 0.3839 - accuracy: 0.7918 - val_loss: 0.0695 - val_accuracy: 1.0000
Epoch 5/5 loss: 0.0351 - accuracy: 1.0000 - val_loss: 0.0183 - val_accuracy: 1.0000

Epoch 1/5 loss: 0.6807 - accuracy: 0.5193 - val_loss: 0.6615 - val_accuracy: 0.5233
Epoch 2/5 loss: 0.6485 - accuracy: 0.5373 - val_loss: 0.6378 - val_accuracy: 0.5309
Epoch 3/5 loss: 0.6204 - accuracy: 0.5641 - val_loss: 0.5772 - val_accuracy: 0.6306
Epoch 4/5 loss: 0.0874 - accuracy: 0.9566 - val_loss: 0.0015 - val_accuracy: 1.0000
Epoch 5/5 loss: 8.0165e-04 - accuracy: 1.0000 - val_loss: 3.8375e-04 - val_accuracy: 1.0000

Epoch 1/5 loss: 0.6818 - accuracy: 0.5173 - val_loss: 0.6676 - val_accuracy: 0.5241
Epoch 2/5 loss: 0.6509 - accuracy: 0.5374 - val_loss: 0.6393 - val_accuracy: 0.5448
Epoch 3/5 loss: 0.6301 - accuracy: 0.5458 - val_loss: 0.6148 - val_accuracy: 0.5622
Epoch 4/5 loss: 0.5852 - accuracy: 0.6121 - val_loss: 0.4589 - val_accuracy: 0.7884
Epoch 5/5 loss: 0.0372 - accuracy: 0.9881 - val_loss: 0.0060 - val_accuracy: 0.9993

Epoch 1/5 loss: 0.7125 - accuracy: 0.4996 - val_loss: 0.6997 - val_accuracy: 0.4929
Epoch 2/5 loss: 0.6962 - accuracy: 0.4948 - val_loss: 0.6935 - val_accuracy: 0.4985
Epoch 3/5 loss: 0.6931 - accuracy: 0.5155 - val_loss: 0.6922 - val_accuracy: 0.5264
Epoch 4/5 loss: 0.6923 - accuracy: 0.5286 - val_loss: 0.6917 - val_accuracy: 0.5362
Epoch 5/5 loss: 0.6917 - accuracy: 0.5343 - val_loss: 0.6913 - val_accuracy: 0.5323

Epoch 1/5 loss: 0.6926 - accuracy: 0.5243 - val_loss: 0.6922 - val_accuracy: 0.5217
Epoch 2/5 loss: 0.6922 - accuracy: 0.5210 - val_loss: 0.6920 - val_accuracy: 0.5217
Epoch 3/5 loss: 0.6919 - accuracy: 0.5247 - val_loss: 0.6916 - val_accuracy: 0.5217
Epoch 4/5 loss: 0.6917 - accuracy: 0.5301 - val_loss: 0.6913 - val_accuracy: 0.5217
Epoch 5/5 loss: 0.6912 - accuracy: 0.5276 - val_loss: 0.6908 - val_accuracy: 0.5220

Epoch 1/5 loss: 0.6928 - accuracy: 0.5358 - val_loss: 0.6925 - val_accuracy: 0.5339
Epoch 2/5 loss: 0.6926 - accuracy: 0.5319 - val_loss: 0.6924 - val_accuracy: 0.5279
Epoch 3/5 loss: 0.6925 - accuracy: 0.5298 - val_loss: 0.6923 - val_accuracy: 0.5343
Epoch 4/5 loss: 0.6924 - accuracy: 0.5332 - val_loss: 0.6922 - val_accuracy: 0.5297
Epoch 5/5 loss: 0.6922 - accuracy: 0.5358 - val_loss: 0.6920 - val_accuracy: 0.5293

Epoch 1/5 loss: 0.6917 - accuracy: 0.5434 - val_loss: 0.6903 - val_accuracy: 0.5306
Epoch 2/5 loss: 0.6876 - accuracy: 0.5395 - val_loss: 0.6843 - val_accuracy: 0.5350
Epoch 3/5 loss: 0.6784 - accuracy: 0.5550 - val_loss: 0.6732 - val_accuracy: 0.5350
Epoch 4/5 loss: 0.6667 - accuracy: 0.5549 - val_loss: 0.6620 - val_accuracy: 0.5299
Epoch 5/5 loss: 0.6547 - accuracy: 0.5597 - val_loss: 0.6508 - val_accuracy: 0.5278

Epoch 1/5 loss: 0.6916 - accuracy: 0.5417 - val_loss: 0.6903 - val_accuracy: 0.5308
Epoch 2/5 loss: 0.6876 - accuracy: 0.5390 - val_loss: 0.6844 - val_accuracy: 0.5305
Epoch 3/5 loss: 0.6789 - accuracy: 0.5533 - val_loss: 0.6742 - val_accuracy: 0.5333
Epoch 4/5 loss: 0.6675 - accuracy: 0.5512 - val_loss: 0.6629 - val_accuracy: 0.5411
Epoch 5/5 loss: 0.6563 - accuracy: 0.5532 - val_loss: 0.6536 - val_accuracy: 0.5332

tagger_we

 Deadline: Apr 11, 7:59 a.m.  3 points  Examples  Tests

In this assignment you will create a simple part-of-speech tagger. For training and evaluation, we will use Czech dataset containing tokenized sentences, each word annotated by gold lemma and part-of-speech tag. The morpho_dataset.py module (down)loads the dataset and provides mappings between strings and integers.

Your goal is to modify the tagger_we.py template and implement the following:

• Use specified RNN cell type (GRU and LSTM) and dimensionality.
• Create word embeddings for training vocabulary.
• Process the sentences using bidirectional RNN.
• Predict part-of-speech tags. Note that you need to properly handle sentences of different lengths in one batch using tf.RaggedTensors.

Epoch 1/5 loss: 1.3015 - accuracy: 0.6197 - val_loss: 0.5595 - val_accuracy: 0.8438
Epoch 2/5 loss: 0.2059 - accuracy: 0.9561 - val_loss: 0.3388 - val_accuracy: 0.8954
Epoch 3/5 loss: 0.0510 - accuracy: 0.9888 - val_loss: 0.3189 - val_accuracy: 0.8941
Epoch 4/5 loss: 0.0306 - accuracy: 0.9920 - val_loss: 0.3265 - val_accuracy: 0.8916
Epoch 5/5 loss: 0.0213 - accuracy: 0.9947 - val_loss: 0.3260 - val_accuracy: 0.8926

Epoch 1/5 loss: 0.9769 - accuracy: 0.7228 - val_loss: 0.4172 - val_accuracy: 0.8750
Epoch 2/5 loss: 0.1204 - accuracy: 0.9740 - val_loss: 0.3330 - val_accuracy: 0.8852
Epoch 3/5 loss: 0.0365 - accuracy: 0.9900 - val_loss: 0.3138 - val_accuracy: 0.8903
Epoch 4/5 loss: 0.0261 - accuracy: 0.9919 - val_loss: 0.3234 - val_accuracy: 0.8840
Epoch 5/5 loss: 0.0203 - accuracy: 0.9935 - val_loss: 0.3246 - val_accuracy: 0.8837

loss: 2.3174 - accuracy: 0.3603 - val_loss: 1.9011 - val_accuracy: 0.4222

loss: 2.1435 - accuracy: 0.4186 - val_loss: 1.5338 - val_accuracy: 0.5498

tagger_cle

 Deadline: Apr 11, 7:59 a.m.  3 points  Examples  Tests

This assignment is a continuation of tagger_we. Using the tagger_cle.py template, implement character-level word embedding computation using a bidirectional character-level GRU.

Once submitted to ReCodEx, you should experiment with the effect of CLEs compared to a plain tagger_we, and the influence of their dimensionality. Note that tagger_cle has by default smaller word embeddings so that the size of word representation (64 + 32 + 32) is the same as in the tagger_we assignment.

Epoch 1/5 loss: 1.2229 - accuracy: 0.6372 - val_loss: 0.4645 - val_accuracy: 0.8702
Epoch 2/5 loss: 0.1907 - accuracy: 0.9598 - val_loss: 0.2491 - val_accuracy: 0.9249
Epoch 3/5 loss: 0.0557 - accuracy: 0.9883 - val_loss: 0.2151 - val_accuracy: 0.9267
Epoch 4/5 loss: 0.0344 - accuracy: 0.9910 - val_loss: 0.2125 - val_accuracy: 0.9277
Epoch 5/5 loss: 0.0262 - accuracy: 0.9925 - val_loss: 0.2069 - val_accuracy: 0.9295

Epoch 1/5 loss: 1.3114 - accuracy: 0.6076 - val_loss: 0.5267 - val_accuracy: 0.8527
Epoch 2/5 loss: 0.3150 - accuracy: 0.9197 - val_loss: 0.2760 - val_accuracy: 0.9161
Epoch 3/5 loss: 0.1540 - accuracy: 0.9588 - val_loss: 0.2244 - val_accuracy: 0.9294
Epoch 4/5 loss: 0.1123 - accuracy: 0.9676 - val_loss: 0.2145 - val_accuracy: 0.9309
Epoch 5/5 loss: 0.0961 - accuracy: 0.9700 - val_loss: 0.2049 - val_accuracy: 0.9344

loss: 2.2428 - accuracy: 0.3493 - val_loss: 1.8235 - val_accuracy: 0.4233

loss: 2.2494 - accuracy: 0.3465 - val_loss: 1.8439 - val_accuracy: 0.4232

tagger_competition

 Deadline: Apr 11, 7:59 a.m.  4 points+5 bonus

In this assignment, you should extend tagger_cle into a real-world Czech part-of-speech tagger. We will use Czech PDT dataset loadable using the morpho_dataset.py module. Note that the dataset contains more than 1500 unique POS tags and that the POS tags have a fixed structure of 15 positions (so it is possible to generate the POS tag characters independently).

You can use the following additional data in this assignment:

• You can use outputs of a morphological analyzer loadable with morpho_analyzer.py. If a word form in train, dev or test PDT data is known to the analyzer, all its (lemma, POS tag) pairs are returned.
• You can use any unannotated text data (Wikipedia, Czech National Corpus, …), and also any pre-trained word embeddings (assuming they were trained on plain texts).

The task is a competition. Everyone who submits a solution with at least 92.5% label accuracy gets 4 points; the rest 5 points will be distributed depending on relative ordering of your solutions. Lastly, 3 bonus points will be given to anyone surpassing pre-neural-network state-of-the-art of 96.35%.

You can start with the tagger_competition.py template, which among others generates test set annotations in the required format. Note that you can evaluate the predictions as usual using the morpho_dataset.py module, either by running with --task=tagger --evaluate=path arguments, or using its evaluate_file method.

tensorboard_projector

You can try exploring the TensorBoard Projector with pre-trained embeddings for 20k most frequent lemmas in Czech and English – after extracting the archive, start tensorboard --logdir dir_where_the_archive_is_extracted.

In order to use the Projector tab yourself, you can take inspiration from the projector_export.py script, which was used to export the above pre-trained embeddings from the Word2vec format.

tagger_crf

 Deadline: Apr 19, 7:59 a.m.  2 points  Examples  Tests

This assignment is an extension of tagger_we task. Using the tagger_crf.py template, implement named entity recognition using CRF loss and CRF decoding from the tensorflow_addons package.

The evaluation is performed using the provided metric computing F1 score of the span prediction (i.e., a recognized possibly-multiword named entity is true positive if both the entity type and the span exactly match).

In practice, character-level embeddings (and also pre-trained word embeddings) would be used to obtain superior results.

Epoch 1/5 loss: 18.5371 - val_loss: 14.0865 - val_f1: 0.0317
Epoch 2/5 loss: 9.7936 - val_loss: 11.5969 - val_f1: 0.2428
Epoch 3/5 loss: 5.9049 - val_loss: 9.8079 - val_f1: 0.3645
Epoch 4/5 loss: 3.1811 - val_loss: 9.5350 - val_f1: 0.4276
Epoch 5/5 loss: 1.7330 - val_loss: 9.2801 - val_f1: 0.4398

Epoch 1/5 loss: 17.6696 - val_loss: 13.5141 - val_f1: 0.1700
Epoch 2/5 loss: 8.1954 - val_loss: 10.2339 - val_f1: 0.4070
Epoch 3/5 loss: 3.7555 - val_loss: 9.4217 - val_f1: 0.4528
Epoch 4/5 loss: 1.6607 - val_loss: 10.1525 - val_f1: 0.4546
Epoch 5/5 loss: 0.8472 - val_loss: 10.6141 - val_f1: 0.4744

Epoch 1/2 loss: 29.9874 - val_loss: 20.2837 - val_f1: 0.0000e+00
Epoch 2/2 loss: 17.2559 - val_loss: 18.0548 - val_f1: 0.0030

Epoch 1/2 loss: 29.1122 - val_loss: 19.1089 - val_f1: 0.0000e+00
Epoch 2/2 loss: 15.7085 - val_loss: 17.1493 - val_f1: 0.0172

tagger_crf_manual

 Deadline: Apr 19, 7:59 a.m.  2 points  Examples  Tests

This assignment is an extension of tagger_crf, where we will perform the CRF loss computation (but not CRF decoding) manually.

The tagger_crf_manual.py template is nearly identical to tagger_crf, the only difference is the crf_loss method, where you should manually implement the CRF loss.

Epoch 1/5 loss: 18.5371 - val_loss: 14.0865 - val_f1: 0.0317
Epoch 2/5 loss: 9.7936 - val_loss: 11.5969 - val_f1: 0.2428
Epoch 3/5 loss: 5.9049 - val_loss: 9.8079 - val_f1: 0.3645
Epoch 4/5 loss: 3.1811 - val_loss: 9.5350 - val_f1: 0.4276
Epoch 5/5 loss: 1.7330 - val_loss: 9.2801 - val_f1: 0.4398

Epoch 1/5 loss: 17.6696 - val_loss: 13.5141 - val_f1: 0.1700
Epoch 2/5 loss: 8.1954 - val_loss: 10.2339 - val_f1: 0.4070
Epoch 3/5 loss: 3.7555 - val_loss: 9.4217 - val_f1: 0.4528
Epoch 4/5 loss: 1.6607 - val_loss: 10.1525 - val_f1: 0.4546
Epoch 5/5 loss: 0.8472 - val_loss: 10.6141 - val_f1: 0.4744

Epoch 1/2 loss: 29.9874 - val_loss: 20.2837 - val_f1: 0.0000e+00
Epoch 2/2 loss: 17.2559 - val_loss: 18.0548 - val_f1: 0.0030

Epoch 1/2 loss: 29.1122 - val_loss: 19.1089 - val_f1: 0.0000e+00
Epoch 2/2 loss: 15.7085 - val_loss: 17.1493 - val_f1: 0.0172

speech_recognition

 Deadline: Apr 19, 7:59 a.m.  5 points+5 bonus

This assignment is a competition task in speech recognition area. Specifically, your goal is to predict a sequence of letters given a spoken utterance. We will be using Czech recordings from the Common Voice, with input sound waves passed through the usual preprocessing – computing Mel-frequency cepstral coefficients (MFCCs). You can repeat this preprocessing on a given audio using the wav_decode and mfcc_extract methods from the common_voice_cs.py module. This module can also load the dataset, downloading it when necessary (note that it has 200MB, so it might take a while). Furthermore, you can listen to the development portion of the dataset.

Additional following data can be utilized in this assignment:

• You can use any unannotated text data (Wikipedia, Czech National Corpus, …), and also any pre-trained word embeddings or language models (assuming they were trained on plain texts).
• You can use any unannotated speech data.

The task is a competition. The evaluation is performed by computing the edit distance to the gold letter sequence, normalized by its length (a corresponding Keras metric EditDistanceMetric is provided by the common_voice_cs.py). Everyone who submits a solution with at most 50% test set edit distance gets 5 points; the rest 5 points will be distributed depending on relative ordering of your solutions. Note that you can evaluate the predictions as usual using the common_voice_cs.py module, either by running with --evaluate=path arguments, or using its evaluate_file method.

Start with the speech_recognition.py template which contains instructions for using the CTC loss and generates the test set annotation in the required format.

lemmatizer_noattn

 Deadline: Apr 25, 7:59 a.m.  3 points  Tests

The goal of this assignment is to create a simple lemmatizer. For training and evaluation, we use the same dataset as in tagger_we loadable by the updated morpho_dataset.py module.

Your goal is to modify the lemmatizer_noattn.py template and implement the following:

• Embed characters of source forms and run a bidirectional GRU encoder.
• Embed characters of target lemmas.
• Implement a training time decoder which uses gold target characters as inputs.
• Implement an inference time decoder which uses previous predictions as inputs.
• The initial state of both decoders is the output state of the corresponding GRU encoded form.

500/500 - 23s - loss: 2.9663 - val_accuracy: 0.1311 - 23s/epoch - 46ms/step

250/250 - 19s - loss: 3.0615 - val_accuracy: 0.0043 - 19s/epoch - 77ms/step

lemmatizer_attn

 Deadline: Apr 25, 7:59 a.m.  3 points  Tests

This task is a continuation of the lemmatizer_noattn assignment. Using the lemmatizer_attn.py template, implement the following features in addition to lemmatizer_noattn:

• The bidirectional GRU encoder returns outputs for all input characters, not just the last.
• Implement attention in the decoders. Notably, project the encoder outputs and current state into same dimensionality vectors, apply non-linearity, and generate weights for every encoder output. Finally sum the encoder outputs using these weights and concatenate the computed attention to the decoder inputs.

Once submitted to ReCodEx, you should experiment with the effect of using the attention, and the influence of RNN dimensionality on network performance.

500/500 - 39s - loss: 2.8889 - val_accuracy: 0.1451 - 39s/epoch - 78ms/step

250/250 - 29s - loss: 3.0417 - val_accuracy: 0.1471 - 29s/epoch - 114ms/step

lemmatizer_competition

 Deadline: Apr 25, 7:59 a.m.  4 points+5 bonus

In this assignment, you should extend lemmatizer_noattn or lemmatizer_attn into a real-world Czech lemmatizer. As in tagger_competition, we will use Czech PDT dataset loadable using the morpho_dataset.py module.

You can also use the same additional data as in the tagger_competition assignment.

The task is a competition. Everyone who submits a solution a solution with at least 96% label accuracy gets 4 points; the rest 5 points will be distributed depending on relative ordering of your solutions. Lastly, 3 bonus points will be given to anyone surpassing pre-neural-network state-of-the-art of 98.76%.

You can start with the lemmatizer_competition.py template, which among others generates test set annotations in the required format. Note that you can evaluate the predictions as usual using the morpho_dataset.py module, either by running with --task=lemmatizer --corpus=czech_pdt_lemmas --evaluate=path arguments, or using its evaluate_file method.

3d_recognition

 Deadline: May 02, 7:59 a.m.  3 points+4 bonus

Your goal in this assignment is to perform 3D object recognition. The input is voxelized representation of an object, stored as a 3D grid of either empty or occupied voxels, and your goal is to classify the object into one of 10 classes. The data is available in two resolutions, either as 20×20×20 data or 32×32×32 data. To load the dataset, use the modelnet.py module.

The official dataset offers only train and test sets, with the test set having a different distributions of labels. Our dataset contains also a development set, which has nearly the same label distribution as the test set.

If you want, it is possible to use the EfficientNet-B0 in this assignment; however, I do not know of a straightforward way to utilize it, apart from rendering the object to a 2D image (or several of them).

The task is a competition. Everyone who submits a solution which achieves at least 88% test set accuracy gets 3 points; the rest 4 points will be distributed depending on relative ordering of your solutions.

You can start with the 3d_recognition.py template, which among others generates test set annotations in the required format.

homr_competition

 Deadline: May 02 May 09, 7:59 a.m.  3 points+5 bonus

Tackle handwritten optical music recognition in this assignment. The inputs are grayscale images of monophonic scores starting with a clef, key signature, and a time signature, followed by several staves. The dataset is loadable using the homr_dataset.py module, and is downloaded automatically if missing (note that is has ~500MB, so it might take a while). No other data or pretrained models are allowed for training.

The task is a competition. The evaluation is performed using the same metric as in speech_recognition, by computing edit distance to the gold sequence, normalized by its length (the EditDistanceMetric is again provided by the homr_dataset.py). Everyone who submits a solution with at most 3% test set edit distance will get 3 points; the rest 5 points will be distributed depending on relative ordering of your solutions. You can evaluate the predictions as usual using the homr_dataset.py module, either by running with --evaluate=path arguments, or using its evaluate_file method.

You can start with the homr_competition.py template, which among others generates test set annotations in the required format.

tagger_transformer

 Deadline: May 09, 7:59 a.m.  3 points  Examples  Tests

This assignment is a continuation of tagger_we. Using the tagger_transformer.py template, implement a Pre-LN Transformer encoder.

Epoch 1/5 loss: 1.5585 - accuracy: 0.5335 - val_loss: 0.8724 - val_accuracy: 0.7128
Epoch 2/5 loss: 0.5397 - accuracy: 0.8519 - val_loss: 0.5609 - val_accuracy: 0.8247
Epoch 3/5 loss: 0.2519 - accuracy: 0.9560 - val_loss: 0.4491 - val_accuracy: 0.8407
Epoch 4/5 loss: 0.1310 - accuracy: 0.9775 - val_loss: 0.4135 - val_accuracy: 0.8476
Epoch 5/5 loss: 0.0796 - accuracy: 0.9843 - val_loss: 0.4004 - val_accuracy: 0.8478

Epoch 1/5 loss: 1.0680 - accuracy: 0.6571 - val_loss: 0.6104 - val_accuracy: 0.7975
Epoch 2/5 loss: 0.2136 - accuracy: 0.9307 - val_loss: 0.5002 - val_accuracy: 0.8464
Epoch 3/5 loss: 0.0605 - accuracy: 0.9811 - val_loss: 0.7676 - val_accuracy: 0.8461
Epoch 4/5 loss: 0.0361 - accuracy: 0.9878 - val_loss: 0.9315 - val_accuracy: 0.8388
Epoch 5/5 loss: 0.0263 - accuracy: 0.9906 - val_loss: 0.9784 - val_accuracy: 0.8446

Epoch 1/5 loss: 1.0682 - accuracy: 0.6598 - val_loss: 0.5239 - val_accuracy: 0.8123
Epoch 2/5 loss: 0.1897 - accuracy: 0.9391 - val_loss: 0.4625 - val_accuracy: 0.8380
Epoch 3/5 loss: 0.0556 - accuracy: 0.9824 - val_loss: 0.6330 - val_accuracy: 0.8226
Epoch 4/5 loss: 0.0337 - accuracy: 0.9885 - val_loss: 0.7936 - val_accuracy: 0.8145
Epoch 5/5 loss: 0.0266 - accuracy: 0.9904 - val_loss: 0.7206 - val_accuracy: 0.8370

Epoch 1/5 loss: 1.1690 - accuracy: 0.6259 - val_loss: 0.5695 - val_accuracy: 0.7975
Epoch 2/5 loss: 0.2457 - accuracy: 0.9220 - val_loss: 0.4771 - val_accuracy: 0.8281
Epoch 3/5 loss: 0.0870 - accuracy: 0.9730 - val_loss: 0.6044 - val_accuracy: 0.8413
Epoch 4/5 loss: 0.0525 - accuracy: 0.9828 - val_loss: 0.7615 - val_accuracy: 0.8355
Epoch 5/5 loss: 0.0430 - accuracy: 0.9854 - val_loss: 0.7607 - val_accuracy: 0.8403

loss: 2.6393 - accuracy: 0.2170 - val_loss: 2.1583 - val_accuracy: 0.3447

loss: 2.1624 - accuracy: 0.3235 - val_loss: 1.9781 - val_accuracy: 0.3119

loss: 2.1716 - accuracy: 0.3277 - val_loss: 1.9632 - val_accuracy: 0.3381

loss: 2.2652 - accuracy: 0.3063 - val_loss: 1.9840 - val_accuracy: 0.3606

sentiment_analysis

 Deadline: May 09, 7:59 a.m.  3 points

Perform sentiment analysis on Czech Facebook data using a provided pre-trained Czech Electra model eleczech-lc-small. The dataset consists of pairs of (document, label) and can be (down)loaded using the text_classification_dataset.py module. When loading the dataset, a tokenizer might be provided, and if it is, the document is also passed through the tokenizer and the resulting tokens are added to the dataset.

Even though this assignment is not a competition, your goal is to submit test set annotations with at least 77% accuracy. As usual, you can evaluate your predictions using the text_classification_dataset.py module, either by running with --evaluate=path arguments, or using its evaluate_file method.

Note that contrary to working with EfficientNet, you need to finetune the Electra model in order to achieve the required accuracy.

You can start with the sentiment_analysis.py template, which among others loads the Electra Czech model and generates test set annotations in the required format. Note that bert_example.py module illustrate the usage of both the Electra tokenizer and the Electra model.

reading_comprehension

 Deadline: May 16, 7:59 a.m.  4 points+5 bonus

Wed May 11, 14:45: Unfortunately, several contexts in the initial version of the dataset were incorrect (not belonging to the questions). The dataset has been fixed just now, so please redownload it. More details in Piazza.

Implement the best possible model for reading comprehension task using a translated version of the SQuAD 1.1 dataset, utilizing a provided Czech RoBERTa model ufal/robeczech-base.

The dataset can be loaded using the reading_comprehension_dataset.py module. The loaded dataset is the direct reprentation of the data and not yet ready to be directly trained on. Each of the train, dev and test datasets are composed of a list of paragraphs, each consisting of:

• context: text with the information;
• qas: list of questions and answers, where each item consists of:
  • question: text of the question;
  • answers: a list of answers, each answer is composed of:
    • text: string of the text, exactly as appearing in the context;
    • start: character offset of the answer text in the context.

In the train and dev sets, each question has exactly one answer, while in the test set there might be several answers. We evaluate the reading comprehension task using accuracy, where an answer is considered correct if its text is exactly equal to some correct answer. You can evaluate your predictions as usual with the reading_comprehension_dataset.py module, either by running with --evaluate=path arguments, or using its evaluate_file method.

The task is a competition. Everyone who submits a solution a solution with at least 65% answer accuracy gets 4 points; the rest 5 points will be distributed depending on relative ordering of your solutions. Note that usually achieving 62% on the dev set is enough to get 65% on the test set (because of multiple references in the test set).

Note that contrary to working with EfficientNet, you need to finetune the RobeCzech model in order to achieve the required accuracy.

You can start with the reading_comprehension.py template, which among others (down)loads the data and the RobeCzech model, and describes the format of the required test set annotations.

vae

 Deadline: Jun 30, 23:59  3 points  Tests

In this assignment you will implement a simple Variational Autoencoder for three datasets in the MNIST format. Your goal is to modify the vae.py template and implement a VAE.

After submitting the assignment to ReCodEx, you can experiment with the three available datasets (mnist, mnist-fashion, and mnist-cifarcars) and different latent variable dimensionality (z_dim=2 and z_dim=100). The generated images are available in TensorBoard logs, and the images generated by the reference solution can be watched here.

Epoch 1/3 reconstruction_loss: 0.5445 - latent_loss: 7.7449 - loss: 306.6298
Epoch 2/3 reconstruction_loss: 0.2925 - latent_loss: 8.9848 - loss: 221.4592
Epoch 3/3 reconstruction_loss: 0.2629 - latent_loss: 3.2707 - loss: 205.3799

Epoch 1/3 reconstruction_loss: 0.4632 - latent_loss: 0.0891 - loss: 246.3138
Epoch 2/3 reconstruction_loss: 0.2757 - latent_loss: 0.0152 - loss: 207.8753
Epoch 3/3 reconstruction_loss: 0.2634 - latent_loss: 0.0058 - loss: 202.1027

gan

 Deadline: Jun 30, 23:59  2 points  Tests

In this assignment you will implement a simple Generative Adversarion Network for three datasets in the MNIST format. Your goal is to modify the gan.py template and implement a GAN.

After submitting the assignment to ReCodEx, you can experiment with the three available datasets (mnist, mnist-fashion, and mnist-cifarcars) and maybe try different latent variable dimensionality. The generated images are available in TensorBoard logs, and the images generated by the reference solution can be watched here.

You can also continue with dcgan assignment.

Epoch 1/5 discriminator_loss: 0.4933 - generator_loss: 3.2423 - loss: 1.2273 - discriminator_accuracy: 0.8306
Epoch 2/5 discriminator_loss: 0.0334 - generator_loss: 4.8349 - loss: 1.6104 - discriminator_accuracy: 1.0000
Epoch 3/5 discriminator_loss: 0.0131 - generator_loss: 5.3357 - loss: 1.7719 - discriminator_accuracy: 1.0000
Epoch 4/5 discriminator_loss: 0.0323 - generator_loss: 5.5625 - loss: 1.8544 - discriminator_accuracy: 0.9980
Epoch 5/5 discriminator_loss: 0.0152 - generator_loss: 6.4666 - loss: 2.1581 - discriminator_accuracy: 0.9990

Epoch 1/5 discriminator_loss: 0.5071 - generator_loss: 2.8793 - loss: 1.1304 - discriminator_accuracy: 0.8286
Epoch 2/5 discriminator_loss: 0.0562 - generator_loss: 3.8137 - loss: 1.2801 - discriminator_accuracy: 1.0000
Epoch 3/5 discriminator_loss: 0.0451 - generator_loss: 4.1467 - loss: 1.3939 - discriminator_accuracy: 1.0000
Epoch 4/5 discriminator_loss: 0.0436 - generator_loss: 5.1381 - loss: 1.7160 - discriminator_accuracy: 0.9980
Epoch 5/5 discriminator_loss: 0.0865 - generator_loss: 4.8655 - loss: 1.6472 - discriminator_accuracy: 0.9918

dcgan

 Deadline: Jun 30, 23:59  1 points  Tests

This task is a continuation of the gan assignment, which you will modify to implement the Deep Convolutional GAN (DCGAN).

Start with the dcgan.py template and implement a DCGAN. Note that most of the TODO notes are from the gan assignment.

After submitting the assignment to ReCodEx, you can experiment with the three available datasets (mnist, mnist-fashion, and mnist-cifarcars). However, note that you will need a lot of computational power (preferably a GPU) to generate the images; the example outputs below were also generated on a GPU, which means the results are nondeterministic. The images generated by the reference solution can be watched here.

Epoch 1/2 discriminator_loss: 2.0305 - generator_loss: 1.1844 - loss: 1.1044 - discriminator_accuracy: 0.5469
Epoch 2/2 discriminator_loss: 1.1953 - generator_loss: 0.9264 - loss: 0.7119 - discriminator_accuracy: 0.7082

Epoch 1/2 discriminator_loss: 1.8414 - generator_loss: 0.9347 - loss: 0.9271 - discriminator_accuracy: 0.5418
Epoch 2/2 discriminator_loss: 1.4439 - generator_loss: 0.9677 - loss: 0.8049 - discriminator_accuracy: 0.5816

crac2022

 Deadline: Evaluation Jun 1-8

If you would like to try participating in a real shared task, right now CRAC 2022 Shared Task on Multilingual Coreference Resolution is running, with the evaluation phase in Jun 1-8.

The goal is to perform coreference resolution on 13 datasets in 10 languages, where coreference resolution is the task of clustering together multiple mentions of the same entity appearing in a textual document (e.g., Joe Biden, the U.S. President, and he).

Note that you should then also write a system description paper – however, it might be possible to send a joint paper, so you would only need to write several paragraphs about your approach.

monte_carlo

 Deadline: Jun 30, 23:59  2 points

Solve the discretized CartPole-v1 environment environment from the Gym library using the Monte Carlo reinforcement learning algorithm. The gym environments have the followng methods and properties:

• observation_space: the description of environment observations
• action_space: the description of environment actions
• reset() → new_state: starts a new episode
• step(action) → new_state, reward, done, info: perform the chosen action in the environment, returning the new state, obtained reward, a boolean flag indicating an end of episode, and additional environment-specific information
• render(): render current environment state

We additionaly extend the gym environment by:

• episode: number of the current episode (zero-based)
• reset(start_evaluation=False) → new_state: if start_evaluation is True, an evaluation is started

Once you finish training (which you indicate by passing start_evaluation=True to reset), your goal is to reach an average return of 475 during 100 evaluation episodes. Note that the environment prints your 100-episode average return each 10 episodes even during training.

Start with the monte_carlo.py template, which parses several useful parameters, creates the environment and illustrates the overall usage.

You will also need the wrappers.py module, which wraps the standard gym API with the above-mentioned added features we use.

During evaluation in ReCodEx, three different random seeds will be employed, and you need to reach the required return on all of them. Time limit for each test is 5 minutes.

reinforce

 Deadline: Jun 30, 23:59  2 points

Solve the continuous CartPole-v1 environment environment from the Gym library using the REINFORCE algorithm. The continuous environment is very similar to the discrete one, except that the states are vectors of real-valued observations with shape env.observation_space.shape.

Your goal is to reach an average return of 475 during 100 evaluation episodes.

Start with the reinforce.py template, which provides a simple network implementation in TensorFlow. However, feel free to use PyTorch instead, if you like.

During evaluation in ReCodEx, two different random seeds will be employed, and you need to reach the required return on all of them. Time limit for each test is 5 minutes.

reinforce_baseline

 Deadline: Jun 30, 23:59  2 points

This is a continuation of the reinforce assignment.

Using the reinforce_baseline.py template, solve the continuous CartPole-v1 environment environment using the REINFORCE with baseline algorithm.

Using a baseline lowers the variance of the value function gradient estimator, which allows faster training and decreases sensitivity to hyperparameter values. To reflect this effect in ReCodEx, note that the evaluation phase will automatically start after 200 episodes. Using only 200 episodes for training in this setting is probably too little for the REINFORCE algorithm, but suffices for the variant with a baseline. In this assignment, you must train your agent in ReCodEx using the provided environment only.

Your goal is to reach an average return of 475 during 100 evaluation episodes.

During evaluation in ReCodEx, two different random seeds will be employed, and you need to reach the required return on all of them. Time limit for each test is 5 minutes.

reinforce_pixels

 Deadline: Jun 30, 23:59  2 points

This is a continuation of the reinforce_baseline assignment.

The supplied cart_pole_pixels_environment.py generates a pixel representation of the CartPole environment as an $80×80$ image with three channels, with each channel representing one time step (i.e., the current observation and the two previous ones).

To pass the assignment, you need to reach an average return of 400 in 100 evaluation episodes. During evaluation in ReCodEx, two different random seeds will be employed, and you need to reach the required return on all of them. Time limit for each test is 10 minutes.

You should probably train the model locally and submit the already pretrained model to ReCodEx.

Start with the reinforce_pixels.py template, which parses several parameters and creates the correct environment.

learning_to_learn

 Deadline: Jun 30, 23:59  4 points  Examples  Tests

Implement a simple variant of learning-to-learn architecture using the learning_to_learn.py template. Utilizing the Omniglot dataset loadable using the omniglot_dataset.py module, the goal is to learn to classify a sequence of images using a custom hierarchy by employing external memory.

The inputs image sequences consists of args.classes random chosen Omniglot classes, each class being assigned a randomly chosen label. For every chosen class, args.images_per_class images are randomly selected. Apart from the images, the input contain the random labels one step after the corresponding images (with the first label being -1). The gold outputs are also the labels, but without the one-step offset.

The input images should be passed through a CNN feature extraction module and then processed using memory augmented LSTM controller; the external memory contains enough memory cells, each with args.cell_size units. In each step, the controller emits:

• args.read_heads read keys, each used to perform a read from memory as a weighted combination of cells according to the softmax of cosine similarities of the read key and the memory cells;
• a write value, which is prepended to the memory (dropping the last cell).

Epoch  1/50 loss: 0.5964 - acc: 0.6422 - acc1: 0.6768 - acc2: 0.6529 - acc5: 0.6360 - acc10: 0.6299 - val_loss: 0.4335 - val_acc: 0.7825 - val_acc1: 0.5195 - val_acc2: 0.7460 - val_acc5: 0.8010 - val_acc10: 0.8725
Epoch  2/50 loss: 0.2660 - acc: 0.8759 - acc1: 0.6137 - acc2: 0.8349 - acc5: 0.9097 - acc10: 0.9272 - val_loss: 0.2693 - val_acc: 0.8744 - val_acc1: 0.6390 - val_acc2: 0.8225 - val_acc5: 0.9135 - val_acc10: 0.9275
Epoch  3/50 loss: 0.1792 - acc: 0.9164 - acc1: 0.6364 - acc2: 0.8870 - acc5: 0.9550 - acc10: 0.9609 - val_loss: 0.2980 - val_acc: 0.8770 - val_acc1: 0.6250 - val_acc2: 0.8460 - val_acc5: 0.9130 - val_acc10: 0.9260
Epoch  4/50 loss: 0.1504 - acc: 0.9296 - acc1: 0.6544 - acc2: 0.9133 - acc5: 0.9643 - acc10: 0.9725 - val_loss: 0.2037 - val_acc: 0.9083 - val_acc1: 0.6390 - val_acc2: 0.8860 - val_acc5: 0.9420 - val_acc10: 0.9555
Epoch  5/50 loss: 0.1326 - acc: 0.9369 - acc1: 0.6741 - acc2: 0.9261 - acc5: 0.9686 - acc10: 0.9772 - val_loss: 0.1829 - val_acc: 0.9168 - val_acc1: 0.6485 - val_acc2: 0.9025 - val_acc5: 0.9560 - val_acc10: 0.9600
Epoch 10/50 loss: 0.0952 - acc: 0.9525 - acc1: 0.6985 - acc2: 0.9542 - acc5: 0.9825 - acc10: 0.9880 - val_loss: 0.1709 - val_acc: 0.9240 - val_acc1: 0.6280 - val_acc2: 0.9140 - val_acc5: 0.9655 - val_acc10: 0.9685
Epoch 20/50 loss: 0.0729 - acc: 0.9613 - acc1: 0.7106 - acc2: 0.9732 - acc5: 0.9916 - acc10: 0.9937 - val_loss: 0.1401 - val_acc: 0.9383 - val_acc1: 0.6845 - val_acc2: 0.9310 - val_acc5: 0.9690 - val_acc10: 0.9805
Epoch 50/50 loss: 0.0579 - acc: 0.9668 - acc1: 0.7243 - acc2: 0.9833 - acc5: 0.9948 - acc10: 0.9961 - val_loss: 0.1271 - val_acc: 0.9444 - val_acc1: 0.7110 - val_acc2: 0.9385 - val_acc5: 0.9760 - val_acc10: 0.9835

Epoch  1/50 loss: 1.5479 - acc: 0.2698 - acc1: 0.3502 - acc2: 0.2777 - acc5: 0.2588 - acc10: 0.2571 - val_loss: 1.4092 - val_acc: 0.3719 - val_acc1: 0.3176 - val_acc2: 0.3430 - val_acc5: 0.3568 - val_acc10: 0.4202
Epoch  2/50 loss: 0.8753 - acc: 0.6209 - acc1: 0.2889 - acc2: 0.4895 - acc5: 0.6703 - acc10: 0.7216 - val_loss: 0.7641 - val_acc: 0.6890 - val_acc1: 0.2538 - val_acc2: 0.5340 - val_acc5: 0.7508 - val_acc10: 0.8050
Epoch  3/50 loss: 0.5346 - acc: 0.7813 - acc1: 0.2553 - acc2: 0.6352 - acc5: 0.8657 - acc10: 0.8919 - val_loss: 0.6430 - val_acc: 0.7511 - val_acc1: 0.2608 - val_acc2: 0.6134 - val_acc5: 0.8286 - val_acc10: 0.8614
Epoch  4/50 loss: 0.4314 - acc: 0.8231 - acc1: 0.2716 - acc2: 0.6970 - acc5: 0.9090 - acc10: 0.9250 - val_loss: 0.5841 - val_acc: 0.7696 - val_acc1: 0.2796 - val_acc2: 0.6414 - val_acc5: 0.8390 - val_acc10: 0.8760
Epoch  5/50 loss: 0.3852 - acc: 0.8410 - acc1: 0.2851 - acc2: 0.7280 - acc5: 0.9260 - acc10: 0.9400 - val_loss: 0.7275 - val_acc: 0.7390 - val_acc1: 0.2836 - val_acc2: 0.6138 - val_acc5: 0.8024 - val_acc10: 0.8456
Epoch 10/50 loss: 0.2885 - acc: 0.8799 - acc1: 0.3195 - acc2: 0.8274 - acc5: 0.9569 - acc10: 0.9656 - val_loss: 0.8520 - val_acc: 0.7335 - val_acc1: 0.2994 - val_acc2: 0.6314 - val_acc5: 0.7852 - val_acc10: 0.8416
Epoch 20/50 loss: 0.2252 - acc: 0.9049 - acc1: 0.3511 - acc2: 0.9009 - acc5: 0.9750 - acc10: 0.9805 - val_loss: 0.5483 - val_acc: 0.8216 - val_acc1: 0.3182 - val_acc2: 0.7828 - val_acc5: 0.8828 - val_acc10: 0.9152
Epoch 50/50 loss: 0.1720 - acc: 0.9233 - acc1: 0.3859 - acc2: 0.9518 - acc5: 0.9870 - acc10: 0.9895 - val_loss: 0.5175 - val_acc: 0.8478 - val_acc1: 0.3636 - val_acc2: 0.8288 - val_acc5: 0.9006 - val_acc10: 0.9324

Epoch 1/3 loss: 0.7764 - acc: 0.5078 - acc1: 0.5375 - acc2: 0.5063 - acc5: 0.5031 - acc10: 0.5000 - val_loss: 0.6923 - val_acc: 0.5175 - val_acc1: 0.7531 - val_acc2: 0.5688 - val_acc5: 0.4500 - val_acc10: 0.4969
Epoch 2/3 loss: 0.6992 - acc: 0.5034 - acc1: 0.5250 - acc2: 0.5031 - acc5: 0.4906 - acc10: 0.5063 - val_loss: 0.6914 - val_acc: 0.5397 - val_acc1: 0.7469 - val_acc2: 0.5844 - val_acc5: 0.5031 - val_acc10: 0.4875
Epoch 3/3 loss: 0.6969 - acc: 0.4975 - acc1: 0.5594 - acc2: 0.5063 - acc5: 0.4844 - acc10: 0.5094 - val_loss: 0.6907 - val_acc: 0.5272 - val_acc1: 0.6781 - val_acc2: 0.5312 - val_acc5: 0.5219 - val_acc10: 0.5000

Epoch 1/3 loss: 1.6505 - acc: 0.2004 - acc1: 0.1937 - acc2: 0.2025 - acc5: 0.2050 - acc10: 0.2087 - val_loss: 1.6086 - val_acc: 0.2075 - val_acc1: 0.2837 - val_acc2: 0.2325 - val_acc5: 0.1900 - val_acc10: 0.1900
Epoch 2/3 loss: 1.6146 - acc: 0.2042 - acc1: 0.2237 - acc2: 0.1912 - acc5: 0.1950 - acc10: 0.2138 - val_loss: 1.6075 - val_acc: 0.2156 - val_acc1: 0.3050 - val_acc2: 0.2325 - val_acc5: 0.1912 - val_acc10: 0.2100
Epoch 3/3 loss: 1.6114 - acc: 0.2031 - acc1: 0.2275 - acc2: 0.2138 - acc5: 0.1838 - acc10: 0.1912 - val_loss: 1.6061 - val_acc: 0.2261 - val_acc1: 0.3363 - val_acc2: 0.2387 - val_acc5: 0.2163 - val_acc10: 0.2013

In the competitions, your goal is to train a model and then predict target values on the given unannotated test set.

Submitting to ReCodEx

When submitting a competition solution to ReCodEx, you can include any number of files of any kind, and either submit them individually or compess them in a .zip file. However, there should be exactly one text file with the test set annotation (.txt) and at least one Python source (.py/ipynb) containing the model training and prediction. The Python sources are not executed, but must be included for inspection.

Competition Evaluation

• For every submission, ReCodEx checks the above conditions (exactly one .txt, at least one .py/ipynb) and whether the given annotations can be evaluated without error. If not, it will report a corresponding error in the logs.

• Before the deadline, ReCodEx prints the exact achieved score, but only if it is worse than the baseline.

  If you surpass the baseline, the assignment is marked as solved in ReCodEx and you immediately get regular points for the assignment. However, ReCodEx does not print the reached score.

• After the competition deadline, the latest submission of every user surpassing the required baseline participates in a competition. Additional bonus points are then awarded according to the ordering of the performance of the participating submissions.

• After the competition results announcement, ReCodEx starts to show the exact performance for all the already submitted solutions and also for the solutions submitted later.

What Is Allowed

• You can use only the given annotated data for training and evaluation.
• You can use the given annotated training data in any way.
• You can use the given annotated development data for evaluation or hyperparameter tuning, but not for the training itself.
• Additionally, you can use any unannotated or manually created data for training and evaluation.
• The test set annotations must be the result of your system (so you cannot manually correct them; but your system can contain other parts than just trained models, like hand-written rules).
• Do not use test set annotations in any way, if you somehow get access to them.
• Unless stated otherwise, you can use any algorithm to solve the competition task at hand. The implementation should be either created by you or it can be based on some publicly available implementation, in which case you must reference it and you must understand it fully.
• If you utilize an already trained model, it must be trained only on the allowed training data, unless stated otherwise.

Install

• Installing to central user packages repository

  You can install all required packages to central user packages repository using pip3 install --user tensorflow==2.8.0 tensorflow-addons==0.16.1 tensorflow-probability==0.16.0 tensorflow-hub==0.12.0 gym==0.20.0 scipy transformers==4.18.0 protobuf~=3.20.1.

• Installing to a virtual environment

  Python supports virtual environments, which are directories containing independent sets of installed packages. You can create a virtual environment by running python3 -m venv VENV_DIR followed by VENV_DIR/bin/pip3 install tensorflow==2.8.0 tensorflow-addons==0.16.1 tensorflow-probability==0.16.0 tensorflow-hub==0.12.0 gym==0.20.0 scipy transformers==4.18.0 protobuf~=3.20.1 (or VENV_DIR/Scripts/pip3 on Windows).

• Windows installation

  • On Windows, it can happen that python3 is not in PATH, while py command is – in that case you can use py -m venv VENV_DIR, which uses the newest Python available, or for example py -3.9 -m venv VENV_DIR, which uses Python version 3.9.

  • If your Windows TensorFlow fails with ImportError: DLL load failed, you are probably missing Visual C++ 2019 Redistributable.

  • If you encounter a problem creating the logs in the args.logdir directory, a possible cause is that the path is longer than 260 characters, which is the default maximum length of a complete path on Windows. However, you can increase this limit on Windows 10, version 1607 or later, by following the instructions.

• macOS installation

  • With an Intel processor, you should not need anything special.

  • If you have Apple Silicon, the installation is a bit more involved, because some Python packages do not yet have an official Arm64 binary build. The easiest workaround is to use Conda, which contains all the required dependencies.

    • Download Conda env.

    • Install it and activate it.

      chmod +x ~/Downloads/Miniforge3-MacOSX-arm64.sh
      sh ~/Downloads/Miniforge3-MacOSX-arm64.sh
      source ~/miniforge3/bin/activate
      

    • Install the Arm64 TensorFlow dependencies.

      conda install -c apple tensorflow-deps==2.8.0
      

    • Install the Arm64 build of TensorFlow.

      python -m pip install tensorflow-macos==2.8.0
      

• GPU support on Linux and Windows

  TensorFlow 2.8 supports NVIDIA GPU out of the box, but you need to install CUDA 11.2 and cuDNN 8.1 libraries yourself.

• GPU support on macOS

  The AMD and Apple Silicon GPUs can be used by installing a plugin providing the GPU acceleration using:

  python -m pip install tensorflow-metal
  

• Errors when running with a GPU

  If you encounter errors when running with a GPU:

  • if you are using the GPU also for displaying, try using the following environment variable: export TF_FORCE_GPU_ALLOW_GROWTH=true
  • you can rerun with export TF_CPP_MIN_LOG_LEVEL=0 environmental variable, which increases verbosity of the log messages.

MetaCentrum

• How to install TensorFlow dependencies on MetaCentrum?

  To install CUDA, cuDNN and Python 3.8 on MetaCentrum, it is enough to run in every session the following command:

  module add python/3.8.0-gcc-rab6t cuda/cuda-11.2.0-intel-19.0.4-tn4edsz cudnn/cudnn-8.1.0.77-11.2-linux-x64-intel-19.0.4-wx22b5t
  

• How to install TensorFlow on MetaCentrum?

  Once you have the required dependencies, you can create a virtual environment and install TensorFlow in it. However, note that by default the MetaCentrum jobs have a little disk space, so read about how to ask for scratch storage when submitting a job, and about quotas,

  TL;DR:

  • Run an interactive CPU job, asking for 16GB scratch space:

    qsub -l select=1:ncpus=1:mem=8gb:scratch_local=16gb -I
    

  • In the job, use the allocated scratch space as a temporary directory:

    export TMPDIR=$SCRATCHDIR
    

  • Finally, create the virtual environment and install TensorFlow in it:

    module add python/3.8.0-gcc-rab6t cuda/cuda-11.2.0-intel-19.0.4-tn4edsz cudnn/cudnn-8.1.0.77-11.2-linux-x64-intel-19.0.4-wx22b5t
    python3 -m venv CHOSEN_VENV_DIR
    CHOSEN_VENV_DIR/bin/pip install --no-cache-dir --upgrade pip setuptools
    CHOSEN_VENV_DIR/bin/pip install --no-cache-dir tensorflow==2.8.0 tensorflow-addons==0.16.1 tensorflow-probability==0.16.0 tensorflow-hub==0.12.0 gym==0.20.0 scipy
    

• How to run a GPU computation on MetaCentrum?

  First, read the official MetaCentrum documentation: Beginners guide, About scheduling system, GPU clusters.

  TL;DR: To run an interactive GPU job with 1 CPU, 1 GPU, 16GB RAM, and 8GB scatch space, run:

  qsub -q gpu -l select=1:ncpus=1:ngpus=1:mem=16gb:scratch_local=8gb -I
  

  To run a script in a non-interactive way, replace the -I option with the script to be executed.

  If you want to run a CPU-only computation, remove the -q gpu and ngpus=1: from the above commands.

AIC

• How to install TensorFlow dependencies on AIC?

  To install CUDA, cuDNN and Python 3.9 on AIC, you should add the following to your .profile:

  export PATH="/lnet/aic/data/python/3.9.9/bin:$PATH"
  export LD_LIBRARY_PATH="/lnet/aic/opt/cuda/cuda-11.2/lib64:/lnet/aic/opt/cuda/cuda-11.2/cudnn/8.1.1/lib64:/lnet/aic/opt/cuda/cuda-11.2/extras/CUPTI/lib64:$LD_LIBRARY_PATH"
  

• How to run a GPU computation on AIC?

  First, read the official AIC documentation: Submitting CPU Jobs, Submitting GPU Jobs.

  TL;DR: To run an interactive GPU job with 1 CPU, 1 GPU, and 16GB RAM, run:

  qrsh -q gpu.q -l gpu=1,mem_free=16G,h_data=16G -pty yes bash -l
  

  To run a script requiring a GPU in a non-interactive way, use

  qsub -q gpu.q -l gpu=1,mem_free=16G,h_data=16G -cwd -b y SCRIPT_PATH
  

  If you want to run a CPU-only computation, remove the -q gpu.q and gpu=1, from the above commands.

Git

• Is it possible to keep the solutions in a Git repository?

  Definitely. Keeping the solutions in a branch of your repository, where you merge them with the course repository, is probably a good idea. However, please keep the cloned repository with your solutions private.

• On GitHub, do not create a public fork with your solutions

  If you keep your solutions in a GitHub repository, please do not create a clone of the repository by using the Fork button – this way, the cloned repository would be public.

  Of course, if you just want to create a pull request, GitHub requires a public fork and that is fine – just do not store your solutions in it.

• How to clone the course repository?

  To clone the course repository, run

  git clone https://github.com/ufal/npfl114
  

  This creates the repository in the npfl114 subdirectory; if you want a different name, add it as a last parameter.

  To update the repository, run git pull inside the repository directory.

• How to keep the course repository as a branch in your repository?

  If you want to store the course repository just in a local branch of your existing repository, you can run the following command while in it:

  git remote add upstream https://github.com/ufal/npfl114
  git fetch upstream
  git checkout -t upstream/master
  

  This creates a branch master; if you want a different name, add -b BRANCH_NAME to the last command.

  In both cases, you can update your checkout by running git pull while in it.

• How to merge the course repository with your modifications?

  If you want to store your solutions in a branch merged with the course repository, you should start by

  git remote add upstream https://github.com/ufal/npfl114
  git pull upstream master
  

  which creates a branch master; if you want a different name, change the last argument to master:BRANCH_NAME.

  You can then commit to this branch and push it to your repository.

  To merge the current course repository with your branch, run

  git merge upstream master
  

  while in your branch. Of course, it might be necessary to resolve conflicts if both you and I modified the same place in the templates.

ReCodEx

• What files can be submitted to ReCodEx?

  You can submit multiple files of any type to ReCodEx. There is a limit of 20 files per submission, with a total size of 20MB.

• What file does ReCodEx execute and what arguments does it use?

  Exactly one file with py suffix must contain a line starting with def main(. Such a file is imported by ReCodEx and the main method is executed (during the import, __name__ == "__recodex__").

  The file must also export an argument parser called parser. ReCodEx uses its arguments and default values, but it overwrites some of the arguments depending on the test being executed – the template should always indicate which arguments are set by ReCodEx and which are left intact.

• What are the time and memory limits?

  The memory limit during evaluation is 1.5GB. The time limit varies, but it should be at least 10 seconds and at least twice the running time of my solution.

Tensors

• How to work with the usual tf.Tensors?

  Read the TensorFlow Tensor guide and also the TensorFlow tensor indexing guide.

• How to work with the tf.RaggedTensors?

  Read the TensorFlow RaggedTensor guide.

• How to convert the tf.RaggedTensor to a tf.Tensor and back?

  Often, you might want to convert a tf.RaggedTensor to a tf.Tensor and then back.

  • To obtain just the valid elements (so the rank of the resulting tf.Tensor is smaller by one):

    tensor_with_valid_elements = ragged_tensor.values
    ...
    new_ragged_tensor = ragged_tensor.with_values(new_tensor_with_valid_elements)
    

  • To obtain a tf.Tensor with the corresponding shape (so padding elements are added where needed):

    tensor_with_padding = ragged_tensor.to_tensor()
    ...
    new_ragged_tensor = tf.RaggedTensor.from_tensor(new_tensor_with_padding, ragged_tensor.row_lengths())
    

tf.data

• How to look what is in a tf.data.Dataset?

  The tf.data.Dataset is not just an array, but a description of a pipeline, which can produce data if requested. A simple way to run the pipeline is to iterate it using Python iterators:

  dataset = tf.data.Dataset.range(10)
  for entry in dataset:
      print(entry)
  

• How to use tf.data.Dataset with model.fit or model.evaluate?

  To use a tf.data.Dataset in Keras, the dataset elements should be pairs (input_data, gold_labels), where input_data and gold_labels must be already batched. For example, given CAGS dataset, you can preprocess training data for cags_classification as (for development data, you would remove the .shuffle):

  train = cags.train.map(lambda example: (example["image"], example["label"]))
  train = train.shuffle(10000, seed=args.seed)
  train = train.batch(args.batch_size)
  

• Is every iteration through a tf.data.Dataset the same?

  No. Because the dataset is only a pipeline generating data, it is called each time the dataset is iterated – therefore, every .shuffle is called in every iteration.

• How to generate different random numbers each epoch during tf.data.Dataset.map?

  When a global random seed is set, methods like tf.random.uniform generate the same sequence of numbers on each iteration.

  Instead, create a Generator object and use it to produce random numbers.

  generator = tf.random.Generator.from_seed(42)
  data = tf.data.Dataset.from_tensor_slices(tf.zeros(10, tf.int32))
  data = data.map(lambda x: x + generator.uniform([], maxval=10, dtype=tf.int32))
  for _ in range(3):
      print(*[element.numpy() for element in data])
  

  When a GPU is visible, you should create the generator explicitly on a CPU using a with tf.device("/cpu:0"): block (on macOS, it will crash otherwise).

• How to call numpy methods or other non-tf functions in tf.data.Dataset.map?

  You can use tf.numpy_function to call a numpy function even in a computational graph. However, the results have no static shape information and you need to set it manually – ideally using tf.ensure_shape, which both sets the static shape and verifies during execution that the real shape matches it.

  For example, to use the bboxes_training method from bboxes_utils, you could proceed as follows:

  anchors = np.array(...)
  
  def prepare_data(example):
      anchor_classes, anchor_bboxes = tf.numpy_function(
          bboxes_utils.bboxes_training, [anchors, example["classes"], example["bboxes"], 0.5], (tf.int32, tf.float32))
      anchor_classes = tf.ensure_shape(anchor_classes, [len(anchors)])
      anchor_bboxes = tf.ensure_shape(anchor_bboxes, [len(anchors), 4])
      ...
  

• How to use ImageDataGenerator in tf.data.Dataset.map?

  The ImageDataGenerator offers a .random_transform method, so we can use tf.numpy_function from the previous answer:

  train_generator = tf.keras.preprocessing.image.ImageDataGenerator(...)
  
  def augment(image, label):
      return tf.ensure_shape(
          tf.numpy_function(train_generator.random_transform, [image], tf.float32),
          image.shape
      ), label
  dataset.map(augment)
  

Debugging

• How to debug problems “inside” computation graphs with weird stack traces?

  At the beginning of your program, run

  tf.config.run_functions_eagerly(True)
  

  The tf.funcions (with the exception of the ones used in tf.data pipelines) are then not traced (i.e., no computation graphs are created) and the pure Python code is executed instead.

• How to debug problems “inside” tf.data pipelines with weird stack traces?

  Unfortunately, the solution above does not affect tracing in tf.data pipelines (for example in tf.data.Dataset.map). However, since TF 2.5, the command

  tf.data.experimental.enable_debug_mode()
  

  should disable any asynchrony, parallelism, or non-determinism and forces Python execution (as opposed to trace-compiled graph execution) of user-defined functions passed into transformations such as tf.data.Dataset.map.

Finetuning

• How to make a part of the network frozen, so that its weights are not updated?

  Each tf.keras.layers.Layer/tf.keras.Model has a mutable trainable property indicating whether its variables should be updated – however, after changing it, you need to call .compile again (or otherwise make sure the list of trainable variables for the optimizer is updated).

  Note that once trainable == False, the insides of a layer are no longer considered, even if some its sub-layers have trainable == True. Therefore, if you want to freeze only some sub-layers of a layer you use in your model, the layer itself must have trainable == True.

• How to choose whether dropout/batch normalization is executed in training or inference regime?

  When calling a tf.keras.layers.Layer/tf.keras.Model, a named option training can be specified, indicating whether training or inference regime should be used. For a model, this option is automatically passed to its layers which require it, and Keras automatically passes it during model.{fit,evaluate,predict}.

  However, you can manually pass for example training=False to a layer when using Functional API, meaning that layer is executed in the inference regime even when the whole model is training.

• How does trainable and training interact?

  The only layer, which is influenced by both these options, is batch normalization, for which:

  • if trainable == False, the layer is always executed in inference regime;
  • if trainable == True, the training/inference regime is chosen according to the training option.

• How to use linear warmup?

  You can prepend any following_schedule by using the following LinearWarmup schedule:

  class LinearWarmup(tf.optimizers.schedules.LearningRateSchedule):
      def __init__(self, warmup_steps, following_schedule):
          self._warmup_steps = warmup_steps
          self._warmup = tf.optimizers.schedules.PolynomialDecay(0., warmup_steps, following_schedule(0))
          self._following = following_schedule
  
      def __call__(self, step):
          return tf.cond(step < self._warmup_steps,
                         lambda: self._warmup(step),
                         lambda: self._following(step - self._warmup_steps))
  

TensorBoard

• Cannot start TensorBoard after installation

  If tensorboard executable cannot be found, make sure the directory with pip installed packages is in your PATH (that directory is either in your virtual environment if you use a virtual environment, or it should be ~/.local/bin on Linux and %UserProfile%\AppData\Roaming\Python\Python3[7-9] and %UserProfile%\AppData\Roaming\Python\Python3[7-9]\Scripts on Windows).

• How to create TensorBoard logs manually?

  Start by creating a SummaryWriter using for example:

  writer = tf.summary.create_file_writer(args.logdir, flush_millis=10 * 1000)
  

  and then you can generate logs inside a with writer.as_default() block.

  You can either specify step manually in each call, or you can set it as the first argument of as_default(). Also, during training you usually want to log only some batches, so the logging block during training usually looks like:

  if optimizer.iterations % 100 == 0:
      with self._writer.as_default(step=optimizer.iterations):
          # logging
  

• What can be logged in TensorBoard?

  • scalar values:

    tf.summary.scalar(name like "train/loss", value, [step])
    

  • tensor values displayed as histograms or distributions:

    tf.summary.histogram(name like "train/output_layer", tensor value castable to `tf.float64`, [step])
    

  • images as tensors with shape [num_images, h, w, channels], where channels can be 1 (grayscale), 2 (grayscale + alpha), 3 (RGB), 4 (RGBA):

    tf.summary.image(name like "train/samples", images, [step], [max_outputs=at most this many images])
    

  • possibly large amount of text (e.g., all hyperparameter values, sample translations in MT, …) in Markdown format:

    tf.summary.text(name like "hyperparameters", markdown, [step])
    

  • audio as tensors with shape [num_clips, samples, channels] and values in $[-1,1]$ range:

    tf.summary.audio(name like "train/samples", clips, sample_rate, [step], [max_outputs=at most this many clips])
    

Requirements

To pass the practicals, you need to obtain at least 80 points, excluding the bonus points. Note that all surplus points (both bonus and non-bonus) will be transfered to the exam. In total, assignments for at least 120 points (not including the bonus points) will be available, and if you solve all the assignments (any non-zero amount of points counts as solved), you automatically pass the exam with grade 1.

To pass the exam, you need to obtain at least 60, 75, or 90 points out of 100-point exam to receive a grade 3, 2, or 1, respectively. The exam consists of 100-point-worth questions from the list below (the questions are randomly generated, but in such a way that there is at least one question from every but the first lecture). In addition, you can get surplus points from the practicals and at most 10 points for community work (i.e., fixing slides or reporting issues) – but only the points you already have at the time of the exam count. You can take the exam without passing the practicals first.

Exam Questions

Lecture 1 Questions

• Considering a neural network with $D$ input neurons, a single hidden layer with $H$ neurons, $K$ output neurons, hidden activation $f$ and output activation $a$, list its parameters (including their size) and write down how the output is computed. [5]

• List the definitions of frequently used MLP output layer activations (the ones producing parameters of a Bernoulli distribution and a categorical distribution). Then write down three commonly used hidden layer activations (sigmoid, tanh, ReLU). [5]

• Formulate the Universal approximation theorem. [5]

Lecture 2 Questions

• Describe maximum likelihood estimation, as minimizing NLL, cross-entropy and KL divergence. [10]

• Define mean squared error and show how it can be derived using MLE. [5]

• Describe gradient descent and compare it to stochastic (i.e., online) gradient descent and minibatch stochastic gradient descent. [5]

• Formulate conditions on the sequence of learning rates used in SGD to converge to optimum almost surely. [5]

• Write down the backpropagation algorithm. [5]

• Write down the mini-batch SGD algorithm with momentum. Then, formulate SGD with Nesterov momentum and show the difference between them. [5]

• Write down the AdaGrad algorithm and show that it tends to internally decay learning rate by a factor of $1/\sqrt{t}$ in step $t$. Then write down the RMSProp algorithm and explain how it solves the problem with the involuntary learning rate decay. [10]

• Write down the Adam algorithm. Then show why the bias-correction terms $(1-\beta^t)$ make the estimation of the first and second moment unbiased. [10]

Lecture 3 Questions

• Considering a neural network with $D$ input neurons, a single ReLU hidden layer with $H$ units and softmax output layer with $K$ units, write down the explicit formulas of the gradient of all the MLP parameters (two weight matrices and two bias vectors), assuming input $\boldsymbol x$, target $g$ and negative log likelihood loss. [10]

• Assume a network with MSE loss generated a single output $o \in \mathbb{R}$, and the target output is $g$. What is the value of the loss function itself, and what is the explicit formula of the gradient of the loss function with respect to $o$? [5]

• Assume a binary-classification network with cross-entropy loss generated a single output $z \in \mathbb{R}$, which is passed through the sigmoid output activation function, producing $o = \sigma(z)$. If the target output is $g$, what is the value of the loss function itself, and what is the explicit formula of the gradient of the loss function with respect to $z$? [5]

• Assume a $K$-class-classification network with cross-entropy loss generated a $K$-element output $\boldsymbol z \in \mathbb{R}^K$, which is passed through the softmax output activation function, producing $\boldsymbol o=\operatorname{softmax}(\boldsymbol z)$. If the target distribution is $\boldsymbol g$, what is the value of the loss function itself, and what is the explicit formula of the gradient of the loss function with respect to $\boldsymbol z$? [5]

• Define $L_2$ regularization and describe its effect both on the value of the loss function and on the value of the loss function gradient. [5]

• Describe the dropout method and write down exactly how it is used during training and during inference. [5]

• Describe how label smoothing works for cross-entropy loss, both for sigmoid and softmax activations. [5]

• How are weights and biases initialized using the default Glorot initialization? [5]

Lecture 4 Questions

• Write down the equation of how convolution of a given image is computed. Assume the input is an image $I$ of size $H \times W$ with $C$ channels, the kernel $K$ has size $N \times M$, the stride is $T \times S$, the operation performed is in fact cross-correlation (as usual in convolutional neural networks) and that $O$ output channels are computed. [5]

• Explain both SAME and VALID padding schemes and write down the output size of a convolutional operation with an $N \times M$ kernel on image of size $H \times W$ for both these padding schemes (stride is 1). [5]

• Describe batch normalization including all its parameters, and write down an algorithm how it is used during training and an algorithm how it is used during inference. Be sure to explicitly write over what is being normalized in case of fully connected layers and in case of convolutional layers. [10]

• Describe overall architecture of VGG-19 (you do not need to remember the exact number of layers/filters, but you should describe which layers are used). [5]

Lecture 5 Questions

• Describe overall architecture of ResNet. You do not need to remember the exact number of layers/filters, but you should draw a bottleneck block (including the applications of BatchNorms and ReLUs) and state how residual connections work when the number of channels increases. [10]

• Draw the original ResNet block (including the exact positions of BatchNorms and ReLUs) and also the improved variant with full pre-activation. [5]

• Compare the bottleneck block of ResNet and ResNeXt architectures (draw the latter using convolutions only, i.e., do not use grouped convolutions). [5]

• Describe the CNN regularization method of networks with stochastic depth. [5]

• Compare Cutout and DropBlock. [5]

• Describe Squeeze and Excitation applied to a ResNet block. [5]

• Draw the Mobile inverted bottleneck block (including explanation of separable convolutions, the expansion factor, exact positions of BatchNorms and ReLUs, but without describing Squeeze and excitation blocks). [5]

• Assume an input image $I$ of size $H \times W$ with $C$ channels, and a convolutional kernel $K$ with size $N \times M$, stride $S$ and $O$ output channels. Write down (or derive) the equation of transposed convolution (or equivalently backpropagation through a convolution to its inputs). [5]

Lecture 6 Questions

• Write down how $\mathit{AP}_{50}$ is computed. [5]

• Considering a Fast-RCNN architecture, draw overall network architecture, explain what a RoI-pooling layer is, show how the network parametrizes bounding boxes and write down the loss. Finally, describe non-maximum suppression and how the Fast-RCNN prediction is performed. [10]

• Considering a Faster-RCNN architecture, describe the region proposal network (what are anchors, architecture including both heads, how are the coordinates of proposals parametrized, what does the loss look like). [10]

• Considering Mask-RCNN architecture, describe the additions to a Faster-RCNN architecture (the RoI-Align layer, the new mask-producing head). [5]

• Write down the focal loss with class weighting, including the commonly used hyperparameter values. [5]

• Draw the overall architecture of a RetinaNet architecture (the FPN architecture including the block combining feature maps of different resolutions; the classification and bounding box generation heads, including their output size). [5]

Lecture 7 Questions

• Write down how the Long Short-Term Memory (LSTM) cell operates, including the explicit formulas. Also mention the forget gate bias. [10]

• Write down how the Gated Recurrent Unit (GRU) operates, including the explicit formulas. [10]

• Describe Highway network computation. [5]

• Why the usual dropout cannot be used on recurrent state? Describe how the problem can be alleviated with variational dropout. [5]

• Describe layer normalization including all its parameters, and write down how it is computed (be sure to explicitly state over what is being normalized in case of fully connected layers and convolutional layers). [5]

• Sketch a tagger architecture utilizing word embeddings, recurrent character-level word embeddings and two sentence-level bidirectional RNNs with a residual connection. [10]

Lecture 8 Questions

• Considering a linear-chain CRF, write down how a score of a label sequence $\boldsymbol y$ is defined, and how can a log probability be computed using the label sequence scores. [5]

• Write down the dynamic programming algorithm for computing log probability of a linear-chain CRF, including its asymptotic complexity. [10]

• Write down the dynamic programming algorithm for linear-chain CRF decoding, i.e., an algorithm computing the most probable label sequence $\boldsymbol y$. [10]

• In the context of CTC loss, describe regular and extended labelings and write down an algorithm for computing the log probability of a gold label sequence $\boldsymbol y$. [10]

• Describe how CTC predictions are performed using a beam-search. [5]

• Draw the CBOW architecture from word2vec, including the sizes of the inputs and the sizes of the outputs and used non-linearities. Also make sure to indicate where the embeddings are being trained. [5]

• Draw the SkipGram architecture from word2vec, including the sizes of the inputs and the sizes of the outputs and used non-linearities. Also make sure to indicate where the embeddings are being trained. [5]

• Describe the hierarchical softmax used in word2vec. [5]

• Describe the negative sampling proposed in word2vec, including the choice of distribution of negative samples. [5]

Lecture 9 Questions

• Draw a sequence-to-sequence architecture for machine translation, both during training and during inference (without attention). [5]

• Draw a sequence-to-sequence architecture for machine translation used during training, including the attention. Then write down how exactly is the attention computed. [10]

• Explain how can word embeddings tying be used in a sequence-to-sequence architecture. [5]

• Write down why are subword units used in text processing, and describe the BPE algorithm for constructing a subword dictionary from a large corpus. [5]

• Write down why are subword units used in text processing, and describe the WordPieces algorithm for constructing a subword dictionary from a large corpus. [5]

• Pinpoint the differences between the BPE and WordPieces algorithms, both during dictionary construction and during inference. [5]

Lecture 11 Questions

• Describe the Transformer encoder architecture, including the description of self-attention (but you do not need to describe multi-head attention), FFN and positions of LNs and dropouts. [10]

• Write down the formula of Transformer self-attention, and then describe multi-head self-attention in detail. [10]

• Describe the Transformer decoder architecture, including the description of self-attention and masked self-attention (but you do not need to describe multi-head attention), FFN and positions of LNs and dropouts. Also discuss the difference between training and prediction regimes. [10]

• Why are positional embeddings needed in Transformer architecture? Write down the sinusoidal positional embeddings used in the Transformer. [5]

• Compare RNN to Transformer – what are the strengths and weaknesses of these architectures? [5]

• Explain how are ELMo embeddings trained and how are they used in downstream applications. [5]

• Describe the BERT architecture (you do not need to describe the (multi-head) self-attention operation). Elaborate also on what positional embeddings are used and what are the GELU activations. [10]

• Describe the GELU activations and explain why are they a combination of ReLUs and Dropout. [5]

• Elaborate on BERT training process (what are the two objectives used and how exactly are the corresponding losses computed). [10]

Lecture 12 Questions

• Write down the variational lower bound (ELBO) in the form of a reconstruction error minus the KL divergence between the encoder and the prior. Then prove it is actually a lower bound on probability $\log P(\boldsymbol x)$ (you can use Jensen's inequality if you want). [10]

• Draw an architecture of a variational autoencoder (VAE). Pay attention to the parametrization of the distribution from the encoder (including the used activation functions), and show how to perform latent variable sampling so that it is differentiable with respect to the encoder parameters (the reparametrization trick). [10]

• Write down the min-max formulation of generative adversarial network (GAN) objective. Then describe what loss is actually used for training the generator in order to avoid vanishing gradients at the beginning of the training. [5]

• Write down the training algorithm of generative adversarial networks (GAN), including the losses minimized by the discriminator and the generator. Be sure to use the version of generator loss which avoids vanishing gradients at the beginning of the training. [10]

• Explain how the class label is used when training a conditional generative adversarial network (CGAN). [5]

• Illustrate that alternating SGD steps are not guaranteed to converge for a min-max problem. [5]

Lecture 13 Questions

• Show how to incrementally update a running average (how to compute an average of $N$ numbers using the average of the first $N-1$ numbers). [5]

• Describe multi-arm bandits and write down the $\epsilon$-greedy algorithm for solving it. [5]

• Define the Markov Decision Process, including the definition of the return. [5]

• Define the value function, such that all expectations are over simple random variables (actions, states, rewards), not trajectories. [5]

• Define the action-value function, such that all expectations are over simple random variables (actions, states, rewards), not trajectories. [5]

• Express the value function using the action-value function, and express the action-value function using the value function. [5]

• Define the optimal value function and the optimal action-value function. Then define optimal policy in such a way that its existence is guaranteed. [5]

• Write down the Monte-Carlo on-policy every-visit $\epsilon$-soft algorithm. [10]

• Formulate the policy gradient theorem. [5]

• Prove the part of the policy gradient theorem showing the value of $\nabla_{\boldsymbol\theta} v_\pi(s)$. [10]

• Assuming the policy gradient theorem, formulate the loss used by the REINFORCE algorithm and show how can its gradient be expressed as an expectation over states and actions. [5]

• Write down the REINFORCE algorithm. [10]

• Show that introducing baseline does not influence validity of the policy gradient theorem. [5]

• Write down the REINFORCE with baseline algorithm. [10]

Lecture 14 Questions

• Sketch the overall structure and training procedure of the Neural Architecture Search. You do not need to describe how exactly is the block produced by the controller. [5]

• Draw the WaveNet architecture (show the overall architecture, explain dilated convolutions, write down the gated activations, describe global and local conditioning). [10]

• Define the Mixture of Logistic distribution used in the Teacher model of Parallel WaveNet, including the explicit formula of computing the likelihood of the data. [5]

• Describe the changes in the Student model of Parallel WaveNet, which allow efficient sampling (how does the latent prior look like, how the output data distribution is modeled in a single iteration and then after multiple iterations). [5]

• Describe the addressing mechanism used in Neural Turing Machines – show the overall structure including the required parameters, and explain content addressing, interpolation with location addressing, shifting and sharpening. [10]

• Explain the overall architecture of a Neural Turing Machine with an LSTM controller, assuming $R$ reading heads and one write head. Describe the inputs and outputs of the LSTM controller itself, then how the memory is read from and written to, and how the final output is computed. You do not need to write down the implementation of the addressing mechanism (you can assume it is a function which gets parameters, memory and previous distribution, and computes a new distribution over memory cells). [10]

Related Courses

Machine Learning for Greenhorns

Introductory course to machine learning, focusing both on theoretical foundations as well as on practical applications in Python.

Deep Reinforcement Learning

Course introducing reinforcement learning, from basic tabular methods to involvement of deep neural networks, focusing both on theory as well as on practical aspects.

Archive

Summer 2020/21

Summer 2019/20

Summer 2018/19

Summer 2017/18

Winter 2016/17