Deep Learning Seminar, Winter 2019/20

In recent years, deep neural networks have been used to solve complex machine-learning problems and have achieved significant state-of-the-art results in many areas. The whole field of deep learning has been developing rapidly, with new methods and techniques emerging steadily.

The goal of the seminar is to follow the newest advancements in the deep learning field. The course takes form of a reading group – each lecture a paper is presented by one of the students. The paper is announced in advance, hence all participants can read it beforehand and can take part in the discussion of the paper.

If you want to receive announcements about chosen paper, sign up to our mailing list ufal-rg@googlegroups.com.

About

SIS code: NPFL117
Semester: winter + summer
E-credits: 3
Examination: 0/2 C
Guarantor: Milan Straka

Timespace Coordinates

The Deep Learning Seminar takes place on Monday at 12:20 in S10. We will first meet on Monday Oct 07.

Requirements

To pass the course, you need to present a research paper and sufficiently attend the presentations.

License

Unless otherwise stated, teaching materials for this course are available under CC BY-SA 4.0.

If you want to receive announcements about chosen paper, sign up to our mailing list ufal-rg@googlegroups.com.

To add your name to a paper the table below, edit the source code on GitHub and send a PR.

Date Who Topic Paper(s)
07 Oct 2019 Milan Straka Transformer Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin: Attention Is All You Need
Peter Shaw, Jakob Uszkoreit, Ashish Vaswani: Self-Attention with Relative Position Representations
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, Douglas Eck: Music Transformer
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov: Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
14 Oct 2019 Ondřej Měkota Transformer Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le. XLNet: Generalized Autoregressive Pretraining for Language Understanding
21 Oct 2019 Tomas Soucek 3D Pointclouds Charles R. Qi, Li Yi, Hao Su, Leonidas J. Guibas: PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
Christopher Choy, JunYoung Gwak, Silvio Savarese: 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks
Christopher Choy, Jaesik Park, Vladlen Koltun: Fully Convolutional Geometric Features
28 Oct 2019 No DL seminar Czech Independence Day
04 Nov 2019 Zdeněk Kasner Neural LMs Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners (OpenAI blog post)
Subramanian, Sandeep, Raymond Li, Jonathan Pilault, and Christopher Pal. On Extractive and Abstractive Neural Document Summarization with Transformer Language Models
11 Nov 2019 - 45 min Viktor Vašátko Adversarial Examples Xiaoyong Yuan, Pan He, Qile Zhu, Xiaolin Li: Adversarial Examples: Attacks and Defenses for Deep Learning
11 Nov 2019 - 45 min Jan Vainer Normalizing flows, Real NVP, Glow Laurent Dinh, Jascha Sohl-Dickstein, Samy Bengio: Density estimation using Real NVP
Diederik P. Kingma, Prafulla Dhariwal: Glow: Generative Flow with Invertible 1x1 Convolutions
18 Nov 2019 Memduh Gokirmak
Abhishek Agrawal
NN Interpretation Hewitt, John, and Christopher D. Manning. A Structural Probe for Finding Syntax in Word Representations
Ning Mei, Usman Sheikh, Roberto Santana and David Soto How the brain encodes meaning: Comparing word embedding and computer vision models to predict fMRI data during visual word recognition
25 Nov 2019 Erdi Düzel Image Segmentation Xiaomei Zhao, Yihong Wu, Guidong Song, Zhenye Li, Yazhuo Zhang, Yong Fan A deep learning model integrating FCNNs and CRFs for brain tumor segmentation
02 Dec 2019 Milan Straka Optimization Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, Cho-Jui Hsieh: Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
Michael R. Zhang, James Lucas, Geoffrey Hinton, Jimmy Ba: Lookahead Optimizer: k steps forward, 1 step back
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Jiawei Han: On the Variance of the Adaptive Learning Rate and Beyond
09 Dec 2019 Václav Volhejn Overfitting and generalization Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals. Understanding deep learning requires rethinking generalization
Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik Mandal. Reconciling modern machine learning practice and the bias-variance trade-off
Hartmut Maennel, Olivier Bousquet, Sylvain Gelly. Gradient Descent Quantizes ReLU Network Features
16 Dec 2019 David Samuel Neural ODEs Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, David Duvenaud. Neural Ordinary Differential Equations
Emilien Dupont, Arnaud Doucet, Yee Whye Teh. Augmented Neural ODEs
Yulia Rubanova, Ricky T. Q. Chen, David Duvenaud. Latent ODEs for Irregularly-Sampled Time Series
23 Dec 2019 No DL seminar Christmas Holiday
30 Dec 2019 No DL seminar Christmas Holiday
06 Jan 2020 David Kubeša Entity Linking Nikolaos Kolitsas, Octavian-Eugen Ganea, Thomas Hofmann. End-to-End Neural Entity Linking
Possibly also Samuel Broscheit. Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking

You can choose any paper you find interesting, but if you would like some inspiration, you can look at the following list. The papers are grouped, each group is expected to be presented on one seminar.

Natural Language Processing

Generative Modeling

Neural Architecture Search (AutoML)

Networks with External Memory

Optimization

Adversarial Examples