Deep Learning Seminar, Summer 2016/17

In recent years, deep neural networks have been used to solve complex machine-learning problems and have achieved significant state-of-the-art results in many areas. The whole field of deep learning has been developing rapidly, with new methods and techniques emerging steadily.

The goal of the seminar is to follow the newest advancements in the deep learning field. The course takes form of a reading group – each lecture a paper is presented by one of the students. The paper is announced in advance, hence all participants can read it beforehand and can take part in the discussion of the paper

  • Deep Learning – Course introducing deep neural networks, from the basics to the latest advances, focusing both on theory as well as on practical aspects.


In summer semester 2016/17, the Deep Learning Seminar takes place on Tuesday at 12:20 in S1. We will first meet on Tuesday Feb 28.

If you want to receive announcements about chosen papers, sign up to our mailing list.

Date Who Paper(s)
28 Feb 2017 Mirek Olšák C. Kaliszyk, F. Chollet, C. Szegedy: HolStep: A Machine Learning Dataset for Higher-order Logic Theorem Proving
TreeRNN-based implementation of Mirek Olšák surpassing accuracy of above paper from 83% to 88%
07 Mar 2017 Dušan Variš Jason Lee, Kyunghyun Cho, Thomas Hofmann: Fully Character-Level Neural Machine Translation without Explicit Segmentation
14 Mar 2017 Karel Král Geoffrey Hinton, Oriol Vinyals, Jeff Dean: Distilling the Knowledge in a Neural Network
Lei Jimmy Ba, Rich Caruana: Do Deep Nets Really Need to be Deep?
21 Mar 2017 Milan Straka Moshe Looks, Marcello Herreshoff, DeLesley Hutchins, Peter Norvig: Deep Learning with Dynamic Computation Graphs
Lingpeng Kong, Chris Alberti, Daniel Andor, Ivan Bogatyy, David Weiss: DRAGNN: A Transition-Based Framework for Dynamically Connected Neural Networks
28 Mar 2017 Lukáš Jendele Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick: Mask R-CNN
Yi Li, Haozhi Qi, Jifeng Dai, Xiangyang Ji, Yichen Wei: Fully Convolutional Instance-aware Semantic Segmentation
04 Apr 2017 Ondrej Škopek Irwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi, Samy Bengio: Neural Combinatorial Optimization with Reinforcement Learning
Oriol Vinyals, Meire Fortunato, Navdeep Jaitly: Pointer Networks
11 Apr 2017 Jan Hajič jr. Diederik P Kingma, Max Welling: Auto-Encoding Variational Bayes
Francisco J. R. Ruiz, Michalis K. Titsias, David M. Blei: The Generalized Reparameterization Gradient
18 Apr 2017 Jindřich Libovický Holger Schwenk, Ke Tran, Orhan Firat, Matthijs Douze: Learning Joint Multilingual Sentence Representations with Neural Machine Translation
25 Apr 2017 Milan Straka Mevlana Gemici et al.: Generative Temporal Models with Memory
Alex Graves et al.: Hybrid computing using a neural network with dynamic external memory
02 May 2017 David Mareček Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, Wang Ling: Learning to Compose Words into Sentences with Reinforcement Learning
09 May 2017 Rudolf Rosa Michael Sejr Schlichtkrull, Anders Søgaard: Cross-Lingual Dependency Parsing with Late Decoding for Truly Low-Resource Languages
16 May 2017 Jindřich Helcl Noam Shazeer et al.: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
23 May 2017 Peter Zborovský Luca Bertinetto et al.: Fully-Convolutional Siamese Networks for Object Tracking

Papers for Inspiration

You can choose any paper you find interesting, but if you would like some inspiration, you can look at the following list.

Collections of Deep Learning Papers

Word Embeddings

  • Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, Adam Kalai: Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings.


Neural Machine Translation

Language Correction

  • Ziang Xie, Anand Avati, Naveen Arivazhagan, Dan Jurafsky, Andrew Y. Ng: Neural Language Correction with Character-Based Attention.

Language Modelling

Reinforcement Learning

  • Frank S. He, Yang Liu, Alexander G. Schwing, Jian Peng: Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening.
  • Natasha Jaques, Shixiang Gu, Richard E. Turner, Douglas Eck: Tuning Recurrent Neural Networks with Reinforcement Learning.
  • Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, Raia Hadsell: Learning to Navigate in Complex Environments.
  • Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, Wang Ling: Learning to Compose Words into Sentences with Reinforcement Learning.
  • Chelsea Finn, Tianhe Yu, Justin Fu, Pieter Abbeel, Sergey Levine: Generalizing Skills with Semi-Supervised Reinforcement Learning.

Program Generation

Adversarial Networks

Network Architectures

  • Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, Nando de Freitas: Learning to learn by gradient descent by gradient descent.
  • Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, Jürgen Schmidhuber: Recurrent Highway Networks.
  • Lingpeng Kong, Chris Alberti, Daniel Andor, Ivan Bogatyy, David Weiss: DRAGNN: A Transition-Based Framework for Dynamically Connected Neural Networks.
  • Bowen Baker, Otkrist Gupta, Nikhil Naik, Ramesh Raskar: Designing Neural Network Architectures using Reinforcement Learning.

Structured Prediction

  • Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, Michael Collins: Globally Normalized Transition-Based Neural Networks.

Image Labeling

  • Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan: Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge.

Image Recognition

Image Enhancement

  • Justin Johnson, Alexandre Alahi, Li Fei-Fei: Perceptual Losses for Real-Time Style Transfer and Super-Resolution.
  • Richard Zhang, Phillip Isola, Alexei A. Efros: Colorful Image Colorization.
  • Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, Wenzhe Shi: Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network.
  • Ryan Dahl, Mohammad Norouzi, Jonathon Shlens: Pixel Recursive Super Resolution.

Speech Synthesis

  • Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, Koray Kavukcuoglu: WaveNet: A Generative Model for Raw Audio.