Deep Learning Seminar 2016

In recent years, deep neural networks have been used to solve complex machine-learning problems and have achieved significant state-of-the-art results in many areas. The whole field of deep learning has been developing rapidly, with new methods and techniques emerging steadily.

The goal of the seminar is to follow the newest advancements in the deep learning field. The course takes form of a reading group – each lecture a paper is presented by one of the students. The paper is announced in advance, hence all participants can read it beforehand and can take part in the discussion of the paper.

Related Courses

  • Deep Learning – Course introducing deep neural networks, from the basics to the latest advances, focusing both on theory as well as on practical aspects.

Summer Semester

In summer semester 2016, the Deep Learning Seminar takes place on Tuesday at 12:20 in S1. We will first meet on Tuesday Feb 28.

If you want to receive announcements about chosen papers, sign up to our mailing list.


Date Who Title Link
28 Feb 2017 Mirek Olšák C. Kaliszyk, F. Chollet, C. Szegedy: HolStep: A Machine Learning Dataset for Higher-order Logic Theorem Proving
    TreeRNN-based implementation of Mirek Olšák surpassing accuracy of above paper from 83% to 88%
07 Mar 2017 Dušan Variš Jason Lee, Kyunghyun Cho, Thomas Hofmann: Fully Character-Level Neural Machine Translation without Explicit Segmentation
14 Mar 2017 Karel Král Geoffrey Hinton, Oriol Vinyals, Jeff Dean: Distilling the Knowledge in a Neural Network
    Lei Jimmy Ba, Rich Caruana: Do Deep Nets Really Need to be Deep?
21 Mar 2017 Milan Straka Moshe Looks, Marcello Herreshoff, DeLesley Hutchins, Peter Norvig: Deep Learning with Dynamic Computation Graphs
    Lingpeng Kong, Chris Alberti, Daniel Andor, Ivan Bogatyy, David Weiss: DRAGNN: A Transition-Based Framework for Dynamically Connected Neural Networks
28 Mar 2017 Lukáš Jendele Kaiming He, Georgia Gkioxari, Piotr Dollár, Ross Girshick: Mask R-CNN
    Yi Li, Haozhi Qi, Jifeng Dai, Xiangyang Ji, Yichen Wei: Fully Convolutional Instance-aware Semantic Segmentation
04 Apr 2017 Ondrej Škopek Irwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi, Samy Bengio: Neural Combinatorial Optimization with Reinforcement Learning
    Oriol Vinyals, Meire Fortunato, Navdeep Jaitly: Pointer Networks
11 Apr 2017 Jan Hajič jr.    
18 Apr 2017      
25 Apr 2017      
02 May 2017      
09 May 2017      
16 May 2017      
23 May 2017      

Papers for Inspiration

You can choose any paper you find interesting, but if you would like some inspiration, you can look at the following list.

Collections of Deep Learning Papers

Word Embeddings

  • Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, Adam Kalai: Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings.


Neural Machine Translation

Language Correction

  • Ziang Xie, Anand Avati, Naveen Arivazhagan, Dan Jurafsky, Andrew Y. Ng: Neural Language Correction with Character-Based Attention.

Language Modelling

Reinforcement Learning

  • Frank S. He, Yang Liu, Alexander G. Schwing, Jian Peng: Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening.
  • Natasha Jaques, Shixiang Gu, Richard E. Turner, Douglas Eck: Tuning Recurrent Neural Networks with Reinforcement Learning.
  • Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, Raia Hadsell: Learning to Navigate in Complex Environments.
  • Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, Wang Ling: Learning to Compose Words into Sentences with Reinforcement Learning.
  • Chelsea Finn, Tianhe Yu, Justin Fu, Pieter Abbeel, Sergey Levine: Generalizing Skills with Semi-Supervised Reinforcement Learning.

Program Generation

Adversarial Networks

Network Architectures

  • Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W. Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, Nando de Freitas: Learning to learn by gradient descent by gradient descent.
  • Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, Jürgen Schmidhuber: Recurrent Highway Networks.
  • Lingpeng Kong, Chris Alberti, Daniel Andor, Ivan Bogatyy, David Weiss: DRAGNN: A Transition-Based Framework for Dynamically Connected Neural Networks.
  • Bowen Baker, Otkrist Gupta, Nikhil Naik, Ramesh Raskar: Designing Neural Network Architectures using Reinforcement Learning.

Structured Prediction

  • Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, Michael Collins: Globally Normalized Transition-Based Neural Networks.

Image Labeling

  • Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan: Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge.

Image Recognition

Image Enhancement

  • Justin Johnson, Alexandre Alahi, Li Fei-Fei: Perceptual Losses for Real-Time Style Transfer and Super-Resolution.
  • Richard Zhang, Phillip Isola, Alexei A. Efros: Colorful Image Colorization.
  • Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, Wenzhe Shi: Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network.
  • Ryan Dahl, Mohammad Norouzi, Jonathon Shlens: Pixel Recursive Super Resolution.

Speech Synthesis

  • Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, Koray Kavukcuoglu: WaveNet: A Generative Model for Raw Audio.