Deep Learning Seminar, Summer 2017/18
In recent years, deep neural networks have been used to solve complex machine-learning problems and have achieved significant state-of-the-art results in many areas. The whole field of deep learning has been developing rapidly, with new methods and techniques emerging steadily.
The goal of the seminar is to follow the newest advancements in the deep learning field. The course takes form of a reading group – each lecture a paper is presented by one of the students. The paper is announced in advance, hence all participants can read it beforehand and can take part in the discussion of the paper.
If you want to receive announcements about chosen paper, sign up to our mailing list ufal-rg@googlegroups.com.
About
SIS code: NPFL117
Semester: summer
E-credits: 3
Examination: 0/2 C
Guarantor: Milan Straka
Timespace Coordinates
The Deep Learning Seminar takes place on Tuesday at 14:00 in S1. We will first meet on Tuesday Feb 27.
Requirements
To pass the course, you need to present a research paper and sufficiently attend the presentations.
License
Unless otherwise stated, teaching materials for this course are available under CC BY-SA 4.0.
If you want to receive announcements about chosen paper, sign up to our mailing list ufal-rg@googlegroups.com.
To add your name and paper to the table below, edit the source code on GitHub and send a PR.
| Date | Who | Paper(s) |
|---|---|---|
| 27 Feb 2018 | Milan Straka | Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, Hervé Jégou: Word Translation Without Parallel Data |
| 06 Mar 2018 | Martin Popel | Michal Rolínek, Georg Martius: L4: Practical loss-based stepsize adaptation for deep learning |
| 13 Mar 2018 | Jan Hajič | Matthias Dorfer, Jan Schlüter, Andreu Vall, Filip Korzeniowsky, Gerhard Widmer: End-to-End Cross-Modality Retrieval with CCA Projection and Pairwise Ranking Loss |
| 20 Mar 2018 | Tomas Soucek | Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le: Learning Transferable Architectures for Scalable Image Recognition |
| 27 Mar 2018 | Petr Bělohlávek | Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Łukasz Kaiser, Noam Shazeer, Alexander Ku: Image Transformer |
| 03 Apr 2018 | Petr Houška | Jason Liang, Elliot Meyerson, Risto Miikkulainen: Evolutionary Architecture Search For Deep Multitask Networks |
| 10 Apr 2018 | Milan Straka | Esteban Real, Alok Aggarwal, Yanping Huang, Quoc V Le: Regularized Evolution for Image Classifier Architecture Search Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, Kevin Murphy: Progressive Neural Architecture Search |
| 17 Apr 2018 | Ronald Cardenas | Jiatao Gu, Hany Hassan, Jacob Devlin, Victor O.K. Li: Universal Neural Machine Translation for Extremely Low Resource Languages |
| 24 Apr 2018 | Jakub Náplava | Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, Oriol Vinyals: Understanding deep learning requires rethinking generalization Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky: Deep Image Prior |
| 01 May 2018 | No DL Seminar | Holiday – May Day |
| 08 May 2018 | No DL Seminar | Holiday – Victory Day |
| 15 May 2018 | Vojtěch Čermák | Adversarial Patch and Defense-GAN: Protecting classifiers against adversarial attacks using generative models |
| 22 May 2018 | Karel Ha | CapsuleGAN: Generative Adversarial Capsule Network (with the background in Dynamic Routing Between Capsules) - slides |
You can choose any paper you find interesting, but if you would like some inspiration, you can look at the following list.
Current Deep Learning Papers
Parsing
- Timothy Dozat, Christopher D. Manning: Deep Biaffine Attention for Neural Dependency Parsing
- Michael Ringgaard, Rahul Gupta, Fernando C. N. Pereira: SLING: A framework for frame semantic parsing
Neural Machine Translation
- Mikel Artetxe, Gorka Labaka, Eneko Agirre, Kyunghyun Cho: Unsupervised Neural Machine Translation
- Guillaume Lample, Ludovic Denoyer, Marc'Aurelio Ranzato: Unsupervised Machine Translation Using Monolingual Corpora Only
- Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, Richard Socher: Non-Autoregressive Neural Machine Translation
Language Modelling
- Gábor Melis, Chris Dyer, Phil Blunsom: On the State of the Art of Evaluation in Neural Language Models
- Anirudh Goyal, Nan Rosemary Ke, Alex Lamb, R Devon Hjelm, Chris Pal, Joelle Pineau, Yoshua Bengio: ACtuAL: Actor-Critic Under Adversarial Learning
Paraphrasing
- Zichao Li, Xin Jiang, Lifeng Shang, Hang Li: Paraphrase Generation with Deep Reinforcement Learning
Natural Language Generation
- Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, Eric P. Xing: Toward Controlled Generation of Text
- Sai Rajeswar, Sandeep Subramanian, Francis Dutil, Christopher Pal, Aaron Courville: Adversarial Generation of Natural Language
Speech Synthesis
- Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C. Cobo, Florian Stimberg, Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalchbrenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, Demis Hassabis: Parallel WaveNet: Fast High-Fidelity Speech Synthesis
- Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ Skerry-Ryan, Rif A. Saurous, Yannis Agiomyrgiannakis, Yonghui Wu: Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions
Image Classification
- Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le: Learning Transferable Architectures for Scalable Image Recognition
Image Recognition
- Yuntian Deng, Anssi Kanervisto, Alexander M. Rush: What You Get Is What You See: A Visual Markup Decompiler
Image Enhancement
- Ryan Dahl, Mohammad Norouzi, Jonathon Shlens: Pixel Recursive Super Resolution
- Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky: Deep Image Prior
Image 3D Reconstruction
- Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, William T Freeman, Joshua B Tenenbaum: MarrNet: 3D Shape Reconstruction via 2.5D Sketches
Training Methods
- Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, Kaiming He: Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Activation Functions
- Prajit Ramachandran, Barret Zoph, Quoc V. Le: Searching for Activation Functions
Regularization
- Sergey Ioffe: Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
- Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, David Lopez-Paz: mixup: Beyond Empirical Risk Minimization
Network Architectures
- Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le: Learning Transferable Architectures for Scalable Image Recognition
- Irwan Bello, Barret Zoph, Vijay Vasudevan, Quoc V. Le: Neural Optimizer Search with Reinforcement Learning
Network Interpretation
- Hao Li, Zheng Xu, Gavin Taylor, Tom Goldstein: Visualizing the Loss Landscape of Neural Nets
Reinforcement Learning
- Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J. Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, Dharshan Kumaran, Raia Hadsell: Learning to Navigate in Complex Environments
- John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov: Proximal Policy Optimization Algorithms
- Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, David Silver: Rainbow: Combining Improvements in Deep Reinforcement Learning
- Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine: Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
Explicit Memory
- Caglar Gulcehre, Sarath Chandar, Yoshua Bengio: Memory Augmented Neural Networks with Wormhole Connections
Hyperparameter Optimization
- Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Elliot Karro, D. Sculley: Google Vizier: A Service for Black-Box Optimization
Generative Adversarial Networks
- Lantao Yu, Weinan Zhang, Jun Wang, Yong Yu: SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
- Leon Sixt, Benjamin Wild, Tim Landgraf: RenderGAN: Generating Realistic Labeled Data
- Ilya Tolstikhin, Sylvain Gelly, Olivier Bousquet, Carl-Johann Simon-Gabriel, Bernhard Schölkopf: AdaGAN: Boosting Generative Models
- Martin Arjovsky, Soumith Chintala, Léon Bottou: Wasserstein GAN
Adversarial Images
- Tom B. Brown, Dandelion Mané, Aurko Roy, Martín Abadi, Justin Gilmer: Adversarial Patch
Adversarial Text
- Robin Jia, Percy Liang: Adversarial Examples for Evaluating Reading Comprehension Systems
Adversarial Speech
- Nicholas Carlini, David Wagner: Audio Adversarial Examples: Targeted Attacks on Speech-to-Text
- Xuejing Yuan, Yuxuan Chen, Yue Zhao, Yunhui Long, Xiaokang Liu, Kai Chen, Shengzhi Zhang, Heqing Huang, Xiaofeng Wang, Carl A. Gunter: CommanderSong: A Systematic Approach for Practical Adversarial Voice Recognition
Artificial Intelligence
- Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, Igor Mordatch: Emergent Complexity via Multi-Agent Competition
- David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, Demis Hassabis: Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
