\documentclass[oneside, a4paper, onecolumn, 11pt]{article}

%%% PACKAGES %%%

\usepackage[left=2cm,top=2cm,bottom=1.5cm,right=2cm]{geometry}
\usepackage{ucs}
\usepackage[T1]{fontenc}
\usepackage[utf8x]{inputenc}
\usepackage{url}
\usepackage{hyperref}
\usepackage{xcolor}
\usepackage{amsmath}
\usepackage{enumitem}

\title{Machine Translation Marathon 2016 -- Lab Session -- Neural Machine Translation}
\author{Rico Sennrich \texttt{<rico.sennrich@ed.ac.uk>} \and Tom Kocmi\texttt{<kocmitom@gmail.com>}}
\date{15. September 2016}
\begin{document}
\maketitle


\section{Neural Machine Translation with Nematus}

\noindent Nematus\footnote{\url{https://github.com/rsennrich/nematus}} is a theano-based toolkit for training and decoding with attentional encoder-decoder networks \cite{DBLP:journals/corr/BahdanauCB14}. It is a fork of the dl4mt-tutorial\footnote{\url{https://github.com/nyu-dl/dl4mt-tutorial}}, and was used by the University of Edinburgh for their top-ranked WMT16 submissions \cite{sennrich-haddow-birch:2016:WMT}. Pre-trained models are available for the language pairs EN$\leftrightarrow$\{CS,DE,RO,RU\}.\footnote{\url{http://statmt.org/rsennrich/wmt16_systems/}}

In this lab session, you will learn how to do the following:

\begin{enumerate}
\item install Nematus
\item translate a document with Nematus and an existing NMT model
\item adapt an existing NMT model to a new domain via continued training \cite{luong2015,2015arXiv151106709S}
\end{enumerate}

\subsection{Install Nematus}

First, choose where you want to install Nematus. It is highly recommended to install Nematus on a machine with a CUDA-capable NVIDIA GPU (with at least 6GB memory).
You will not need administrative privileges to install Nematus itself, but administrative privileges are required to install some prerequisites, such as nvidia-docker, or CUDA and CUDNN. If you do not have access to a machine where all dependencies are installed, or where you have administrative privileges, you can use an Amazon EC2 machine.

If available, we recommend installation via Docker,
but we provide installation instructions with and without Docker.

\subsubsection{installation in Docker container}

\begin{enumerate}
\item install docker and nvidia-docker: \url{https://github.com/NVIDIA/nvidia-docker}. On Amazon EC2, follow these instructions to create a GPU instance and install nvidia-docker: \url{https://github.com/NVIDIA/nvidia-docker/wiki/Deploy-on-Amazon-EC2}.
\item download Nematus:
\begin{verbatim}
git clone https://github.com/rsennrich/nematus
\end{verbatim}
\item build docker image:
\begin{verbatim}
cd nematus && docker build -t nematus-docker -f Dockerfile.gpu . && cd ..
\end{verbatim}
\end{enumerate}

\subsubsection{installation without Docker (in Python virtual environment)}

\begin{enumerate}
\item install CUDA (and ideally CUDNN)
\item install theano in python virtual environment
\begin{verbatim}
pip install --user virtualenv #install virtualenv
virtualenv virtual_environment #create environemnt
source virtual_environment/bin/activate #start environment
pip install numpy numexpr cython tables theano ipdb #install theano
\end{verbatim}
\item you may need to install BLAS library and other dependencies\\
on Debian Linux:
\begin{verbatim}
sudo apt-get install liblapack-dev libblas-dev gfortran libhdf5-serial-dev
\end{verbatim}
\item install Nematus and scripts for preprocessing
\begin{itemize}
\item mosesdecoder (just for preprocessing, no installation required)
\begin{verbatim}
git clone https://github.com/moses-smt/mosesdecoder
\end{verbatim}
\item subword-nmt (for BPE segmentation)
\begin{verbatim}
git clone https://github.com/rsennrich/subword-nmt
\end{verbatim}
\item Nematus (DL4MT fork; for training NMT)
\begin{verbatim}
git clone https://www.github.com/rsennrich/nematus
cd nematus && python setup.py install && cd ..
\end{verbatim}
\end{itemize}
\end{enumerate}

\section{Translating with Nematus}

\noindent Pre-trained models are available at \url{http://data.statmt.org/rsennrich/wmt16\_systems/} for 8 translation directions.
This tutorial assumes that we work with the translation direction English$\to$German.

\begin{enumerate}
\item download model and enter directory:
\begin{verbatim}
wget -r --cut-dirs=2 -e robots=off -nH -np -R index.html* \
http://data.statmt.org/rsennrich/wmt16_systems/en-de/
cd en-de
chmod a+x translate.sh
\end{verbatim}
\item if you installed Nematus via Docker, start the Docker instance:
\begin{verbatim}
nvidia-docker run -v `pwd`:/playground -it nematus-docker
\end{verbatim}
\item if you did not install Nematus via Docker, adjust paths in \emph{translate.sh} to point to your location of nematus, mosesdecoder and subword-nmt
\item translate a test sentence:
\begin{verbatim}
echo "This is a test." | ./translate.sh
\end{verbatim}
\end{enumerate}

\section{Adapting an Existing Model}

\noindent The pre-trained models are trained on data provided by the WMT shared translation task\footnote{\url{http://www.statmt.org/wmt16/}}, and optimized for News texts. In this section, you will adapt the model to the TED data set, which is the test domain of the IWSLT shared translation task\footnote{\url{https://sites.google.com/site/iwsltevaluation2016/mt-track}}.
We will use the pre-trained model from the previous section.

\begin{enumerate}
\item download training and development data:
\begin{verbatim}
wget https://wit3.fbk.eu/archive/2016-01//texts/en/de/en-de.tgz
mkdir iwslt
tar -xf en-de.tgz -C iwslt
\end{verbatim}
\item preprocess training data. While you are free to define your own preprocessing when training a new model, we highly recommend to use the preprocessing that was used for the original model when continuing training thereof. For this case, execute the following commands for preprocessing (as seen in \url{http://data.statmt.org/rsennrich/wmt16\_systems/en-de/translate.sh}):
\end{enumerate}
\begin{verbatim}
#English (source side)
cat iwslt/en-de/train.tags.de-en.en | \
/path/to/mosesdecoder/scripts/tokenizer/normalize-punctuation.perl -l en | \
/path/to/mosesdecoder/scripts/tokenizer/tokenizer.perl -l en -penn | \
/path/to/mosesdecoder/scripts/recaser/truecase.perl -model truecase-model.en | \
/path/to/subword-nmt/apply_bpe.py -c ende.bpe > corpus.bpe.en

#German (target side)
wget http://data.statmt.org/rsennrich/wmt16_systems/de-en/truecase-model.de
cat iwslt/en-detrain.tags.de-en.de | \
/path/to/mosesdecoder/scripts/tokenizer/normalize-punctuation.perl -l de | \
/path/to/mosesdecoder/scripts/tokenizer/tokenizer.perl -l de | \
/path/to/mosesdecoder/scripts/recaser/truecase.perl -model truecase-model.de | \
/path/to/subword-nmt/apply_bpe.py -c ende.bpe > corpus.bpe.de
\end{verbatim}


\begin{enumerate}[resume]
\item determine your training configuration. You can use the file at \url{https://raw.githubusercontent.com/rsennrich/wmt16-scripts/master/sample/config.py} as a starting point. Some parameters, like the vocabulary size, must fit that of the original model -- you can check the model parameters of the original model in the corresponding JSON file \footnote{\url{http://data.statmt.org/rsennrich/wmt16\_systems/en-de/model.npz.json}}. 

\begin{itemize}
\item set \emph{datasets} to the preprocessed training sets from the previous step.
\item ensure that the \emph{dictionaries}, vocabulary size (\emph{n\_words} and \emph{n\_words\_src}), and size of the embedding/hidden layers (\emph{dim} and \emph{dim\_words}) correspond to that of the original model.
\item to continue training an existing model, \emph{saveto} must point to the original model, and \emph{reload\_} must be True.
\item add an option "max\_epochs=1" to only perform training for one epoch on the IWSLT dataset. The model would quickly overfit with multiple epochs, and one epoch should take 30-60min, depending on your hardware.
\item remove \emph{external\_validation\_script} for now\footnote{This is used for {\sc Bleu} early stopping; look at \url{https://raw.githubusercontent.com/rsennrich/wmt16-scripts/master/sample/validate.sh} for an example.}, and set \emph{valid\_datasets} to None (validation is generally useful, but we skip it in this tutorial for simplicity).
\item the model should train well on a GPU with 8GB of memory; to run on a 6GB GPU, there are various ways to reduce memory consumption, for instance reducing the \emph{batch\_size} to 60.
\end{itemize}
\item start training (this will overwrite your original \emph{model.npz}, so you should make a copy first):
\end{enumerate}
\begin{verbatim}
THEANO_FLAGS=mode=FAST_RUN,floatX=float32,device=gpu,lib.cnmem=0 python config.py
\end{verbatim}
\begin{enumerate}[resume]
\item you can translate a validation set from IWSLT before and after adaptation to observe the difference in {\sc Bleu}.
IWSLT uses the NIST scoring tools (mteval-vX.pl) for scoring {\sc Bleu}, and input and references are wrapped in XML.
Here's an example showing how to translate and score a test set, assuming that your translation system is \emph{./translate.sh}
\end{enumerate}
{\scriptsize
\begin{verbatim}
/path/to/mosesdecoder/scripts/ems/support/input-from-sgm.perl < iwslt/en-de/IWSLT16.TED.tst2013.en-de.en.xml > tst2013.en
./translate.sh < tst2013.en > tst2013.out
/path/to/mosesdecoder/scripts/ems/support/wrap-xml.perl german \
iwslt/en-de/IWSLT16.TED.tst2013.en-de.en.xml < tst2013.out > tst2013.out.xml
/path/to/mosesdecoder/scripts/generic/mteval-v13a.pl -c -s iwslt/en-de/IWSLT16.TED.tst2013.en-de.en.xml \
-r iwslt/en-de/IWSLT16.TED.tst2013.en-de.de.xml -t tst2013.out.xml
\end{verbatim}}

On \emph{IWSLT16.TED.tst2013}, we obtained a {\sc Bleu} score of 0.2677 with the pre-trained model -- you should be able to reproduce this number exactly.
After one epoch of continued training, we obtained a {\sc Bleu} score of 0.3007 -- your score will vary because the datasets are randomly shuffled before training, which makes results non-deterministic.

\bibliographystyle{apalike}
\bibliography{bibliography.bib}

\end{document}
