Install Moses Support Libraries

• Create some workspace directory for all this work and enter it.
  
• Download SRILM and install it.
  
• Download GIZA and mkcls. The original versions don't compile on modern systems. Try Chris Dyer's patched versions instead.
  cd GIZA++-v2/
make
make snt2cooc.out
cd ../mkcls-v2/
make

• Copy GIZA++ and mkcls to a bin location for Moses Scripts
  mkdir -p bin
cp GIZA++-v2/GIZA++ bin/
cp GIZA++-v2/snt2cooc.out bin/
cp mkcls-v2/mkcls bin/


Install Moses

• Check out the Moses code via Subversion:
  mkdir -p moses
svn co https://mosesdecoder.svn.sourceforge.net/svnroot/mosesdecoder/trunk moses

• Compile Moses
  cd moses
./regenerate-makefiles.sh
./configure --with-srilm=/path-to-srilm
make -j 4


Install Moses Scripts

• Compile Moses Scripts
  The support scripts used by Moses are "released" by a Makefile which edits their paths to match your local environment. First, you need to edit the Makefile definition of two variables:
  mkdir -p bin/moses-scripts

###Edit moses/scripts/Makefile
TARGETDIR=/full-path-to-workspace/bin/moses-scripts
BINDIR=/full-path-to-workspace/bin
###

cd moses/scripts/
make release

  This will create a folder named bin/moses-scripts/scripts-YYYYMMDD-HHMM with released versions of all the scripts. You will call these versions when training/tuning Moses.
  Moses scripts also require a SCRIPTS_ROOTDIR environment variable to be set. The output of make release should indicate this.
  export SCRIPTS_ROOTDIR=/full-path-to-workspace/bin/moses-scripts/scripts-YYYYMMDD-HHMM


Install Additional Scripts

• Download scripts.tgz and extract them:
  tar xzf scripts.tgz

• These scripts include:
  • Tokenizer scripts/tokenizer.perl
  • Lowercaser scripts/lowercase.perl
  • SGML-Wrapper scripts/wrap-xml.perl
• Dowload the NIST BLEU scoring tool:
  wget ftp://jaguar.ncsl.nist.gov/mt/resources/mteval-v11b.pl

Prepare Data

• Tokenize training data
  mkdir -p working-dir/corpus
scripts/tokenizer.perl -l fr < wmt07/training/europarl-v3.fr-en.fr > working-dir/corpus/europarl.tok.fr
scripts/tokenizer.perl -l en < wmt07/training/europarl-v3.fr-en.en > working-dir/corpus/europarl.tok.en

• Filter out long sentences
  bin/moses-scripts/scripts-YYYYMMDD-HHMM/training/clean-corpus-n.perl working-dir/corpus/europarl.tok fr en working-dir/corpus/europarl.clean 1 40

• Lowercase training data
  scripts/lowercase.perl < working-dir/corpus/europarl.clean.fr > working-dir/corpus/europarl.lowercased.fr
scripts/lowercase.perl < working-dir/corpus/europarl.clean.en > working-dir/corpus/europarl.lowercased.en


Build Language Model

• Tokenize English language model data
  mkdir -p working-dir/lm
scripts/tokenizer.perl -l en < wmt07/training/europarl-v3.en > working-dir/lm/europarl.tok

• Lowercase language model data
  scripts/lowercase.perl < working-dir/lm/europarl.tok > working-dir/lm/europarl.lowercased

• Use SRILM to build language model
  SRILM makes a platform-specific folder within its bin directory, this instruction assumes i686.
  /path-to-srilm/bin/i686/ngram-count -order 5 -interpolate -kndiscount -text working-dir/lm/europarl.lowercased -lm working-dir/lm/europarl.lm


Train Model

• Run training script:
  bin/moses-scripts/scripts-YYYYMMDD-HHMM/training/train-factored-phrase-model.perl -scripts-root-dir bin/moses-scripts/scripts-YYYYMMDD-HHMM -root-dir working-dir -corpus working-dir/corpus/europarl.lowercased -f fr -e en -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm 0:5:working-dir/lm/europarl.lm:0


Tuning (i.e., Optimize System Component Weights, a.k.a. Minimum Error Rate Training)

• Tokenize tuning sets
  mkdir -p working-dir/tuning
scripts/tokenizer.perl -l fr < wmt07/dev/dev2006.fr > working-dir/tuning/input.tok
scripts/tokenizer.perl -l en < wmt07/dev/dev2006.en > working-dir/tuning/reference.tok

• Lowercase tuning sets
  scripts/lowercase.perl < working-dir/tuning/input.tok > working-dir/tuning/input
scripts/lowercase.perl < working-dir/tuning/reference.tok > working-dir/tuning/reference

• Run tuning script
  Note that this step can take many hours, even days, to run.
  bin/moses-scripts/scripts-YYYYMMDD-HHMM/training/mert-moses.pl working-dir/tuning/input working-dir/tuning/reference moses/moses-cmd/src/moses working-dir/model/moses.ini --working-dir working-dir/tuning --rootdir bin/moses-scripts/scripts-YYYYMMDD-HHMM

• Insert weights into configuration file
scripts/reuse-weights.perl working-dir/tuning/moses.ini < working-dir/model/moses.ini > working-dir/tuning/moses.weight-reused.ini