eman
...a feature-packed experiment manager.
Tutorial
To understand the concept and basics, make sure to read The Design of Eman, an Experiment Manager
SMT Playground
Eman is bundled with a set of tools for experimenting in SMT. We provide seeds for building and running many SMT tools, such as Moses, GIZA++ or SRILM.
Our tool corpman is useful for managing multilingual, factored text corpora and deriving new corpora as needed by the experiments–either by directly extracting the required factors or by constructing the factors based on user-defined rules (stems and lemmas can be easily produced this way).
Most importantly, we provide highly flexible seeds that execute all of the common (and many less common) tasks in SMT experiments: word alignment, training language models, defining and training translation models, running tuning algorithms and several methods of automatic MT evaluation.
This tutorial has three parts. In the first one, you will define and run a toy experiment simply by cloning an existing traceback (definition of the whole experiment pipeline). In the second part, you will learn how to create an experiment from scratch. In the last part, you will go a bit further and explore some of eman's cool features.
A Toy Experiment
See the Download section for information on how to get eman and SMT playground.
Make sure that eman is in your $PATH.
You might need to install some additional Perl modules (such as YAML). Use CPAN to get them.
Go to the SMT playground directory.
In the directory eman.samples, sample data and experiment tracebacks are located. Let's clone the simplest one:
eman clone < eman.samples/cs-en-mini.traceback
A collection of steps was just created by eman. Your playground should include directories similar to these:
s.align.96ca0ceb.20120926-2359 # word alignment step
s.corpus.c043f477.20120926-2359 # train/dev/test corpora
s.corpus.de8b80ac.20120926-2359
s.corpus.e9fd27d2.20120926-2359
s.corpus.b8c74dab.20120926-2359
s.corpus.d5c9b3a5.20120926-2359
s.corpus.00c48dfa.20120926-2359
s.evaluator.040cd8af.20120926-2359 # evaluation of the final model
s.lm.13fcfd3c.20120926-2359 # language model training
s.mert.3778ca44.20120926-2359 # tuning step
s.model.0906c6cd.20120926-2359 # step defining the model (tm, lm, factors,...)
s.mosesgiza.de9e2e29.20120926-2359 # Moses, GIZA++
s.srilm.01dd617d.20120926-2359 # SRILM
s.translate.ab0d9b84.20120926-2359 # translation of the test set
s.tm.ac8e743a.20120926-2359 # translation model training
The experiment steps have been created but eman did not start the experiment yet. The evaluator step is the final one. You can view the dependency structure of the experiment (or traceback) using this command:
eman traceback s.evaluator.040cd8af.20120926-2359
Use options -vars and -status to display more details about the steps.
You can run the experiment:
eman start s.evaluator.040cd8af.20120926-2359
Use commands such as eman tb -stat s.evaluator.040cd8af.20120926-2359 or eman ls -stat to track the progress.
Eman will automatically assure that steps are executed only when their dependencies are finished. This feature works both locally and on the SGE cluster.
When all steps have finished, you can check the BLEU score like this:
cat s.evaluator.040cd8af.20120926-2359/scores
Creating New Experiments
You will need several tools for experimenting in MT. Download and compile Moses and GIZA++:
eman init --start mosesgiza
Compile the language modeling toolkit SRILM:
eman init --start srilm
Prepare some text data (corpora), split them into sections and individual languages. Let's assume that you are translating from English into French. Your files may then look like this:
en.train fr.train en.dev fr.dev en.test fr.test
Import them into your playground using a series of commands like this (vary the section and language–this is easily scriptable and you usually only need to do it once!):
OUTCORP=en-fr-train OUTLANG=en OUTFACTS=form \ OUTLINECOUNT=30000 TAKE_FROM_COMMAND="cat /path/to/corpus/en.train" \ eman init -start corpus
All eman commands must be executed from the playground directory.
By executing eman init -start, you created the step and started it at the same time.
Shortly, running eman ls -stat should show the corpus and status DONE. If not, check its log (e.g. s.corpus.00c48dfa.20120926-2359/log.o123456).
It is useful to update both eman's and corpman's index when making major changes. It is usually also the first "debugging" step when something stops working:
eman reindex && ./corpman reindex
When you have imported all your corpora, you can word-align the parallel training data:
GIZASTEP=s.mosesgiza.de9e2e29.20120926-2359 \
CORPUS=en-fr-train SRCALIAUG=en+form TGTALIAUG=fr+form \
eman init -start align
Note that you need to replace the mosesgiza step ID with a valid one from your playground.
If the command succeeded, eman returned the ID of the newly created step (let's call it xyz). We are going to need it right away for creating a translation model:
SRCAUG=en+form TGTAUG=fr+form ALISTEP=s.align.xyz \
DECODINGSTEPS=t0-0 eman init -start tm
We always specify the used factors (form). Eman fully supports complex factored setups. The definition of decoding steps allows to specify any factor configuration. In this case, we just translate (t) the 0th factor on the source side to the 0th factor on the target side.
Now let's train a language model:
SRILMSTEP=s.srilm.01dd617d.20120926-2359 CORP=en-fr-train \
CORPAUG=fr+form ORDER=4 eman init -start lm
We can now prepare the full Moses configuration, using step model:
TMS=s.tm.xyz LMS=0:s.lm.xyz eman init -start model
The 0 before language model step ID represents the factor it should be applied on. Multiple language models are separated by ":::".
You can now tune the model:
MODELSTEP=s.model.xyz DEVCORP=en-fr-dev REFAUG=fr+form eman init -start mert
The seed mert actually supports running several common tuning algorithms, such as PRO or Batch MIRA.
Create the step that translates your test data:
MERTSTEP=s.mert.xyz TESTCORP=en-fr-test eman init -start translate
Run evaluation:
TRANSSTEP=s.translate.xyz eman init -start evaluator
That's all! You just created a complete MT experiment using eman. Once evaluator finishes, you can read the score like this:
cat s.evaluator.xyz/scores
Beyond Basics
The manual page contains comprehensive documentation of eman's features. In this section, we introduce a few of them.
Cloning of Steps
Once your playground is populated with steps, you can easily create new ones using cloning. You can create identical copies–this is useful e.g. for repeating tuning:
eman clone -start s.mert.xyz
By specifying different values for variables, you can create modified versions of existing steps:
ORDER=3 eman clone -start s.lm.xyz
Cloning of Experiments
You can also clone whole experiments (tracebacks). A cloneable traceback needs to be created using the following command:
eman traceback -vars s.evaluator.xyz > traceback
You can then clone the whole experiment. You can of course modify the traceback and create different experiments with identical basic structure. E.g. you can revert the translation direction.
When you are happy with the changes, clone the experiment:
eman clone < traceback
Listing Steps
Eman creates a directory for each step. The playground can soon become hard to navigate manually. For this purpose, eman has two basic commands: ls and select.
Using eman ls is simple:
eman ls mert -stat # lists all mert steps and their statuses
Eman select has compact and extensible syntax (documented in the manpage) and its usage resembles database queires:
eman sel t tm v SRCAUG=en+form d # lists steps 'tm' that translate from English forms and are DONE
Sharing Steps
You can easily use steps from other playgrounds using eman addremote:
eman addremote /home/boris/statmt/playground/ boris && eman reindex
After updating eman's index, you will see the remote steps in playground listing (eman ls/select) and your steps will be able to use the remote steps. Corpman will also search the added playgrounds.