TectoMT is a highly modular NLP (Natural Language Processing) software system implemented in Perl programming language under Linux. It is primarily aimed at Machine Translation, from English to Czech in the first phase, making use of the ideas and technology created during the Prague Dependency Treebank project. At the same time, it is also hoped to significantly facilitate and accelerate development of software solutions of many other NLP tasks, especially due to re-usability of the numerous integrated processing modules (called blocks), which are equipped with uniform object-oriented interfaces.

This document describes TectoMT from the technical viewpoint. The theoretical background related to the translation itself (the question of lexical transfer etc.) are not discussed here.

TectoMT -- as a whole -- is not an end-user application, and will never be. It is an experimental development environment, too large and too complex to become a widely used robust product; also, authors rights and specific licenses associated with some of the integrated components must be respected. However, building and public releasing of real end-user applications (consisting of selected TectoMT components) is possible and supported by the current TectoMT architecture.

When we started developing the pilot version of TectoMT in autumn 2005, our motivation for building the system was twofold.

First, we believe that the abstraction power offered by the tectogrammatical layer of language representation (as introduced by Petr Sgall in 1960's and recently implemented within the Prague Dependency Treebank project, t-layer for short) can contribute to the state-of-the-art in Machine Translation. Not only that the system based on "tecto" should not loose its linguistic interpretability in any phase and thus it should allow for simple debugging and monotonous improvements. Compared to the popular n-gram translation models, there are advantages also from the statistical viewpoint. Namely, abstracting from the repertoires of language means (such as inflection, agglutination, word order, functional words, intonation), which are used to varying extent in different languages for expressing non-lexical meanings, should make the training data contained in available parallel corpora much less sparse (data sparseness is a notorious problem in MT), and thus better machine-learnable.

Second, even if the first assumption might be wrong, we are sure it would be helpful for me and our colleagues at the institute to be able to integrate existing NLP tools (be they ours or external) into a common software framework. Thus we could ultimately get rid of the endless format conversions and frustrating ah-hoc tweaking of other people's source codes whenever one wants to perform any single operation on any single piece of linguistic data.

'Can' relates to two meanings here: (a) 'to be able' and (b) 'to be allowed.'

Ad (a): As mentioned above, TectoMT is rather a development software framework, far from the end-user application shape. It can be effectively used only by programmers with at least a basic experience in Linux/bash, including e.g. writing/understanding simple Makefiles, and with advanced knowledge of Perl, including OO-programming. Experience in working with (and customizing of) the tree editor TrEd might be also very useful, as well as knowledge of PML (xml-based Prague Markup Language) and especially knowledge of the layered annotation scenario of the Prague Dependency Treebank.

Ad (b): As for licensing, most TectoMT source codes are available under GNU General Public License, version 2.0, which is always explicitly noted in the files. However, the license status of the system as a whole is not formally clear at this moment, as there will always be some components in TectoMT which we are allowed to use but not to freely distribute or re-license under GNU GPL. So all TectoMT developers are asked not to distribute TectoMT as a whole or its parts outside UFAL, unless they carefully checked all the licence issues.

The users of TectoMT are kindly asked to refer to the first published work about TectoMT:

Zdeněk Žabokrtský, Jan Ptáček, and Petr Pajas. 2008. TectoMT: Highly modular MT system with tectogrammatics used as transfer layer. In Proceedings of the Third Workshop on Statistical Machine Translation, pages 167–170, Columbus, Ohio, June. Association for Computational Linguistics.

The work on this project was partially supported by 1ET101120503 and MSM 0021620838.

The implementation of TectoMT is based on the following design decisions:

As already said, TectoMT system is composed of (1) the small versioned development part, and (2) the large unversioned part called share.

The development (versioned) part is structured as follows (of course, the subdirectory listings are not complete):

The shared (unversioned) part of TectoMT is structured as follows:

In TectoMT, linguistic representations of running texts are organized in the following hierarchy:

For those, who are acquainted with the structures used in PDT 2.0, the most important difference lies in bundles: the level added between documents and trees, which comprises all layers of representation of a given sentence. As one document is stored as one physical file, all layers of language representations can be stored in one file in TectoMT (unlike in PDT 2.0).

The notion of 'layer' has a combinatorial nature in TectoMT. It corresponds not only the layer of language description as used e.g. in the Prague Dependency Treebank, but it is also specific for a given language (e.g., possible values of morphological tags are typically different for different languages) and even for how the data on the given layer were created (whether by analysis from the lower layer or by synthesis/transfer).

Thus, the set of TectoMT layers is Cartesian product {S,T} x {English,Czech} x {W,M,P,A,T}, in which:

TectoMT layers are denoted by stringifying the three coordinates: for example, analytical representation of an English sentence acquired by sentence analysis is denoted as SEnlishA. This naming convention is used on many places in TectoMT: for naming trees in a bundle (and corresponding xml elements), for naming blocks, for node identifier generating, etc.

Unlike layers in PDT 2.0, the set of TectoMT layers should not be understood as totally ordered. Of course, there is a strong intuition based the abstraction axis of languages description (SEnglishT requires more abstraction than SEnglishM), but the intuiting might not be sufficient in some cases (SEnglishP and SEnglishA represent roughly the same level of abstraction).

The linguistic structures in TectoMT are represented using the following object-oriented interface/types:

Classes TectoMT::{Document,Bundle,Node} have their own documentation, here we list only the basic methods for navigating through a TectoMT document (Perl variables such as $document are used only for illustration purposes, but there are no global/predefined variables like this in TectoMT). "Contained" objects encapsulated in "container" objects can be accessed as follows:

There are also methods for accessing the container objects from the contained objects:

There are several methods for traversing tree topology, such as

Attributes of documents, bundles or nodes can be accessed by attribute getters and setters:

$attr_name is always a string (following the Fslib conventions in the case of structured attributes, e.g. using slash in structured attributed, e.g. 'gram/gender').

New classes, with functionality specific only for some layers, can be derived from TectoMT::Node. For example, methods for accessing effective children/parents should be defined for nodes of dependency trees. Thus, there are for example classes TectoMT::Node::SEnglishA or TectoMT::Node::SCzechA offering methods get_eff_parents and get_eff_children, which are inherited from a general analytical 'abstract class' TectoMT::Node::A (which itself is derived from TectoMT::Node). Please note that the names of the 'terminal' classes are the same as the layer names. If there is no specific class defined for some layer, TectoMT::Node is used as a default for nodes on this layer.

All these classes are stored in devel/libs/core. Obviously, they are crucial for functioning of most other components of TectoMT, so their functionality should be carefully checked after any changes.

Technically, the data structures are not stored directly in TectoMT::{Document,Bundle,Node} representation, but there is an underlying representation using Petr Pajas's Fslib library. Practically the only data stored in TectoMT objects (besides some indexing) are references to Fslib objects. Combination of a new OO API (TectoMT) with the previously existing library (Fslib) used for the underlying memory representation was chosen because of the following reasons:

Outside the core libraries, there is almost no need to access the underlying Fslib representation -- the data should be accessed exclusively via the TectoMT interface (unless some very special Fslib functionality is needed). However, the underlying Fslib representation can be accessed from the TectoMT instances as follows:

The main file format used in TectoMT is TMT (.tmt ending). TMT format is an application of PML. Thus, TMT files are PML instances of a PML schema. The schema is stored in ${TMT_ROOT}/pml/tmt_schema.xml. This schema merges and changes (more or less additively) the PML schemata from PDT 2.0.

The PML schema directly renders the logical structure of data: there can be one document in one tmt-file, the document has its attributes and contains a sequence of bundles, each bundle has its attributes and contains a set of trees (named by layer names), each tree consists of nodes, which again contain attributes.

Files in the TMT format are readable by naked eye, but this is in fact useful only when writing and debugging format convertors from TMT to other formats or back. Otherwise, it is much more comfortable to view the data in TrEd.

In TectoMT, one should never write components accessing directly the TMT files (of course, with the only exception of convertors from other formats to TMT or back). Instead, the data should be accessed by the components exclusively via the above mentioned object-oriented Perl API.

In TectoMT, there is the following hierarchy of processing units (software components that process data):

Technically, blocks are Perl classes derived from TectoMT::Block. In order to make them easily readable for other TectoMT developers, please use the following conventional structure when writing new blocks:

Example of a simple block, which causes that negation particles in English will be considered as a part of verb forms during the transition from the SEnglishA layer to the SEnglishT layer:

Blocks are stored in subdirectories of the libs/blocks/ directory. Most blocks are distributed among the directories according to their position along the virtual path through the Vauquois triangle. More specifically, they are part of a transition from layer L1 to layer L2. Such blocks are stored in the <L1>_to_<L2> directory, e.g. in SEnglishA_to_SEnglishT. But there are also blocks for other purposes, e.g. evaluation blocks (libs/blocks/Eval/) or data extraction blocks (libs/blocks/Print/).

Scenarios have a strictly linear nature: the blocks are applied on tmt documents one after another, there can be no branches or cycles.

In Perl, a scenario instance can be created and applied on a TectoMT document instance as follows:

In Bash, applying a sequence of blocks on a TMT file looks e.g. as follows (brunblocks alias will be described later):

Typically, an application consists of three steps (not counting the 0th step of initialization the common development environment, as will be described later): (1) conversion of the input data into TMT, (2) applying a scenario on the TMT files, (3) conversion from TMT into the desired output format.

By Common Development Environment we understand several system variables' and aliases' settings. Such settings are recommended to be always performed in the current shell before starting work with TectoMT. There are two reasons for such initialization:

Initialization of the Common Development Environment is performed in Bash by sourcing config/init_devel_environ.sh, which manifests as follows:

If you have a .tmt file and a sequence of blocks that crashes somewhere, minimize it to speed up the loop of bug fixing or to attach it to a bug report for someone.

devel/tools/tests/auto_diagnose.pl will automatically create a minimal testcase for you: the first problematic sentence will be extracted from the .tmt file and analyzed just before the first crashing block. Finally the command line (brunblocks) to run the minimized test case is provided as a tiny shell script.

In a grid environment of Sun Grid Engine (where commands like qsub work), you can use qrunblocks to apply a scenario on a set of files in parallel.

qrunblocks is available in $TMT_ROOT/devel/tools/cluster_utils/qrunblocks.

qrunblocks filelist blocks

We successfully parsed nearly a gigaword of Czech texts (51 million sentences) and 6 million of Czech-English parallel sentences up to the t-layers in TectoMT using the following dataflow:

Results on daily tests on various platforms are available here: http://ufallab.ms.mff.cuni.cz/~bojar/cruise_control_tmt/

If you wish to receive a notification about new problems, add your e-mail address to the variable RCPT in devel/tests/Makefile.

The notification is sent only in case a test (on a particular platform) passed yesterday but fails today.

Yes, indeed. The main purpose of the test suite is to let everyone fix bugs.

If a test you need or created fails, try changing relevant files (blocks/libraries/Makefiles/...). Then run the test on command-line (see below), and if you succeed, commit!

To run a test yourself, do the following:

cd devel/tests
make nice_file_names
# or
make try_test.test_tag_tnt
# or
make try_application.demo_translation_en2cs
# or any other test

There are several ways to add a new test. Choose the method according to you additional wishes.

To have a test launched every day on all tested platforms, simply add it to the variable TESTS_TO_RUN in devel/tests/Makefile.