<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" "http://www.oasis-open.org/docbook/xml/4.4/docbookx.dtd">
<article>
  <articleinfo>
    <title>BTrEd/NTrEd Tutorial</title>

    <author>
      <firstname>Petr</firstname>

      <surname>Pajas</surname>
    </author>
  </articleinfo>

  <sect1>
    <title>Preliminaries</title>

    <para>Both <literal>btred</literal> and <literal>ntred</literal> are TrEd
    macro processors. TrEd macros are based on Perl, so you should be familiar
    with basics of Perl programming language. Most notably, you should know
    how to deal with Perl's basic data structures (scalars, arrays, hashes)
    and references. You should also know how to write perl subroutines. Basic
    knowledge of perl object oriented programming is also required (as a
    minimum, you should know the
    <literal>$object-&gt;method(arguments)</literal> syntax).</para>
  </sect1>

  <sect1>
    <title>What BTrEd and NTrEd are and why you may need them?</title>

    <para>You might find these programs useful if you need to
    automatically process a larger amount of files you normally edit
    in TrEd. By processing, we mean programatically editing the data
    searching in the them, and so on, without user interaction.
    </para>
    <para>
    <literal>btred</literal> provides a feature-rich environment for
    all these types of tasks. It can be used either directly from the
    command line or as a server which keeps the trees in memory and
    evaluates macros on clients requests.  <literal>ntred</literal> is
    the interface between the client and a cluster of one or more
    <literal>btred</literal> servers.</para>
  </sect1>

  <sect1>
    <title>Getting started</title>

    <para>In the most usual case, you run <literal>btred</literal> as follows:
    <literallayout>btred -m <replaceable>my_macros</replaceable>
      -e <replaceable>my_function</replaceable> <replaceable>files</replaceable> ...</literallayout>
    <literal>btred</literal> first reads your macros from the files
    <replaceable>my_macros</replaceable>, then starts opening given
    <literal>files</literal> one at a time and executes function
    <replaceable>my_function</replaceable> on each of them.</para>

    <para>The file <literal>my_macros</literal> may for example look as the
    following one: <literallayout>
sub is_childless_verb {
  my ($node) = @_;
  if (not($node-&gt;firstson()) and $node-&gt;{tag}=~/^V/) {
    return 1;
  } else {
    return 0;
  }
}

sub find_childless_verbs {
  do {{
    while ($this) {
      if (is_childless_verb($this)) {
         print ThisAddress(),"\n";
      }
      $this=$this-&gt;following;
    } 
  }} while (NextTree());
}
      </literallayout></para>

    <para>To apply the above macros on a set of files, you simply type
    <literallayout>btred -m my_macros -e find_childless_verbs *.fs</literallayout>Let's
    see what happens in the code: <literal>find_childless_verbs</literal>
    iterates over all trees by calling <literal>NextTree()</literal> to move
    to the next tree. Within each tree it iterates over all nodes, using
    <literal>following</literal> method on current node
    <literal>$this</literal>. This method returns the following node in the
    recursive depth-first ordering. For each node it consults the
    <literal>is_childless_verb</literal> function declared above and if this
    function returns <literal>1</literal>, prints a string of the form
    <literal><replaceable>filename.fs</replaceable>#<replaceable>tree-no</replaceable>.<replaceable>node-position</replaceable></literal>
    generated by a default macro function called ThisAddress(). Output can be
    passed to <application>TrEd</application> as a file-list and TrEd will
    open each of the files at the exact tree and node.</para>

    <para>The code in <literal>is_childless_verb</literal> checks if the given
    node has no children (which in turn means, it has no first son, as node's
    children are all male in the terminology used) and is a verb according to
    the first letter of the attribute <literal>tag</literal>, containing
    required morphological information.</para>

    <para>It is quite apparent that most of the code of
    <literal>find_childless_verbs</literal> is spent on iterating over all
    nodes of all trees. <literal>btred</literal> can do this for you, if you
    use use the options <literal>-T</literal> (to iterate over all trees) and
    <literal>-N</literal> to iterate over all nodes within each of processed
    trees. Command-line options which don't take arguments can be put
    together, so you can write the above two as <literal>-TN</literal>.</para>

    <para>Because it is rather easy to forget to put options like
    <literal>-T</literal> or <literal>-N</literal> to the command-line, you
    can write them directly into the code as shown below. It is sometimes
    handy to include other options, such as <literal>-e
    find_childless_verbs</literal> indicating, which macro should be called on
    each iteration. The script then looks as follows: <literallayout>
	#!btred -TN -e find_childless_verbs
sub is_childless_verb {
   my ($node) = @_;
   if (not($node-&gt;firstson()) and $node-&gt;{tag}=~/^V/) {
     return 1;
   } else {
     return 0;
   }
}

sub find_childless_verbs {
   if (is_childless_verb($this)) {
      print ThisAddress(),"\n";
   }
}
      </literallayout> and can be executed simply as <literallayout>$ btred -m my_macros *.fs</literallayout></para>

    <para>If your script is so simple that you don't bother openning an
    editor, you can put your code directly on the command line. Here is an
    example equivalent to the code of <literal>my_macros</literal>.
    <literallayout>$ btred -TN -e 'print ThisAddress(),"\n" unless ($this-&gt;firstson() or $this-&gt;{tag}!~/^V/)' *.fs</literallayout></para>
  </sect1>

  <sect1>
    <title>Processing too slow? Use NTrEd!</title>

    <para>Processing a large amount of data with many incremental scripts (or
    just incremental versions of one script) may take a very long time.
    Usually, most of the time is not spent on your script but on opening,
    parsing (and possibly saving) the data files. This is where NTrEd offers
    big improvement, because it 1) allows you to utilize computing power of
    more than one machine, 2) reads the files only once and keeps them in
    memory. As a result, depending on situation, it may shorten the time
    needed for one pass from several minutes to just a few seconds.</para>

    <sect2>
      <title>Requirements</title>

      <para>NTrEd requires you to have <emphasis>password-free
      access</emphasis> to all machines you use. There are many ways to achive
      this, the most common of which are a) using <literal>ssh2</literal>
      authorization keys together with a <literal>ssh-agent2</literal>, b)
      using <literal>Kerberos</literal>, or <literal>c) .rhosts</literal>
      files.</para>

      <para>The remote machines <emphasis>must have shared
      filesystems</emphasis>. As a minimum, they should share your home
      directory and a directory containing your data.</para>
    </sect2>

    <sect2>
      <title>Getting started</title>

      <para>The hostnames of the machines you wish to use with
      <literal>NTrEd</literal> can either be specified on the command-line,
      but it is more comfortable to put them in a file called
      <literal>.ntred_serverlist</literal> contained in your home directory.
      This file should contain one or more lines of the form
      <literal><replaceable>hostname</replaceable>:<replaceable>port</replaceable></literal>.
      Empty lines are ignored, and lines starting with a hash sign
      <literal>#</literal> are treated as comments. You may specify one
      hostname several times, provided you use different ports. This is
      particularly useful to utilize the whole power of multi-processor
      systems. You may choose any port number above <literal>1024</literal>,
      but you should try to avoid collisions with other services running on
      the remote machines, including <application>btred</application>
      instances run by other users.</para>

      <para>You may always override the list by using the
      <literal>--servers</literal>.</para>

      <para>Once you have .ntred_serverlist prepared and you have configured
      your system so that you can log in to all listed servers via ssh without
      typing a password, you can try to start the servers and load some
      data:</para>

      <literallayout>$ ntred -i *.fs</literallayout>

      <para>If all went well, you see the ntred hub distributing your data
      (*.fs) among btred servers. If you use some other tool than ssh to log
      into the servers, you can specify it using <option>--ssh</option>
      option, for example:</para>

      <literallayout>$ ntred -i --ssh /usr/bin/rsh *.fs</literallayout>

      <para>Once all data files are distributed among the servers, you'll see
      the following line:</para>

      <literallayout>NTRED-HUB: Waiting for a new client request.</literallayout>

      <para>Now, open a new console (xterm, or whatever) and run the
      following:</para>

      <literallayout>$ ntred --list-files</literallayout>

      <para>On standard output, you'll see a list of open files, printed by
      each btred server. Note that the order of the files may be quite
      random. Standard error oputput contains various messages showing e.g.
      which server is being communicated by the hub etc. It's now time to run
      some more interesting code on the servers. We may start with the one
      already crafted for btred above:</para>

      <literallayout>$ ntred -TN -e 'print ThisAddress(),"\n" unless ($this-&gt;firstson() or $this-&gt;{tag}!~/^V/)'</literallayout>

      <para>You may see that this is almost identical to the btred example
      above, except that this time we don't have to specify any filenames,
      since we already have files loaded on the server (an of course, this
      time it will be significantly faster, esp. in case of large amount of
      data files). </para>

      <para>Usually, btred scripts can be reused with ntred without changed.
      You only have to remember that:</para>

      <itemizedlist>
        <listitem>
          <para>each btred server processes it's own files without
          communicating with the others, so if for example, your script does
          some statistics and prints it at the end, you'll get as many results
          as there are servers, so you'll have to collect the output and merge
          the results somehow to obtain the overall statistics for the whole
          data. </para>
        </listitem>

        <listitem>
          <para>Your scripts may modify the data on the servers. The servers
          can remember what files were changed by your scrips but you have to
          explicitly tell in your macros if you're making changes (by calling
          <literal>ChangingFile()</literal>). You can list the changed files
          with <literal>ntred --list-changed-files</literal>. The changes are
          kept in the memory only, unless you explicitly say the servers to
          save them (using <literal>ntred --save-files</literal> - saves all -
          or <literal>ntred --save-changed-files</literal>).</para>
        </listitem>

        <listitem>
          <para>You can open the data from the server's memory (with all
          changes) for inspection or manual processing in TrEd. It only
          suffices to give the full path name to a file and precede it with
          <literal>ntred://</literal> protocol prefix. You may optionally
          specify tree number after <literal>@ sign. You can also specify node
          to be made active in TrEd. For example, t</literal>o see node with
          recursive ordering 22 in the 10th tree in the file /my/data/foo.fs
          as it is stored in the servers' memory, you would issue</para>

          <literallayout>$ tred ntred:///my/data/foo.fs#1.22@10</literallayout>

          <para>For your convenience, there is a predefined macro
          <literal>Position($node)</literal> which prints the URI of the above
          form for the given node (or for <literal>$this</literal> if no node
          is specified). Here is a nice example of the power of Unix
          redirection and this toolkit: it shows nodes selected by our old
          good macro in TrEd:</para>

          <literallayout>$ ntred -TN -e 'Position unless ($this-&gt;firstson() or $this-&gt;{tag}!~/^V/) | tred -l-</literallayout>
        </listitem>

        <listitem>
          <para><literal>Most ntred</literal> commands can be used with the
          option <literal>-L</literal> which allows you to specify files (or a
          list of files) to apply the commands to. The files must be already
          open by the servers. If apropriate, you may also specify individual
          trees or even nodes using the syntax shown above. </para>
        </listitem>
      </itemizedlist>
    </sect2>
  </sect1>

  <sect1>
    <title>Quick API reference</title>

    <para>Here is a brief list of the macro API. More complete (yet still
    incomplete) information can be found in TrEd's User Manual. This list
    doesn't include contributed extensions for various specific projects, such
    as PDT.</para>

    <variablelist>
      <varlistentry>
        <term>$this</term>

        <listitem>
          <para>This variable contains the currently processed node.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>$root</term>

        <listitem>
          <para>This variable contains the root of the currently processed
          tree.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>$node-&gt;attr(<replaceable>attribute</replaceable>)</term>

        <listitem>
          <para>Get the value of node's attribute named
          <replaceable>attribute</replaceable>.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>$node-&gt;set_attr(<replaceable>attribute</replaceable>},
          <replaceable>value</replaceable>)</term>

        <listitem>
          <para>Set the value of node's attribute named
            <replaceable>attribute</replaceable> to
            <replaceable>value</replaceable>.
	  </para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>$node-&gt;children()</term>

        <listitem>
          <para>Returns a list of node's child nodes.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>$node-&gt;parent()</term>

        <listitem>
          <para>Returns node's parent node.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>$node-&gt;lbrother()</term>

        <listitem>
          <para>Returns node's left sibling node.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>$node-&gt;rbrother()</term>

        <listitem>
          <para>Returns node's right sibling node.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>$node-&gt;descendants()</term>

        <listitem>
          <para>Returns all nodes in the node's subtree
          <emphasis>not</emphasis> including the node itself.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>$node-&gt;following()</term>

        <listitem>
          <para>Returns a node following a given node in the depth-first
          recursive ordering.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>$node-&gt;previous()</term>

        <listitem>
          <para>Returns a node preceding a given node in the depth-first
          recursive ordering.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>FS()</term>

        <listitem>
          <para>Returns the current Treex::PML::FSFormat object. It is an object containing
          information of currently defined attributes and possibly their
          special semantic (e.g. which attribute is responsible for the
          topological left-to-right ordering of the tree, which attribute is
          used to mark nodes as <literal>hidden</literal>, etc).</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>$node-&gt;visible_children(FS())</term>

        <listitem>
          <para>Return all visible child nodes (i.e. not marked as hidden) of
          the given node.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>$node-&gt;visible_descendants(FS())</term>

        <listitem>
          <para>Return all visible child nodes (i.e. not marked as hidden) of
          the given node.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>NextTree()</term>

        <listitem>
          <para>Setup $root and $this to the root of the next tree in the
          current file to be processed. Returns <literal>undef</literal> if
          all trees in the current file has already been processed.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>ChangingFile()</term>

        <listitem>
          <para>Call this macro to let btred know, you're making
          changes.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>CutPaste($node_to_cut,$target_to_paste)</term>

        <listitem>
          <para>Cuts given node and pastes it on the target node so it becomes
          it's child. It dies if the target node is in the subtree of the
          source node (since in that case cut/paste operations makes no
          sense).</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>Cut($node)</term>

        <listitem>
          <para>Disconnects a given node from it's paren and siblings.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>PasteNode($node,$new_parent)</term>

        <listitem>
          <para>Make a given node a new child of a given parent node. </para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>CloneSubtree($node)</term>

        <listitem>
          <para>Return a new identical copy of the current sub-tree.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>$node = Treex::PML::Factory->createNode()</term>

        <listitem>
          <para>Create a new node.</para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>IsHidden($node)</term>

        <listitem>
          <para>Return true if a given node is marked as hidden.</para>
        </listitem>
      </varlistentry>
    </variablelist>
  </sect1>
</article>
