EVALD 3.0 (Evaluator of Discourse)

Documentation to EVALD 3.0 and EVALD 3.0 for Foreigners

EVALD 3.0 and EVALD 3.0 for Foreigners serve for automatic evaluation of surface coherence (cohesion) in Czech texts written by native speakers of Czech, or non-native speakers of Czech, respectively.

The evaluation part (the backend server) is implemented in Treex (http://ufal.cz/treex), a highly modular NLP framework written in the Perl programming language, and uses the Weka toolkit (http://www.cs.waikato.ac.nz/ml/weka/) for the final prediction of a coherence mark. It can be used directly from a command line or as a backend for a client. The frontend part is implemented as a web server, accessible with a web browser.

There are three possible ways of using EVALD 3.0 and EVALD 3.0 for Foreigners:

1. interactively as a web demo and RESTful web service hosted at the LINDAT/CLARIN server (available for EVALD 4.0 and EVALD 4.0 for Foreigners only!),

2. interactively but locally, with both the server and the client running on the same machine (or two machines in the same network),

3. in a batch mode run on the local machine.

1. EVALD as a LINDAT/CLARIN web service

No installation is needed in this case; in a web browser (such as Firefox or Chrome), go to https://lindat.mff.cuni.cz/services/evald/ for EVALD 3.0, or to https://lindat.mff.cuni.cz/services/evald-foreign/ for EVALD 3.0 for Foreigners.

How to use the web interface:

Write or copy the text to be tested in the textfield and click on the button "Evaluate!". The evaluation takes approx. ten seconds, then the result is displayed, e.g.:

2. EVALD as a backend and frontend server

Both the backend and the frontend components are distributed via the Docker software (https://www.docker.com/), which needs to be installed first. Docker greatly simplifies the installation process of the two components and allows them to be run on Linux-based operating systems, Windows 10, as well as Mac OS X.

How to install the app:

1. To install Docker, please follow the instructions for your operating system: MS Windows, Linux, Mac OS X. Please see the memory requirements for the EVALD backend and frontend components below and set up the docker environment accordingly.
2. Download the backend component by running:

   docker pull ufal/evald.treex-server:3.0

3. Download the frontend component by running:

   docker pull ufal/evald.php-server:3.0

How to run the app:

1. The backend component must be run with one parameter: --port=$PORT specifying the port number, on which the backend server listens to requests (e.g. 34567). The other optional parameter is --app=$APP specifying the type of application to run: L1 or native to run Evald 3.0, L2 or foreign to run Evald 3.0 for Foreigners. If this optional parameter is omitted, both applications are run and it is decided in runtime which one of them should be used. Run the following command to run the backend server in background:

   docker run -d --expose $PORT -p $PORT:$PORT -t ufal/evald.treex-server:3.0
     run_evald_server.pl --port=$PORT [--app=$APP]

2. The previous command prints out the ID of the running backend container (represented by a $ID variable in the following). Another way to find out the ID is to run the following command and look for the ID of the running ufal/evald.treex-server:3.0 component:

   docker ps

3. Find out the IP address of the running backend component (represented by a $IP_ADDR variable in the following, e.g., 172.17.0.2):

   docker inspect -f '{{ .NetworkSettings.IPAddress }}' $ID

   If you are about to execute the frontend component on a different machine in the same network, store the IP address of the machine where backend server is run to the variable $IP_ADDR.
4. The frontend component allows two parameters to be set: --L1=$L1_URL specifying the URL where a running backend component to process L1 queries is available, and analogously --L2=$L2_URL for a backend component to process L2 queries. URLs may combine the backend component's IP address, the port on which it listens, and more specifying path. The combination must be in form "http://$IP_ADDR:$PORT/$PATH" (e.g. http://172.17.0.2:34567/native). Note that if you are running a backend component without the --app=$APP parameter, the L1 application is accessible using $PATH=native whereas L2 using $PATH=foreign. Execute the frontend server in background, which will be listening to requests on a standard HTTP port 80:

   docker run -d -p 80:80 ufal/evald.php-server:3.0 run_server.sh [--L1=$L1_URL]
     [--L2=$L2_URL]

5. Open your favorite web browser and visit the webpages http://localhost/index.php or http://localhost/index-foreign.php to start working with Evald 3.0 or Evald 3.0 for Foreigners, respectively.

How to stop the app:

1. Execute the following command to find the IDs of the two running components:

   docker ps

2. Use Docker to stop the running background processes:

   docker stop $BACKEND_ID
   docker stop $FRONTEND_ID

System requirements:

• Linux with sudo privileges, Windows 10, Mac OS X
• Docker
• backend component:
  • 2.5 GB free on a hard drive, 5 GB RAM
• frontend component:
  • 500 MB free on hard drive, 30 MB RAM

3. EVALD in a batch mode

EVALD can be run in a batch mode to process just the selected documents stored as a plain text in the UTF-8 encoding and stop. There are two options how to install and use it in a batch mode: as a dockerized container, or directly as a Treex scenario.

Batch mode as a dockerized container

The dockerized backend component ufal/evald.treex-server:3.0 can be used also in a batch mode. The advantage of this approach is easy installation and the possibility to run the application on the operating systems other than Linux.

How to install the app:

Follow the instructions above to install the backend component ufal/evald.treex-server:3.0.

How to run the app:

1. Create two directories - one for input documents, and the second one for the outputs. The absolute paths to the directories are represented by variables $FROM_DIR_ABS_PATH and $TO_DIR_ABS_PATH, respectively. Put your texts into the $FROM_DIR_ABS_PATH directory.
2. To process all the texts inside the $FROM_DIR_ABS_PATH directory with the scenario specified by the $APP parameter (see above), run the following command:

   docker run -v $FROM_DIR_ABS_PATH:/from/ -v $TO_DIR_ABS_PATH:/to/ -t ufal/evald.treex-server:3.0
     run_evald.sh -f /from -t /to -a $APP

   The parameter -v directory_outside_Docker:directory_inside_Docker serves to map in a runtime a directory outside the Docker container to a specified directory inside the container, which makes all the content of the directory visible for the container. However, additional steps must be done for this feature to function in Docker for Windows. See the instructions here.

System requirements

Requirements are the same as for the backend component in the previous case.

Batch mode as a Treex scenario

In this approach the main processing tool Treex is installed without being wrapped in a Docker container. On the one hand, it allows the user to gain better control over the processing pipeline, on the other hand, installation of Treex and supporting tools is a rather complex task that is not recommended for inexperienced users. Furthermore, Treex in the version used in EVALD is not guaranteed to work out directly on the operating systems other than Linux.

How to install the app:

The instructions described here are very rough. Please contact us on evald@ufal.mff.cuni.cz to get the details and help you with the installation.

Treex (http://ufal.cz/treex) needs to be installed on the local machine, along with all (mostly CPAN) dependencies for the Czech text analysis. Treex must be in the revision tagged as EVALD_3.0 (https://github.com/ufal/treex/releases/tag/EVALD_3.0). In addition, Vowpal Wabbit 8.1.1 (https://github.com/JohnLangford/vowpal_wabbit/releases/tag/8.1.1) must be installed to the location installed_tools/ml/vowpal_wabbit-v8.1-3cf3f692/ relative to the Treex Share directory. The Treex scenario to be run is a part of the LINDAT/CLARIN EVALD 3.0 distribution (http://hdl.handle.net/11234/1-2863, file Evald-3.0.scen), or the LINDAT/CLARIN EVALD 3.0 for Foreigners distribution (http://hdl.handle.net/11234/1-2864, file Evald-3.0-Foreign.scen), respectively.

How to run the app:

Replace $FILE with the actual file name and run one of the following commands (for texts written by native or non-native speakers, respectively):

treex -Lcs from=$FILE Read::Text Evald-3.0.scen

treex -Lcs from=$FILE Read::Text Evald-3.0-Foreign.scen

The evaluation of the text (the overall score and also evaluation for individual sets of features) is printed during the run of the Treex scenario, along with probabilities of the predicted results; the first mark (unnamed feature set) represents the overall evaluation, the subsequent marks reflect individual qualities of the text, e.g.:

 - feature set:                    class: B1       probability: 0.67
 - feature set: +spell             class: B1       probability: 0.516
 - feature set: +morph             class: B1       probability: 0.79
 - feature set: +vocab             class: C2       probability: 0.65
 - feature set: +syntax            class: B1       probability: 0.69
 - feature set: +conn_qua          class: B1       probability: 0.62
 - feature set: +conn_div          class: B2       probability: 0.66
 - feature set: +pron,+coref       class: B1       probability: 0.58
 - feature set: +tfa               class: B2       probability: 0.66

System requirements:

This version of the application can be installed and run only on Linux. On the other hand, there is no need for sudo privileges and for Docker to be installed. The hardware requirements correspond to those listed above for the backend component.

Technical Support

If you have questions or need technical support, please contact evald@ufal.mff.cuni.cz.

How to cite EVALD 3.0 or EVALD 3.0 for Foreigners

• Jiří Mírovský, Michal Novák, Kateřina Rysová, Magdaléna Rysová, Eva Hajičová: EVALD 3.0 – Evaluator of Discourse. Data/software, LINDAT/CLARIN digital library, Prague, Czech Republic, http://hdl.handle.net/11234/1-2863, Nov 2018.
• Jiří Mírovský, Michal Novák, Kateřina Rysová, Magdaléna Rysová, Eva Hajičová: EVALD 3.0 for Foreigners – Evaluator of Discourse. Data/software, LINDAT/CLARIN digital library, Prague, Czech Republic,  http://hdl.handle.net/11234/1-2864, Nov 2018.

There are also papers describing the related research and experiments:

• Michal Novák, Jiří Mírovský, Kateřina Rysová, Magdaléna Rysová: Topic–Focus Articulation: A Third Pillar of Automatic Evaluation of Text Coherence. In: Advances in Computational Intelligence (LNAI 11289): 17th Mexican International Conference on Artificial Intelligence, MICAI 2018, Proceedings, Part II, Springer, Switzerland, ISBN 978-3-030-04497-8, pp. 1-14, 2018, in print.
• Michal Novák, Kateřina Rysová, Magdaléna Rysová, Jiří Mírovský: Incorporating Coreference to Automatic Evaluation of Coherence in Essays. In: Statistical Language and Speech Processing, Springer International Publishing, Cham, Switzerland, ISBN 978-3-319-68455-0, ISSN 1611-3349, pp. 58-69, 2017. WWW: https://link.springer.com/content/pdf/10.1007%2F978-3-319-68456-7_5.pdf
• Kateřina Rysová, Magdaléna Rysová, Jiří Mírovský, Michal Novák: Introducing EVALD – Software Applications for Automatic Evaluation of Discourse in Czech. In: International Conference Recent Advances in Natural Language Processing 2017: Proceedings, INCOMA Ltd., Šumen, Bulgaria, ISSN 1313-8502, pp. 634-641, 2017.
• Kateřina Rysová, Magdaléna Rysová, Jiří Mírovský: Automatic Evaluation of Surface Coherence in L2 Texts in Czech. In: Proceedings of the 28th Conference on Computational Linguistics and Speech Processing ROCLING XXVIII (2016), The Association for Computational Linguistics and Chinese Language Processing (ACLCLP), Taipei, Taiwan, ISBN 978-957-30792-9-3, pp. 214-228, 2016. WWW: http://aclweb.org/anthology/O/O16/O16-1021.pdf

Acknowledgment

EVALD 3.0 and EVALD 3.0 for Foreigners were developed at the Institute of Formal and Applied Linguistics (ÚFAL, http://ufal.mff.cuni.cz/), Faculty of Mathematics and Physics, Charles University, with the financial support of the Ministry of Culture of the Czech Republic, project Automatic Evaluation of Text Coherence in Czech (DG16P02B016, http://ufal.mff.cuni.cz/grants/evald-evaluator-discourse).