NLP Applications and tools

• Common NLP applications (areas of research):
  • machine translation
  • improving search (information extraction, retrieval, semantics, etc.)
  • dialogue systems
  • sentiment analysis
• all of this needs pre-processed data with explicit markup. Statistical methods need training (and evaluation) data.

Analysis of text

1. sentence segmentation
2. tokenisation
3. stemming, POS tagging, lematisation a morfological analysis
4. (surface) syntactic parser, chunker (= identification of clauses)
5. deep syntax (e.g. in Treex via modifying the surface parse tree)
6. other units (on various layers of description):
   • named entity recognition (NER): person, place, date, …
   • coreference (pronominal, nominal)
   • time relations (X immediately after Y and together with Z, etc.)
   • Word Sense Disambiguation (see t-lemma)

Complex analysis: A chain of many steps. Or a joint problem for a statistical system that tries to learn all of them together. (It makes sense, they influence each other more in a complex way in reality. Not realy a simple chain.)

NLP Toolkits on the web

• Generic pipelining systems
  • Weblicht:
    • chaining of web services,
    • transfer format TCF
  • Treex:
    • local or cluster run (locally distributed)
    • Perl + API wrapping standard tools + many additional blocks
    • chaining of Blocks via Scenarios
    • PML format
• Modern DNN-based Toolkits
  • Stanza
  • Trankit
  • UDPipe: (see below)
• State of the art tools: Stanza, UDPipe (when updated), Trankit
• More toolkits, like TurkuNLP, available in 🐍

LINDAT Tools

Services run at LINDAT/CLARIN as simple web applications. Each app has a clickable GUI (a HTML web form) and a REST API.

• MorphoDiTa
• NameTag
• UDPipe
  • becoming a complex analytical tools; currently “just” a parsing pipeline

NLTK

• Best documentation: NLTK Book (both paper and web)
• a platform to build NLP programs in Python,
• active community
• NLP framework: adding NLP functionality to Python
• data and models (corpora, lexical databases) are optional parts (downloadable inside the framework)
• not a ready-made state-of-the-art toolkit, but a nice way to learn
  • AMLV00058 – Programování pro korpusovou lingvistiku: Python a NLTK

Homework

Play with curl tool and examples of using it to process data in UDPipe and other LINDAT tools. See “REST API Documentation” of the tools in the links above.