Motivation

• UNIX – simple tools from the beginning developed for text processing (office system)
• No need for GUI (computing machines, access from anywhere, even from a phone)
• Very large data processing (up to billions of words)
• Simple automation of repeated tasks (“scripts”)
• Self-reliance in research
• More options → new ideas

Requirements

• Homework
• Final test
• Attendance (less attendance >> more homework)
• Homeworks (well done) are required for passing the course

S131 lab at the Faculty of Arts

• your university account
• Windows + WSL
• install your own linux distribution (MS Store)

Unix

• command line + graphical interface (optional, not needed in Windows)
• shell: command interpreter
  • internal commands of the shell (builtin), and external: small separate programs
    • difference in man (where to find help)

Literature

• Libor Forst: Shell v příkladech. Matfyzpress 2010, ISBN 978-80-7378-152-1
• Learning the bash Shell
• bash Cookbook
• perldoc perlrun (meaning of command-line switches like -e, -E, -0, etc.)

Course structure

1. Processing text as a necessary basis for (not only) computational linguistics
2. Why use the Unix shell; the most basic commands
3. more commands to manipulate texts
4. text editors
5. search using regular expressions
6. using regular expressions to edit text
7. basic principles of formulation and validation of hypotheses, application to data, accuracy, completeness, value of results
8. removal of diacritics, segmentation into sentences, tokenization
9. rule automatic identification of word types
10. using current stare-of-the-art NLP toolkits (web API or local programmes)

Text Processing

• Hypothesis → data analysis (experiment) → evaluation → confirmation or not
• Analyze all available relevant data, not small samples (selection bias)
• The availability of a large number of electronic texts
• Internet (Czech internet has billions of words of text)
• Digitisation of library funds (OCR)
• New media digital first (conference proceedings, journals, …)
• Social media (Twitter Most Available - API)

SSH, PuTTY

• Terminal
• Secure (remote) shell
• unix (linux, Mac, bsd, …): command ssh in the terminal + sshd
• Many platforms (incl. all modern telephones)
• Windows: PuTTY to access remote unix or
  • Windows Subsystem for Linux (WSL) + a Linux distro of your choice from MS Store
  • full linux command line in Windows, best of both worlds
• remote server you can use from anywhere: aip.ufal.mff.cuni.cz
  • read https://aic.ufal.mff.cuni.cz/index.php/Main_Page

Shell

• Command interpreter
• builtin and external commands
• Linux, MacOS
• bash (echo $SHELL – other shells too, but bash)
• man
• passwd
• directory structure, /, $HOME (~), . , ..

Shell II (bash)

• command -p [--optional] --switch argument another_argument

Basic commands – files and directories (folders)

• ls (-l -a -R)
• cd [path]
• pwd
• mv
• cp (-r -i -u)
• rm (-r -f)
• mkdir, rmdir

Shell III

Basic commands II – text files and variables

• cat, head, tail
• less for viewing files
• environment variables:
  • PI="3,14159"
  • echo $PI
• input and output, >, >>, <

less

• viewing files. Even very large (but we have head and tail to manage the size)
• less -M shows a nice status line with line numbers
• [number]g … go to the line number (1 if not specified)
• G … go to the end of the file
• man less for many more options. Do not use “v”, because you will not know how to get out of there.

Homework

1. Who wants to use their own computers and did’t get the instalation of Ubuntu finished today, do it before the next class and test it by performing the rest of these tasks
2. You should practice the basic commands from today’s lecture. To make it a bit more fun, I have uploaded one play by a famout Czech writer Karel Capek here: https://ufal.mff.cuni.cz/~stranak/matka.txt
   • Download the file to your desktop
   • open the Ubuntu terminal and get to the Desktop, then view file with less
   • use head (and man head) to get only the metadata of the file (the lines with some text before the line “MATKA”). Save this text into a separate file.
   • use tail to save only the final list of Capek’s works (including the header “První vydání knih Karla Čapka”) and save this into another file
   • copy, move and remove some of these files. Try renaming them too (rename is a kind of a move).
   • bonus point: use the commands we learned to only isolate the line “1934 - Povětroň”