Remote work on a unix server

• From a unix terminal (linux, MacOS, Windows + Ubuntu/SuSe/…): ssh
• From Windows without linux installed: PuTTY
• Copy data using SCP (secure copy): scp (see man scp), WinSCP
• New account: change the password on the first login: passwd
• Create a directory lecture_2
• Copy the file matka.txt from a directory Capek in my home directory into lecture_2
• check the text encoding, if needed, then correct with iconv

Downloading from a web and opnening a zip file in commandline

If you are working on a remote server, you may just have the comand line. But it is all you need 😎

• wget file
  • MacOS doesn’t have wget by default, but all unixes have curl:
  • curl -O "https://raw.githubusercontent.com/kanripo/KR1h0004/master/KR1h0004_001.txt"
• there are even web browsers in terminal: lynx, links
• unzip "filename" (if you downloaded an archive)

Stream EDitor – sed perl

• Address: line nr. or a regex. 2 addresses are lines: from, to

• find and replace jsou regexes.

• perl -ne 'print if /find/'

• perl -ple 's/find/replace/g' … substitute, globally

  • -e … “one-liner” mode, i.e. a small program is passed in quotes: -e 'some program'
  • -n … cycle over all lines of input (no print)
  • -p … cycle over all lines of input + print the result to STDOUT (standard output)
  • -l … normalise line endings (Windows and Unix use different characters to end lines)
  • -C … use UTF-8 Unicode for all input and output (needed to represent non-latin characters)
    • -C must stand separately and first

sort, uniq, occurrences

• sort [-n -r -f -k] … Numerical, Reverse, Fold lowercase to uppercase, K – by which [k]olumn
• uniq [-c -i] … Count, case Insensitive
• sort | uniq -c | sort -nr … the most common and useful combo

The | “pipe” character takes the standard output (STDOUT) of one command and “pipes” it to the standard input (STDIN) of another command. You can chain commands like this almost indefinitely.

• it is very useful to pipe comands into less, if they produce long output, especially during experimentation: do something here | less

Count word forms

• use perl, sort a uniq to count frequencies of word forms in Čapek’s play Matka.
• perl -C -ple 's/ +/\n/g' matka-utf8.txt | sort | uniq -c | sort -nr | less
• perl -C -ple 's/ +/\n/g' matka-utf8.txt | sort | uniq -c | sort -nr >frekvence.slova

Why is this solution not good enough?

Segmentation does not solve interpunction. Can you improve it? (you only need the information from this class)

Homework

• go to https://github.com/kanripo,
• find the Analects by Confucius,
• download it (or just the first chapter)
• make a statistic by character counts
  • This is how to put a newline after every character, and respect Unicode:
  • perl -C -plE 's/(.)/$1\n/g;'