Setting up your Linux on Windows II – Windows Terminal

• Install modern Windows Terminal
• When it starts, it loads the PowerShell
  • just type “ubuntu” and you are in Linux
• Unicode should work out-of-the-box, including CJK characters or emoji
• SSH is built-in, so even without installing Ubuntu, just with this new Terminal, it is possible to connect to remote Unix machines.
• By default, this new Windows Terminal has nice Ctrl+C and Ctrl+V for Copy/Paste compatibility with Windows, but it means it is not possible to use Ctrl+V to escape the following character with otherwise special meaning in Terminal to input it into the Terminal directly.
  • If you want to be able to type tabulator by typing Ctrl+V + tab, go to Setting of the Windows Terminal and disable the Ctrl+V shortcut used for Paste. You can still paste with Ctrl+Shift+V

Unix warm-up

• How do I sum up a column of (not necesarry whole) numbers?
  • Let us assume it is very large (millions or billions of lines), so MS Excel is not a sensible option.
  • shell itself can do only whole-numbers arithmetics
• traditional Unix solution:

cut -f1 table-with-numbers.tsv| awk '{s+=$1}END{print s}' 

• but we can once again use perl and it is actually a bit faster:
  • it is not even really longer code, and no nead to learn awk

cut -f1 table-with-numbers.tsv| perl -nlE '$sum+=$_;END{say $sum}'

Unix warm-up II

There are 2 ways to get the combination sort |uniq -c |sort -nr accessible any time under the name of your choice as a sinle command, for instance Sort

1. an application program

We have created this program in the previous lecture (see slides 4). Now it may be called, but you have to always find it and call it directly with it’s location. To get the commabd available any time and anywhere it must be put into one of the directories the shell searches when looking for commands. Their list (separated by :) is stored in the environment variable $PATH:

env
echo $PATH

Unix warm-up II (cont.)

2. a shell function

The file $HOME/.profile includes commands executed on a startup of a shell. We can add a new one.

pico $HOME/.profile

now add this line and save the file:

function Sort { sort $@ | uniq -c | sort -nr }

in a new shell started from this moment the Sort function should be available

POS tagging (continued)

Look at the gold data (correct POS) to check a hypothesis: “I should probably make a rule for punctuation, i.e. tokens that do not start with a ‘word character’.”

cat pdt.train.3col| perl -nlE'say if /^\W/' |cut -f3| Sort | head            
172561 PUNCT
 846 SYM
  40 NUM

cat pdt.train.3col| perl -C -plE'
s/$/\tNOUN/;
s/^([.,;:?!].*)NOUN$/$1PUNCT/;

[more rules here]

'

• one perl program, multiple commands (separated by ';')

POS Tagging development

head -10000 pdt.train.3col >small-pdt.train.3col

• small and fast training data for basic work
• easy to see accuracy without calculations:

perl -ple '[my rules here]' | \
perl -nle 'print if /(.)\t\g1$/' | \
wc -l

• perldoc perlretut, read the section “Backreferences”

Calculating the accuracy from shell variables using perl

• standard numerical operators
• (exported) shell variables accessed via %ENV, e.g. $ENV{'HOME'}
• only exported shell variables available
  • try MY_TEST=jahoda; env; export MY_TEST; env in shell

u-pl0:~$ OK=157
u-pl0:~$ ALL=479
u-pl0:~$ export OK ALL
u-pl0:~$ perl -E 'say "Accuracy: ", 
        $ENV{'OK'}/($ENV{'ALL'}/100), 
        "%."'
Accuracy: 32.776617954071%.

POS Statistics per lemma

grep -i 'klaus' pdt.train.3col | Sort
 144 Klaus  Klaus   PROPN
  74 Klause Klaus   PROPN
  24 Klausem    Klaus   PROPN
  11 Klausovi   Klaus   PROPN
   9 Klausův    Klausův ADJ
   6 Klausovu   Klausův ADJ
   6 Klausova   Klausův ADJ
   3 Klausově   Klausův ADJ
   2 klausule   klauzule    NOUN
   2 Klausovy   Klausův ADJ
   2 KLAUS  Klaus   PROPN
   1 Protiklausovská    protiklausovský ADJ
   1 Klausovým  Klausův ADJ
   1 Klausovo   Klausův ADJ
   1 KLAUSE Klaus   PROPN

• It may pay off to create a rule for Klaus* \t PROPN
• Possesives are ADJ, but too infrequent