{ "nbformat": 4, "nbformat_minor": 0, "metadata": { "colab": { "provenance": [] }, "kernelspec": { "name": "python3", "display_name": "Python 3" }, "language_info": { "name": "python" } }, "cells": [ { "cell_type": "markdown", "metadata": { "id": "0a85e3c5" }, "source": [ "# Universal Dependencies with `udapi-python`" ] }, { "cell_type": "markdown", "metadata": { "id": "002b6b4c" }, "source": [ "This notebook introduces students of computational linguistics to the `udapi-python` library, a powerful tool for working with Universal Dependencies treebanks. We will cover installation, downloading a treebank, and performing a basic search for linguistic features." ] }, { "cell_type": "markdown", "metadata": { "id": "e35be180" }, "source": [ "## 1. Install `udapi`\n", "\n", "First, we need to install the `udapi` Python library. This can be done using `pip`." ] }, { "cell_type": "code", "metadata": { "id": "5abd4bd6", "outputId": "d42f4ed7-1e2c-4786-e891-366407f6260d", "colab": { "base_uri": "https://localhost:8080/" } }, "source": [ "%%bash\n", "# (Preceding a single line with ! makes that line interpreted by shell instead of python.\n", "# Inserting %%bash at the beginning of the cell makes the whole cell interpreted by bash.)\n", "pip install --upgrade udapi\n", "udapy -h" ], "execution_count": 1, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Collecting udapi\n", " Downloading udapi-0.5.2-py3-none-any.whl.metadata (2.0 kB)\n", "Collecting colorama (from udapi)\n", " Downloading colorama-0.4.6-py2.py3-none-any.whl.metadata (17 kB)\n", "Requirement already satisfied: termcolor in /usr/local/lib/python3.12/dist-packages (from udapi) (3.3.0)\n", "Downloading udapi-0.5.2-py3-none-any.whl (418 kB)\n", " ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 418.7/418.7 kB 7.4 MB/s eta 0:00:00\n", "Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)\n", "Installing collected packages: colorama, udapi\n", "Successfully installed colorama-0.4.6 udapi-0.5.2\n", "usage: udapy [optional_arguments] scenario\n", "\n", "udapy - Python interface to Udapi - API for Universal Dependencies\n", "\n", "Examples of usage:\n", " udapy -s read.Sentences udpipe.En < in.txt > out.conllu\n", " udapy -T < sample.conllu | less -R\n", " udapy -HAM ud.MarkBugs < sample.conllu > bugs.html\n", "\n", "positional arguments:\n", " scenario A sequence of blocks and their parameters.\n", "\n", "options:\n", " -h, --help show this help message and exit\n", " -q, --quiet Warning, info and debug messages are suppressed. Only fatal errors are reported.\n", " -v, --verbose Warning, info and debug messages are printed to the STDERR.\n", " -s, --save Add write.Conllu to the end of the scenario\n", " -T, --save_text_mode_trees\n", " Add write.TextModeTrees color=1 to the end of the scenario\n", " -H, --save_html Add write.TextModeTreesHtml color=1 to the end of the scenario\n", " -A, --save_all_attributes\n", " Add attributes=form,lemma,upos,xpos,feats,deprel,misc (to be used after -T and -H)\n", " -C, --save_comments Add print_comments=1 (to be used after -T and -H)\n", " -M, --marked_only Add marked_only=1 to the end of the scenario (to be used after -T and -H)\n", " -N, --no_color Add color=0 to the end of the scenario, this overrides color=1 of -T and -H\n", " -X EXTRA, --extra EXTRA\n", " Add a specified parameter (or a block name) to the end of the scenario\n", " For example 'udapy -TNX attributes=form,misc -X layout=align < my.conllu'\n", " --gc By default, udapy disables Python garbage collection and at-exit cleanup\n", " to speed up everything (especially reading CoNLL-U files). In edge cases,\n", " when processing many files and running out of memory, you can disable this\n", " optimization (i.e. enable garbage collection) with 'udapy --gc'.\n", "\n", "See http://udapi.github.io\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "5e91ad9b" }, "source": [ "## 2. Download and Load a UD Treebank\n", "\n", "Udapi allows easy access to Universal Dependencies treebanks. As an example, we will download several treebanks from the parallel UD collection. Feel free to add downloads of other treebanks you are interested in. See https://universaldependencies.org/ for the list of available treebanks. Besides the UD homepage, you can also try https://universaldependencies.org/languages.html, where you can filter languages by family and genus." ] }, { "cell_type": "code", "source": [ "%%bash\n", "rm -rf UD_*\n", "git clone https://github.com/UniversalDependencies/UD_Czech-PUD.git\n", "git clone https://github.com/UniversalDependencies/UD_English-PUD.git\n", "git clone https://github.com/UniversalDependencies/UD_Spanish-PUD.git\n", "git clone https://github.com/UniversalDependencies/UD_Portuguese-PUD.git\n", "git clone https://github.com/UniversalDependencies/UD_Czech-FicTree.git\n", "git clone https://github.com/UniversalDependencies/UD_Portuguese-Porttinari.git\n", "git clone https://github.com/UniversalDependencies/UD_Hungarian-Szeged.git" ], "metadata": { "id": "ariBQpI5anaA", "outputId": "ccf64302-307e-49f2-a819-3ea3f76bf4e0", "colab": { "base_uri": "https://localhost:8080/" } }, "execution_count": 2, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "Cloning into 'UD_Czech-PUD'...\n", "Cloning into 'UD_English-PUD'...\n", "Cloning into 'UD_Spanish-PUD'...\n", "Cloning into 'UD_Portuguese-PUD'...\n", "Cloning into 'UD_Czech-FicTree'...\n", "Cloning into 'UD_Portuguese-Porttinari'...\n", "Cloning into 'UD_Hungarian-Szeged'...\n" ] } ] }, { "cell_type": "markdown", "source": [ "Now we will access the treebank data from Python. The `udapi.core.document.Document` class is used to load the treebank. You can edit the code cell below and specify the treebank you want to work with (it must be one of the treebanks you downloaded above)." ], "metadata": { "id": "1Hycu-W2a0qD" } }, { "cell_type": "code", "metadata": { "id": "341d320e", "collapsed": true, "outputId": "194f0e6b-c9eb-4b27-a101-5d00fd2e7ede", "colab": { "base_uri": "https://localhost:8080/" } }, "source": [ "import glob\n", "from udapi.core.document import Document\n", "\n", "# Read the CoNLL-U file.\n", "###############################################################################\n", "# TODO: Change this variable to explore different datasets.\n", "treebank = 'UD_Portuguese-Porttinari' # Student can modify this line.\n", "\n", "# List all .conllu files in the specified treebank folder.\n", "conllu_files = glob.glob(f\"{treebank}/*.conllu\")\n", "print(f\"Found {len(conllu_files)} CoNLL-U files in {treebank}:\")\n", "for file in sorted(conllu_files):\n", " print(file)\n", "print(f\"Loading {treebank}...\")\n", "# Each CoNLL-U file will be stored as one Document object.\n", "# Note that some treebanks use the '# newdoc' comment to mark document boundaries within each file.\n", "# That is a different notion of document!\n", "filedocs = []\n", "nsent = 0\n", "for file in sorted(conllu_files):\n", " doc = Document(filename=file)\n", " filedocs.append(doc)\n", " nsent += len(doc.bundles)\n", "print(f\"Loaded {nsent} sentences from {treebank}.\")\n" ], "execution_count": 3, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Found 3 CoNLL-U files in UD_Portuguese-Porttinari:\n", "UD_Portuguese-Porttinari/pt_porttinari-ud-dev.conllu\n", "UD_Portuguese-Porttinari/pt_porttinari-ud-test.conllu\n", "UD_Portuguese-Porttinari/pt_porttinari-ud-train.conllu\n", "Loading UD_Portuguese-Porttinari...\n", "Loaded 8418 sentences from UD_Portuguese-Porttinari.\n" ] } ] }, { "cell_type": "markdown", "metadata": { "id": "8c5329d2" }, "source": [ "## 3. Find and Print Lemmas of Personal Pronouns\n", "\n", "Now, let's explore the loaded treebank. We'll iterate through each word (node) in every sentence (bundle) in every CoNLL-U file of the treebank and identify personal pronouns (UPOS tag `PRON` and PronType feature `Prs`). Then, we'll print their lemmas." ] }, { "cell_type": "code", "metadata": { "id": "2c2ab7f9", "collapsed": true, "outputId": "b92a0cdc-d68e-420e-9410-fc6e423ab78b", "colab": { "base_uri": "https://localhost:8080/" } }, "source": [ "personal_pronoun_lemmas = set()\n", "\n", "for doc in filedocs:\n", " # Bundle is a unit that corresponds to one sentence in Udapi.\n", " # In theory it may have multiple trees for that sentence; but normally there is just one.\n", " for bundle in doc.bundles:\n", " root = bundle.get_tree()\n", " nodes = root.descendants\n", " for node in nodes:\n", " # Check if the word is a pronoun (UPOS tag 'PRON')\n", " # and if its PronType feature is 'Prs' (Personal Pronoun)\n", " if node.upos == 'PRON' and node.feats['PronType'] == 'Prs':\n", " personal_pronoun_lemmas.add(node.lemma)\n", "\n", "print(f\"\\nLemmas of personal pronouns found in {treebank}:\")\n", "for lemma in sorted(list(personal_pronoun_lemmas)):\n", " print(lemma)" ], "execution_count": 4, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\n", "Lemmas of personal pronouns found in UD_Portuguese-Porttinari:\n", "ele\n", "eu\n", "lhe\n", "lo\n", "me\n", "meu\n", "mim\n", "nos\n", "nosso\n", "nós\n", "o\n", "se\n", "seu\n", "si\n", "te\n", "tu\n", "você\n" ] } ] }, { "cell_type": "markdown", "source": [ "### Exercise\n", "\n", "The following code cell is similar to the previous one but it is more general, intended as a template for simple searches. Look for the TODO comments! They mark places where you should complete the code. (You cannot just run the cell as it is now. It will not work.)" ], "metadata": { "id": "Vm2vLh97F1Sq" } }, { "cell_type": "code", "metadata": { "collapsed": true, "id": "Pb97vqNzG-E1", "outputId": "152e3336-efb0-4d98-e924-f71a059c42ff", "colab": { "base_uri": "https://localhost:8080/" } }, "source": [ "# We will collect the results of our query in the list called \"results\".\n", "results = []\n", "\n", "for doc in filedocs:\n", " # Bundle is a unit that corresponds to one sentence in Udapi.\n", " # In theory it may have multiple trees for that sentence; but normally there is just one.\n", " for bundle in doc.bundles:\n", " root = bundle.get_tree()\n", " nodes = root.descendants\n", " for node in nodes:\n", " ###############################################################################\n", " # TODO: Now it is time to set the requirements for the nodes we are looking for.\n", " # Modify the condition on the next line to reflect your search criteria.\n", " # Note that \"equals to\" is expressed by \"==\" (two equal symbols, not just one),\n", " # \"is not equal to\" is expressed by \"!=\". You can combine multiple conditions\n", " # using the logical operators \"and\", \"or\", \"not\" (and round brackets if needed).\n", " # You can query various attributes of the current node:\n", " # node.ord ... the integer identifying position of the word in the sentence (column ID in CoNLL-U)\n", " # node.form ... the word form (column FORM in CoNLL-U)\n", " # node.lemma ... the lemma of the word (column LEMMA in CoNLL-U)\n", " # node.upos ... the universal POS tag (column UPOS in CoNLL-U)\n", " # node.feats['FFFF'] ... FFFF is a name of a feature, for example node.feats['Gender']\n", " # node.deprel ... the label of the incoming relation from the parent to this node (column DEPREL in CoNLL-U)\n", " # node.misc['MMMM'] ... MMMM is a name of an attribute in the MISC column.\n", " # The inventories of attributes are specific to each treebank. An example of\n", " # an attribute that is available in multiple treebanks and can be useful is\n", " # 'Translit' (transcription/transliteration from a foreign script) and 'Gloss'\n", " # (translation of the current word to English or another major language).\n", " if node.upos == 'AUX':\n", " ###############################################################################\n", " # TODO: What do we want to remember from the node as a result? Perhaps the lemma?\n", " # Or upos? Or a feature? Any node attribute that could be used in the requirements\n", " # (see above) could also be used as the result. Replace the XXXX below.\n", " results.append(node.lemma)\n", "\n", "###############################################################################\n", "# TODO: Remember, Grew Match shows 1000 results at most. Here we can set our\n", "# limit:\n", "max_results_to_show = 10\n", "n = len(results)\n", "print(f\"Treebank {treebank}\\nFound {n} results:\")\n", "# Discard the results that are over the limit.\n", "if n > max_results_to_show:\n", " results = results[:max_results_to_show]\n", "for result in results:\n", " print(result)\n" ], "execution_count": 5, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Treebank UD_Portuguese-Porttinari\n", "Found 4806 results:\n", "ter\n", "ser\n", "ser\n", "ser\n", "estar\n", "ser\n", "ser\n", "haver\n", "ser\n", "ser\n" ] } ] }, { "cell_type": "markdown", "source": [ "## 4. Some Technical Code to Enable Clustering Tables\n", "\n", "You don't need to understand or modify the following cell but you must run it so that we can display nice clustering tables later." ], "metadata": { "id": "iErWCksIoTr5" } }, { "cell_type": "code", "metadata": { "id": "9762ff17" }, "source": [ "from collections import defaultdict\n", "import pandas as pd\n", "\n", "def generate_two_dimensional_table(clusters, display_mode='#'):\n", " \"\"\"\n", " Generates, sorts, and displays a two-dimensional table from a clusters dictionary.\n", "\n", " Args:\n", " clusters (dict): A dictionary where keys are (parent_tag, child_tag) tuples\n", " and values are counts.\n", " display_mode (str): The mode for displaying cell values.\n", " Options: '#': absolute counts,\n", " 'H%': horizontal percentage (row-wise),\n", " 'V%': vertical percentage (column-wise),\n", " '%': percentage of grand total.\n", " Returns:\n", " pandas.DataFrame: The DataFrame used for display.\n", " \"\"\"\n", " import pandas as pd\n", "\n", " # Get all unique parent and child tags\n", " parent_tags = sorted(list(set([pair[0] for pair in clusters.keys()])))\n", " child_tags = sorted(list(set([pair[1] for pair in clusters.keys()])))\n", "\n", " # Create a DataFrame to store the counts (initial data only)\n", " df_raw_counts = pd.DataFrame(0, index=parent_tags, columns=child_tags)\n", "\n", " # Populate the DataFrame with counts from the clusters dictionary\n", " for (parent, child), count in clusters.items():\n", " df_raw_counts.loc[parent, child] = count\n", "\n", " # Calculate row sums for sorting purposes\n", " row_sums_for_sorting = df_raw_counts.sum(axis=1)\n", "\n", " # Sort rows based on these sums in descending order\n", " sorted_row_indices = row_sums_for_sorting.sort_values(ascending=False).index\n", " df_raw_counts = df_raw_counts.reindex(sorted_row_indices)\n", "\n", " # Calculate column sums for sorting purposes (after row sorting)\n", " column_sums_for_sorting = df_raw_counts.sum(axis=0)\n", "\n", " # Sort columns based on these sums in descending order\n", " sorted_column_names = column_sums_for_sorting.sort_values(ascending=False).index\n", " df_raw_counts = df_raw_counts.reindex(columns=sorted_column_names)\n", "\n", " # --- Prepare DataFrame for display based on display_mode ---\n", " df_display = df_raw_counts.copy() # Start with raw counts\n", "\n", " # Calculate absolute totals for later use (these will always be absolute counts)\n", " absolute_row_totals = df_raw_counts.sum(axis=1)\n", " absolute_column_totals = df_raw_counts.sum(axis=0)\n", " absolute_grand_total = df_raw_counts.sum().sum()\n", "\n", " if display_mode == 'H%':\n", " # Horizontal percentage: percent of the total count of the row\n", " df_display = df_raw_counts.div(absolute_row_totals.replace(0, 1), axis=0) * 100\n", " df_display = df_display.round(2).fillna(0)\n", " df_display = df_display.map(lambda x: f'{x:.2f}%')\n", " elif display_mode == 'V%':\n", " # Vertical percentage: percent of the total count of the column\n", " df_display = df_raw_counts.div(absolute_column_totals.replace(0, 1), axis=1) * 100\n", " df_display = df_display.round(2).fillna(0)\n", " df_display = df_display.map(lambda x: f'{x:.2f}%')\n", " elif display_mode == '%':\n", " # Percent of the total count in the table\n", " df_display = (df_raw_counts / absolute_grand_total) * 100\n", " df_display = df_display.round(2).fillna(0)\n", " df_display = df_display.map(lambda x: f'{x:.2f}%')\n", " elif display_mode == '#':\n", " # Absolute counts (already in df_raw_counts, convert to string for display consistency)\n", " df_display = df_raw_counts.astype(str)\n", " else:\n", " raise ValueError(\"Invalid display_mode. Choose from '#', 'H%', 'V%', '%'.\")\n", "\n", " # Add the absolute Row_Total column\n", " df_display['Total'] = absolute_row_totals.astype(str)\n", "\n", " # Add the absolute Col_Total row\n", " col_total_row_for_display = absolute_column_totals.astype(str)\n", " col_total_row_for_display['Total'] = str(absolute_grand_total) # Grand total\n", " df_display.loc['Total'] = col_total_row_for_display\n", "\n", " return df_display" ], "execution_count": 6, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "zHDPvM_HRDTD" }, "source": [ "## 5. Query Properties of the Node and Its Parent\n", "\n", "So far we were only looking at the properties of one node (although one of the properties was `node.deprel`, the label of the incoming dependency relation from its parent). We can query all these properties also for the parent of the current node. For example, `node.parent.upos` denotes the UPOS category of the parent. Let's examine the UPOS tags of the parent and the child of a particular relation type." ] }, { "cell_type": "code", "metadata": { "collapsed": true, "id": "E5VrnY1iSi5k", "outputId": "44032f26-bd88-4622-b54b-ef84b6b63ab3", "colab": { "base_uri": "https://localhost:8080/", "height": 441 } }, "source": [ "# clusters will store: {(parent UPOS, child UPOS): count}\n", "clusters = defaultdict(int)\n", "\n", "for doc in filedocs:\n", " # Bundle is a unit that corresponds to one sentence in Udapi.\n", " # In theory it may have multiple trees for that sentence; but normally there is just one.\n", " for bundle in doc.bundles:\n", " root = bundle.get_tree()\n", " nodes = root.descendants\n", " for node in nodes:\n", " ###################################################################\n", " # TODO: You can modify your search criteria here.\n", " if node.deprel == 'obj' and node.parent.upos == 'VERB':\n", " ###############################################################\n", " # TODO: You can modify your clustering keys here. For example,\n", " # instead of upos, you could collect the lemma of the node and\n", " # its parent.\n", " # Get the pair of UPOS tags (first parent, then child).\n", " wordorder = 'VO' if node.parent.ord < node.ord else 'OV'\n", " upospair = (node.parent.lemma, wordorder)\n", " clusters[upospair] += 1\n", "\n", "# Now display the clusters as a table.\n", "\n", "###############################################################################\n", "# TODO: Set your preferred display mode here.\n", "# Options: '#': absolute counts\n", "# 'H%': horizontal percentage (percent of the total count of the row)\n", "# 'V%': vertical percentage (percent of the total count of the column)\n", "# '%': percent of the total count in the table\n", "display_mode = '#' # Change this variable to select the desired display mode\n", "\n", "# Generate the table of clusters.\n", "df_display = generate_two_dimensional_table(clusters, display_mode=display_mode)\n", "\n", "# Display the table\n", "print(f\"Parent UPOS (rows) vs. Child UPOS (columns) for 'nsubj' dependency (Mode: {display_mode}):\")\n", "display(df_display)\n" ], "execution_count": 13, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Parent UPOS (rows) vs. Child UPOS (columns) for 'nsubj' dependency (Mode: #):\n" ] }, { "output_type": "display_data", "data": { "text/plain": [ " VO OV Total\n", "ter 703 35 738\n", "fazer 330 39 369\n", "haver 359 7 366\n", "dar 139 6 145\n", "ver 119 16 135\n", "... ... ... ...\n", "demolir 1 0 1\n", "lapidar 1 0 1\n", "libertar 1 0 1\n", "livrar 0 1 1\n", "Total 6683 559 7242\n", "\n", "[1031 rows x 3 columns]" ], "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
VOOVTotal
ter70335738
fazer33039369
haver3597366
dar1396145
ver11916135
............
demolir101
lapidar101
libertar101
livrar011
Total66835597242
\n", "

1031 rows × 3 columns

\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", " \n", " \n", " \n", "
\n", "\n", "
\n", "
\n" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "dataframe", "variable_name": "df_display", "summary": "{\n \"name\": \"df_display\",\n \"rows\": 1031,\n \"fields\": [\n {\n \"column\": \"VO\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 52,\n \"samples\": [\n \"35\",\n \"13\",\n \"3\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"OV\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 18,\n \"samples\": [\n \"35\",\n \"39\",\n \"1\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"Total\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 52,\n \"samples\": [\n \"35\",\n \"10\",\n \"4\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}" } }, "metadata": {} } ] }, { "cell_type": "code", "metadata": { "id": "b30fe1c1", "outputId": "d7da6d49-22c1-4741-e9d7-dccbfb7a18d7", "colab": { "base_uri": "https://localhost:8080/", "height": 35 } }, "source": [ "# @title Download the table for further analysis\n", "from google.colab import files\n", "\n", "# Define the filename for the CSV file\n", "output_csv_filename = 'clusters.tsv'\n", "\n", "# Save the DataFrame to a tab-separated CSV file\n", "df_display.to_csv(output_csv_filename, sep='\\t', encoding='utf-8')\n", "\n", "print(f\"File '{output_csv_filename}' is ready for download.\")\n", "files.download(output_csv_filename)" ], "execution_count": 12, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "File 'clusters.tsv' is ready for download.\n" ] }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "application/javascript": [ "\n", " async function download(id, filename, size) {\n", " if (!google.colab.kernel.accessAllowed) {\n", " return;\n", " }\n", " const div = document.createElement('div');\n", " const label = document.createElement('label');\n", " label.textContent = `Downloading \"${filename}\": `;\n", " div.appendChild(label);\n", " const progress = document.createElement('progress');\n", " progress.max = size;\n", " div.appendChild(progress);\n", " document.body.appendChild(div);\n", "\n", " const buffers = [];\n", " let downloaded = 0;\n", "\n", " const channel = await google.colab.kernel.comms.open(id);\n", " // Send a message to notify the kernel that we're ready.\n", " channel.send({})\n", "\n", " for await (const message of channel.messages) {\n", " // Send a message to notify the kernel that we're ready.\n", " channel.send({})\n", " if (message.buffers) {\n", " for (const buffer of message.buffers) {\n", " buffers.push(buffer);\n", " downloaded += buffer.byteLength;\n", " progress.value = downloaded;\n", " }\n", " }\n", " }\n", " const blob = new Blob(buffers, {type: 'application/binary'});\n", " const a = document.createElement('a');\n", " a.href = window.URL.createObjectURL(blob);\n", " a.download = filename;\n", " div.appendChild(a);\n", " a.click();\n", " div.remove();\n", " }\n", " " ] }, "metadata": {} }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "application/javascript": [ "download(\"download_4b2297cd-5f0e-41cd-8100-edd22f30d4cd\", \"clusters.tsv\", 25991)" ] }, "metadata": {} } ] }, { "cell_type": "markdown", "metadata": { "id": "6NPuZ2aTqtHd" }, "source": [ "## 6. Query Properties of the Children of a Node\n", "\n", "So we know how to access the parent of a node. What about its children? Easy! As you may have guessed, there is a property called `node.children`. The main difference from `node.parent` is that it returns a _list_ of nodes. While all nodes (except the root) have exactly one parent, they may have more than one children (or none at all)." ] }, { "cell_type": "code", "metadata": { "collapsed": true, "id": "M1DHjmwFrexW", "outputId": "5cd5fecc-5d05-4424-fad7-82f5c6fb7ff1", "colab": { "base_uri": "https://localhost:8080/", "height": 739 } }, "source": [ "# clusters will store: {(parent UPOS, child UPOS): count}\n", "clusters = defaultdict(int)\n", "\n", "for doc in filedocs:\n", " # Bundle is a unit that corresponds to one sentence in Udapi.\n", " # In theory it may have multiple trees for that sentence; but normally there is just one.\n", " for bundle in doc.bundles:\n", " root = bundle.get_tree()\n", " nodes = root.descendants\n", " for node in nodes:\n", " # node.children is a list of nodes (the list can also be empty),\n", " # the nodes in the list are in the same order as in the sentence.\n", " # We can filter the list and get only the children that meet\n", " # particular criteria. Here we ask for children whose deprel (i.e.,\n", " # relation with the current node) is 'fixed'.\n", " fixed_children = [child for child in node.children if child.deprel == 'fixed']\n", " if len(fixed_children) > 0:\n", " # We found a fixed multiword expression! Its surface form is\n", " # the combination of the form of the current node and the forms\n", " # of the children.\n", " fixed_forms = [node.form] + [child.form for child in fixed_children]\n", " extended_forms = [node.prev_node.form] + fixed_forms + [fixed_children[-1].next_node.form]\n", " surface_form = ' '.join(extended_forms)\n", " # Fixed expressions often behave like a part-of-speech different\n", " # from the UPOS of their first word. In that case they should\n", " # have the feature ExtPos with that \"external POS\".\n", " extpos = fixed_children[0].feats['ExtPos']\n", " if not extpos:\n", " extpos = fixed_children[0].upos\n", " # For each fixed expression we will remember its extpos and\n", " # deprel (not that the incoming dependency relation for the\n", " # whole expression is found at the current node).\n", " clusters[(surface_form.lower(), extpos + ' ' + node.deprel)] += 1\n", "\n", "# Now display the clusters as a table.\n", "\n", "###############################################################################\n", "# TODO: Set your preferred display mode here.\n", "# Options: '#': absolute counts\n", "# 'H%': horizontal percentage (percent of the total count of the row)\n", "# 'V%': vertical percentage (percent of the total count of the column)\n", "# '%': percent of the total count in the table\n", "display_mode = '#' # Change this variable to select the desired display mode\n", "\n", "# Generate the table of clusters.\n", "df_display = generate_two_dimensional_table(clusters, display_mode=display_mode)\n", "\n", "# Display the table\n", "print(f\"Parent UPOS (rows) vs. Child UPOS (columns) for 'nsubj' dependency (Mode: {display_mode}):\")\n", "display(df_display)\n" ], "execution_count": 19, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Parent UPOS (rows) vs. Child UPOS (columns) for 'nsubj' dependency (Mode: #):\n" ] }, { "output_type": "display_data", "data": { "text/plain": [ " SCONJ mark ADP advmod DET mark PRON mark PRON nsubj \\\n", ", em o entanto , 0 0 0 0 0 \n", " em o entanto , 0 0 0 0 0 \n", " ainda assim , 0 0 0 0 0 \n", " enquanto isso , 0 0 0 0 0 \n", " a seguir , 0 0 0 0 0 \n", "... ... ... ... ... ... \n", "apareceu como se fosse 1 0 0 0 0 \n", "apenas em cima de você 0 0 0 0 0 \n", "apliquem mais de 5 0 1 0 0 0 \n", "aconteceu tal e qual a 0 0 0 0 0 \n", "Total 299 147 84 77 62 \n", "\n", " NOUN advmod DET advmod DET cc PRON obj PRON obl ... \\\n", ", em o entanto , 0 0 30 0 0 ... \n", " em o entanto , 0 0 14 0 0 ... \n", " ainda assim , 0 0 0 0 0 ... \n", " enquanto isso , 0 0 0 0 0 ... \n", " a seguir , 0 0 0 0 0 ... \n", "... ... ... ... ... ... ... \n", "apareceu como se fosse 0 0 0 0 0 ... \n", "apenas em cima de você 0 0 0 0 0 ... \n", "apliquem mais de 5 0 0 0 0 0 ... \n", "aconteceu tal e qual a 0 0 0 0 0 ... \n", "Total 54 48 44 41 34 ... \n", "\n", " ADJ cc PRON nsubj:pass ADV cc PRON ccomp CCONJ mark \\\n", ", em o entanto , 0 0 0 0 0 \n", " em o entanto , 0 0 0 0 0 \n", " ainda assim , 0 0 0 0 0 \n", " enquanto isso , 0 0 0 0 0 \n", " a seguir , 0 0 0 0 0 \n", "... ... ... ... ... ... \n", "apareceu como se fosse 0 0 0 0 0 \n", "apenas em cima de você 0 0 0 0 0 \n", "apliquem mais de 5 0 0 0 0 0 \n", "aconteceu tal e qual a 0 0 0 0 1 \n", "Total 3 3 3 2 1 \n", "\n", " PRON acl:relcl PRON appos PRON root SCONJ advmod Total \n", ", em o entanto , 0 0 0 0 30 \n", " em o entanto , 0 0 0 0 14 \n", " ainda assim , 0 0 0 0 7 \n", " enquanto isso , 0 0 0 0 6 \n", " a seguir , 0 0 0 0 5 \n", "... ... ... ... ... ... \n", "apareceu como se fosse 0 0 0 0 1 \n", "apenas em cima de você 0 0 0 0 1 \n", "apliquem mais de 5 0 0 0 0 1 \n", "aconteceu tal e qual a 0 0 0 0 1 \n", "Total 1 1 1 1 1062 \n", "\n", "[939 rows x 33 columns]" ], "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
SCONJ markADP advmodDET markPRON markPRON nsubjNOUN advmodDET advmodDET ccPRON objPRON obl...ADJ ccPRON nsubj:passADV ccPRON ccompCCONJ markPRON acl:relclPRON apposPRON rootSCONJ advmodTotal
, em o entanto ,00000003000...00000000030
<root> em o entanto ,00000001400...00000000014
<root> ainda assim ,0000000000...0000000007
<root> enquanto isso ,0000000000...0000000006
<root> a seguir ,0000000000...0000000005
..................................................................
apareceu como se fosse1000000000...0000000001
apenas em cima de você0000000000...0000000001
apliquem mais de 50100000000...0000000001
aconteceu tal e qual a0000000000...0000100001
Total2991478477625448444134...3332111111062
\n", "

939 rows × 33 columns

\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", " \n", " \n", " \n", "
\n", "\n", "
\n", "
\n" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "dataframe", "variable_name": "df_display" } }, "metadata": {} } ] }, { "cell_type": "code", "metadata": { "id": "tSDxHxcpv6Ml", "outputId": "a8d37968-8e73-4b0c-ca3a-fdea2898a239", "colab": { "base_uri": "https://localhost:8080/", "height": 35 } }, "source": [ "# @title Download the table for further analysis\n", "from google.colab import files\n", "\n", "# Define the filename for the CSV file\n", "output_csv_filename = 'clusters.tsv'\n", "\n", "# Save the DataFrame to a tab-separated CSV file\n", "df_display.to_csv(output_csv_filename, sep='\\t', encoding='utf-8')\n", "\n", "print(f\"File '{output_csv_filename}' is ready for download.\")\n", "files.download(output_csv_filename)" ], "execution_count": 18, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "File 'clusters.tsv' is ready for download.\n" ] }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "application/javascript": [ "\n", " async function download(id, filename, size) {\n", " if (!google.colab.kernel.accessAllowed) {\n", " return;\n", " }\n", " const div = document.createElement('div');\n", " const label = document.createElement('label');\n", " label.textContent = `Downloading \"${filename}\": `;\n", " div.appendChild(label);\n", " const progress = document.createElement('progress');\n", " progress.max = size;\n", " div.appendChild(progress);\n", " document.body.appendChild(div);\n", "\n", " const buffers = [];\n", " let downloaded = 0;\n", "\n", " const channel = await google.colab.kernel.comms.open(id);\n", " // Send a message to notify the kernel that we're ready.\n", " channel.send({})\n", "\n", " for await (const message of channel.messages) {\n", " // Send a message to notify the kernel that we're ready.\n", " channel.send({})\n", " if (message.buffers) {\n", " for (const buffer of message.buffers) {\n", " buffers.push(buffer);\n", " downloaded += buffer.byteLength;\n", " progress.value = downloaded;\n", " }\n", " }\n", " }\n", " const blob = new Blob(buffers, {type: 'application/binary'});\n", " const a = document.createElement('a');\n", " a.href = window.URL.createObjectURL(blob);\n", " a.download = filename;\n", " div.appendChild(a);\n", " a.click();\n", " div.remove();\n", " }\n", " " ] }, "metadata": {} }, { "output_type": "display_data", "data": { "text/plain": [ "" ], "application/javascript": [ "download(\"download_a5838868-fd52-42a5-ac91-225007c848de\", \"clusters.tsv\", 7206)" ] }, "metadata": {} } ] }, { "cell_type": "markdown", "source": [ "## 7. Collect All Forms of All Lemmas\n", "\n", "This will help us to use various criteria to select and display paradigms of individual lexemes." ], "metadata": { "id": "ns5_9deBHmMS" } }, { "cell_type": "code", "source": [ "# dictionary will store: {UPOS: {lemma: {form: {featstring: count}}}}\n", "dictionary = defaultdict(lambda: defaultdict(lambda: defaultdict(lambda: defaultdict(int))))\n", "all_lemmas = {}\n", "all_forms = {}\n", "\n", "for doc in filedocs: # Iterate through each Document in the list of loaded CoNLL-U files\n", " for bundle in doc.bundles: # Iterate through bundles within each Document\n", " root = bundle.get_tree()\n", " nodes = root.descendants\n", " for node in nodes:\n", " # Exclude nodes marked as typos\n", " if not (node.feats and node.feats.get('Typo') == 'Yes'):\n", " upos = node.upos\n", " lemma = node.lemma.lower()\n", " form = node.form.lower()\n", " # The feature ExtPos, if present, is not interesting.\n", " # It is not about morphology but about syntax of the sentence.\n", " # Note: This will erase the feature from the loaded document, so we will not be able to work with it. It will however not destroy it in the file on the disk.\n", " node.feats['ExtPos'] = ''\n", " featstring = str(node.feats) if node.feats else 'NO_FEATS'\n", " dictionary[upos][lemma][form][featstring] += 1\n", " all_lemmas[lemma] = True\n", " all_forms[form] = True\n", "\n", "print(f\"Collected {len(all_forms)} distinct forms and {len(all_lemmas)} distinct lemmas in total.\")\n", "print(f\"This makes the average morphological richness {len(all_forms)/len(all_lemmas)} forms per lemma.\")" ], "metadata": { "collapsed": true, "id": "8dzIOUjLq2mZ" }, "execution_count": null, "outputs": [] }, { "cell_type": "code", "source": [ "# @title Select the paradigm with most forms and print it.\n", "import pandas as pd\n", "\n", "maxforms = 0\n", "winners = []\n", "for upos in dictionary:\n", " uposdict = dictionary[upos]\n", " for lemma in uposdict:\n", " lemmadict = uposdict[lemma]\n", " nforms = len(lemmadict)\n", " # We can choose to consider only one UPOS category.\n", " #if upos != 'NOUN':\n", " # continue\n", " if nforms > maxforms:\n", " maxforms = nforms\n", " winners = [(upos, lemma)]\n", " elif nforms == maxforms:\n", " winners.append((upos, lemma))\n", "print(f\"Maximum number of forms found in one lexeme is {maxforms}.\")\n", "print(f\"It was observed with the following UPOS + LEMMA combinations:\")\n", "for upos, lemma in winners:\n", " print(f\"{upos} {lemma}\")\n", "\n", "################################################################################\n", "# TODO: Change this to == 1 if you only want to see unique winners.\n", "if len(winners) >= 1:\n", " print()\n", " selected_upos = winners[0][0]\n", " selected_lemma = winners[0][1]\n", " ############################################################################\n", " # TODO: You can modify the two variables above! We have identified the lemma\n", " # with the most distinct forms so that we can illustrate the morphology as\n", " # broadly as possible. But if you are interested in another lexeme, just\n", " # uncomment the following two lines and enter the UPOS and LEMMA you want to\n", " # see!\n", " # selected_upos = 'NOUN'\n", " # selected_lemma = 'house'\n", " lemmadict = dictionary[selected_upos][selected_lemma]\n", "\n", " # Prepare data for a DataFrame\n", " table_data = []\n", " for form in sorted(lemmadict):\n", " for featstring in sorted(lemmadict[form]):\n", " table_data.append({\n", " 'Form': form,\n", " 'Features': featstring,\n", " 'Count': lemmadict[form][featstring]\n", " })\n", "\n", " df_paradigm = pd.DataFrame(table_data)\n", " print(f\"Paradigm for {selected_upos} '{selected_lemma}':\")\n", "\n", " # Manually format for left-aligned first two columns and right-aligned last column\n", " if not df_paradigm.empty:\n", " # Calculate max widths for 'Form' and 'Features' columns\n", " max_form_width = max(len(str(x)) for x in df_paradigm['Form'].tolist() + ['Form'])\n", " max_features_width = max(len(str(x)) for x in df_paradigm['Features'].tolist() + ['Features'])\n", " max_count_width = max(len(str(x)) for x in df_paradigm['Count'].tolist() + ['Count'])\n", "\n", " # Print header\n", " print(f\"{('Form').ljust(max_form_width)} {('Features').ljust(max_features_width)} {('Count').rjust(max_count_width)}\")\n", " # Print separator\n", " print(f\"{'-'*max_form_width} {'-'*max_features_width} {'-'*max_count_width}\")\n", "\n", " # Print data rows\n", " for index, row in df_paradigm.iterrows():\n", " lemmamark = ' <=== LEMMA' if row['Form'] == selected_lemma else ''\n", " print(f\"{str(row['Form']).ljust(max_form_width)} {str(row['Features']).ljust(max_features_width)} {str(row['Count']).rjust(max_count_width)}{lemmamark}\")\n" ], "metadata": { "id": "Ce0sCa157Xsk" }, "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": { "id": "44151faf" }, "source": [ "## Tips for other things to show\n", "\n", "Run the commandline interface to Udapi, capture its HTML output, then either show it in a notebook cell or let the user download it." ] }, { "cell_type": "code", "metadata": { "id": "e385b881" }, "source": [ "from IPython.display import HTML\n", "from google.colab import files\n", "\n", "# Capture HTML output from a shell command in a variable.\n", "###!!! Saving this as udapi_output.html will not work because it mixes STDOUT with STDERR, and the latter is not even in HTML.\n", "#html_output = !cat UD_Czech-PUD/cs_pud-ud-test.conllu | udapy -HAM util.Mark node='node.upos == \"PRON\"'\n", "!cat UD_Czech-PUD/cs_pud-ud-test.conllu | udapy -HAM util.Mark node='node.upos == \"PRON\"' > udapi_output.html\n", "\n", "# html_output is a list of strings, we have to join them into one string.\n", "#html_content = \"\\n\".join(html_output)\n", "\n", "# Show the HTML content in a notebook cell.\n", "#HTML(html_content)\n", "\n", "# Save the HTML content in a file in the virtual machine.\n", "output_filename = 'udapi_output.html'\n", "#with open(output_filename, 'w') as f:\n", "# f.write(html_content)\n", "\n", "# Offer the file for download.\n", "print(f\"File '{output_filename}' is ready for download.\")\n", "files.download(output_filename)\n" ], "execution_count": null, "outputs": [] } ] }