Prague Arabic Dependency Treebank ++

This Blog Has Moved

noreply@blogger.com (Otakar Smrz) — Sun, 21 Mar 2010 14:57:00 +0000

This blog is now located at http://padt-online.blogspot.com/. You will be automatically redirected in 30 seconds, or you may click here. For feed subscribers, please update your feed subscriptions to http://padt-online.blogspot.com/feeds/posts/default.

ElixirFM 1.1 Update + Wiki + API

noreply@blogger.com (Otakar Smrz) — Wed, 13 Jan 2010 00:38:00 +0000

The ElixirFM Functional Arabic Morphology project has released an update of its libraries, executables, data, and documentation at SourceForge.

The current version 1.1.927 includes important improvements in the performance of the system and comes with enhanced user and programming interfaces. Next to the ElixirFM Online Interface, the project also features:

ElixirFM Wiki: documentation for the project has been set up, which now brings notable information for the computational linguists and interested developers who would like to explore the ElixirFM system more deeply and use it in their applications
ElixirFM API: there is a powerful ElixirFM programming interface for Perl which allows you to invoke the elixir executable from your code and further parse and process the results easily

The ElixirFM lexicon has been extended and refined, and a number of words have been encoded in a way that makes their deep word structure more explicit. The sources of the lexicon plus the editing software are available freely upon request.

ElixirFM now operates more smoothly in all its modes. In particular, the resolve mode involves solution pruning and its morphological analyses now comply with most linguistic constraints. Likewise, the online inflect and derive modes have been integrated with lookup, due to which word form generation becomes much more intuitive and yet more enjoyable.

ElixirFM is published under the GNU General Public License GNU GPL 3. Everyone is welcome to participate in this project!

ElixirFM 1.1 Online Interface

noreply@blogger.com (Otakar Smrz) — Tue, 03 Mar 2009 22:52:00 +0000

In the recent months, the ElixirFM project has undergone considerable improvement in various respects. We have worked most on developing the programming library and on refining the lexicon. On top of these essential components, we have built a user-friendly web application, the ElixirFM 1.1 Online Interface.

ElixirFM is a computational model of the morphology of Modern Written Arabic. It provides the user with four different modes of operation, in addition to the unique lexical resource and the other open-source functions of the implementation.

Resolve: provides tokenization and morphological analysis of the inserted text, even if you omit some symbols or do not spell everything correctly. You can experiment with entering the text not only in the original script and orthography, but also in other notations, including a purely phonetic transcription.
Inflect: lets you inflect words into the forms required by context. You only need to define the grammatical parameters of the expected word forms. You can either enter natural language descriptions, or you can specify the parameters using the positional morphological tags.
Derive: lets you derive words of similar meaning but different grammatical category. You only need to tell the desired grammatical categories, using either natural language descriptions, or the positional morphological tags.
Lookup: can lookup lexical entries by the citation form and nests of entries by the root. You can even search the dictionary using English.

The online interface includes example queries for each of the modes. It further incorporates several interactive tools to facilitate the browsing of the results returned by the system.

Information on the programming libraries and the research context of the project is in part available in our papers. Yet, we would like to extend the documentation according to the requirements of the users, and would be happy to discuss any unclear issues with anyone interested.

ElixirFM is published under the GNU General Public License GNU GPL 3. Everyone is welcome to participate in this project!

Enjoy ... and let us know in case of questions or comments :)

SourceForge Projects

noreply@blogger.com (Otakar Smrz) — Wed, 09 Jul 2008 09:20:00 +0000

The SourceForge open-source software repository offers a number of projects related to computational processing of Arabic:

ElixirFM: High-level implementation of Functional Arabic Morphology
Encode Arabic: Implementations for encodings of Arabic, in Haskell and Perl
AraMorph: Buckwalter Arabic morphological analyzer
Arabic WordNet: Multi-lingual concept dictionary mapping word senses in Arabic to those in the English Princeton WordNet
Sarf: Arabic morphology system that can generate and inflect Arabic verbs, derivative nouns, and gerunds
Arabic Spellchecker Word Lists: Arabic word list for spell checkers

Users can register with SourceForge and subscribe to the monitoring service of every project, in order to receive notifications of new updates.

A Word on the Million Words

noreply@blogger.com (Otakar Smrz) — Fri, 02 May 2008 13:34:00 +0000

Work on the new PADT 2.0 is now in progress. The recent developments are described in our submission to the LREC 2008 Workshop on Arabic & Local Languages:

Prague Arabic Dependency Treebank: A Word on the Million Words: [paper]

According to the paper, the expected contents of PADT 2.0 will include these annotations:

PADT 2.0 Corpus Fun. Morphology Dep. Syntax Tectogrammatics Notes

Total 1,095,610 1,281,858 1,001,908 30,894 merged annotations

Prague 328,240 383,482 282,252 30,894 original annotations

Penn 767,370 898,376 719,656 converted annotations

Prague Corpus Fun. Morphology Dep. Syntax Tectogrammatics Notes

AEP 99,360 116,717 116,717 9,690 Arabic English Parallel News

EAT 48,371 55,097 55,097 13,934 English-Arabic Treebank

ASB 11,881 14,254 14,254 Arabic Gigaword

NHR 21,445 25,329 12,613 Arabic Gigaword

HYT 85,683 100,537 41,855 5,228 Arabic Gigaword

XIN 61,500 71,548 41,716 2,042 Arabic Gigaword

Penn Corpus Fun. Morphology Dep. Syntax Tectogrammatics Notes

1v3 151,546 172,386 172,386 Penn Arabic Treebank 1v3

2v2 141,515 161,217 161,217 Penn Arabic Treebank 2v2

3v2 335,250 394,466 394,466 Penn Arabic Treebank 3v2

4v1 149,784 178,720 Penn Arabic Treebank 4v1

PADT 2.0	Corpus	Fun. Morphology	Dep. Syntax	Tectogrammatics	Notes
Total	1,095,610	1,281,858	1,001,908	30,894	merged annotations
Prague	328,240	383,482	282,252	30,894	original annotations
Penn	767,370	898,376	719,656		converted annotations
Prague	Corpus	Fun. Morphology	Dep. Syntax	Tectogrammatics	Notes
AEP	99,360	116,717	116,717	9,690	Arabic English Parallel News
EAT	48,371	55,097	55,097	13,934	English-Arabic Treebank
ASB	11,881	14,254	14,254		Arabic Gigaword
NHR	21,445	25,329	12,613		Arabic Gigaword
HYT	85,683	100,537	41,855	5,228	Arabic Gigaword
XIN	61,500	71,548	41,716	2,042	Arabic Gigaword
Penn	Corpus	Fun. Morphology	Dep. Syntax	Tectogrammatics	Notes
1v3	151,546	172,386	172,386		Penn Arabic Treebank 1v3
2v2	141,515	161,217	161,217		Penn Arabic Treebank 2v2
3v2	335,250	394,466	394,466		Penn Arabic Treebank 3v2
4v1	149,784	178,720			Penn Arabic Treebank 4v1

Your suggestions and comments are very welcome. Thank you.

Resolve Online Interface

noreply@blogger.com (Otakar Smrz) — Thu, 11 Oct 2007 23:19:00 +0000

The resolve function of ElixirFM has been made accessible via this online interface.

You can enter Arabic words in various notations, including the genuine orthography and the most popular transliterations. Symbols for vowels or diacritics can be omitted. The words will be analyzed as to their inflectional features as well as morphological structure.

Example requests are provided.

Enjoy ... and let us know in case of questions or comments :)

Functional Arabic Morphology Thesis

noreply@blogger.com (Otakar Smrz) — Wed, 19 Sep 2007 08:05:00 +0000

I am pleased to announce that I have completed and defended the doctoral thesis on the novel computational model of Arabic inflectional and derivational morphology:

Functional Arabic Morphology. Formal System and Implementation: [thesis] [summary] [slides]

The ElixirFM implementation is available via SourceForge.

We intend to improve the work further and integrate ElixirFM closely with MorphoTrees as well as with both levels of syntactic representation in the Prague Arabic Dependency Treebank.

ElixirFM and Functional Arabic Morphology

noreply@blogger.com (Otakar Smrz) — Thu, 03 May 2007 16:26:00 +0000

The ACL 2007 Workshop on Computational Approaches to Semitic Languages: Common Issues and Resources has accepted our submission:

ElixirFM — Implementation of Functional Arabic Morphology: [paper] [www]

The article is in its final version for the proceedings, but anyway, your suggestions or comments are very welcome. Thank you.

CoNLL Shared Task 2007

noreply@blogger.com (Otakar Smrz) — Wed, 17 Jan 2007 00:51:00 +0000

The CoNLL Shared Task 2007 has been announced. The extended data of PADT will be used in the competition, and we provide their rough characteristics:

Total 116,800 tokens 3,044 trees 378 files annotated on the levels of analytical syntax and morphology

AEP 9,500 tokens 242 trees 29 files Arabic English Parallel News

AFE 13,000 tokens 411 trees 48 files Arabic 10K-word English Translation

ALH 14,500 tokens 312 trees 41 files Arabic Gigaword

ANN 12,500 tokens 209 trees 17 files Arabic Gigaword

HYT 25,500 tokens 457 trees 47 files Arabic Gigaword

XIA 26,500 tokens 888 trees 111 files Arabic Gigaword

XNH 15,000 tokens 525 trees 85 files Arabic Gigaword

Total	116,800 tokens	3,044 trees	378 files	annotated on the levels of analytical syntax and morphology
AEP	9,500 tokens	242 trees	29 files	Arabic English Parallel News
AFE	13,000 tokens	411 trees	48 files	Arabic 10K-word English Translation
ALH	14,500 tokens	312 trees	41 files	Arabic Gigaword
ANN	12,500 tokens	209 trees	17 files	Arabic Gigaword
HYT	25,500 tokens	457 trees	47 files	Arabic Gigaword
XIA	26,500 tokens	888 trees	111 files	Arabic Gigaword
XNH	15,000 tokens	525 trees	85 files	Arabic Gigaword

This year's data differ from the last year's set in two important respects:

The extent and the quality of annotations have improved. We added new data sources, esp. AEP and AFE (with paragraph-aligned translations available). Other data sources are the newspaper texts published by Al Hayat, An Nahar, Ummah Press Service, and Xinhua.
The morphology of the former data has been reannotated using MorphoTrees, so that the format of all data is consistent now and the informativity of the morphological tags is considerably higher. Lemmas and glosses based on the Buckwalter lexicon are also provided.

The morphological class identifiers consist of the part-of-speech category and its refinement, and their meanings read:

VI VP VC imperfect, perfect, and imperative verb forms N- A- D- nouns, adjectives, and adverbs

C- P- I- conjunctions, prepositions, interjections G- Q- Y- graphical symbols, numbers, abbreviations

F- FN FI particles, esp. negative and interrogative S- SD SR pronouns, esp. demonstrative and relative

-- isolated definite articles Z- proper names

The attributes and morphosyntactic features associated with individual tokens, i.e. the nodes in the dependency tree, include the following kinds of information. A feature can be linguistically applicable but unresolved by the annotation, in which case it is not listed with the token:

Mood Indicative, Subjunctive, or Jussive of imperfect verbs, with D if undecided between S and J

Voice Active or Passive

Person 1 speaker, 2 addressee, 3 others

Gender morphologically overt 'gender', Masculine or Feminine

Number morphologically overt 'number', Singular, Dual, or Plural

Case 1 nominative, 2 genitive, 4 accusative

Defin morphological 'definiteness', Indefinite, Definite, Reduced, or Complex

MemberOf member of syntactic Coordination or Apposition

ClauseHead the token is the head of the given type of a subordinate clause

GramCoref the pronoun S- is a grammatical coreferent, unlike other pronouns that are textual coreferents

InputForm the token is the first of all that analyze the given orthographical word in the original input

TokenGloss a clue to the morphological structure of the token

The inventory of analytical dependency functions is further explained in one document or another:

Pred verbal predicate Coord coordination

Pnom nominal predicate Apos apposition

PredE existential predicate Ante anteposition

PredC conjunction as the clause's head AuxC conjunction

PredP preposition as the clause's head AuxP preposition

Sb subject AuxE emphasizing expression

Obj object AuxM modifying expression

Adv adverbial AuxY auxiliary, part of compound

Atr attribute AuxG graphical symbol

Atv complement AuxK sentence separator

ExD ellipsis, no actual dependency _ excessive token, esp. due to typo

The conversion script from the original FS format to the CoNLL format produces files with the .conll extension. The script is run as follows:

  btred -Qm padt-conll.btred syntax/*.syntax.fs
  mkdir conll
  mv syntax/*.syntax.fs.conll conll/

The data use the UTF-8 encoding as required. It might however be preferred to view the data in the Buckwalter transliteration, if rendering the Arabic script poses some problems. We recommend using the Encode Arabic libraries in Perl or Haskell to easily convert the data.

For using the Perl library from a command line, a code like this would do:

  # calling the module's functions in a one-liner

  cat PADT-data-in-CoNLL-format | \
      perl -MEncode::Arabic -pe '$_ = encode "buckwalter", decode "utf8", $_'

  # running the scripts installed with the module

  cat PADT-data-in-CoNLL-format | encode "buckwalter"

To use the module for reducing the vocalization, or to choose the XML-compliant variant of the Buckwalter transliteration, one can set the modes of conversion easily. Consider e.g. the following script, which removes any vocalization marks from the tokenized word forms supplied in the second column of the CoNLL data:

  use Encode::Arabic ':modes';

  enmode "buckwalter", 'full', 'xml';
  demode "buckwalter", 'noneplus', 'xml';

  while ($line = <>) {

      @cols = split /\t/, decode "utf8", $line;

      if (@cols < 2) {

          print $line;
          next; 
      }

      unless ($cols[1] =~ /[\x{20}-\x{7F}]/) {

          $in_buck = encode "buckwalter", $cols[1];
          $cols[1] = decode "buckwalter", $in_buck;
         
          warn $in_buck . "\n";
      }
      
      print encode "utf8", join "\t", @cols;
  }

More examples are available in the CPAN documentation.

The link to the last year's CoNLL-X and the proceedings ...

Prague Treebanking for Everyone: Video Lectures

noreply@blogger.com (Otakar Smrz) — Sat, 30 Dec 2006 15:36:00 +0000

Within the Vilem Mathesius Lecture Series 21, there was a two-day tutorial Prague Treebanking for Everyone. Now, everyone has the chance to view the video records of the lectures online.

The talk on MorphoTrees and the other levels of annotation in the Prague Arabic Dependency Treebank project is available as a video stream or in other video formats, along with the original slides and the printable handout.

Tips and Tricks

noreply@blogger.com (Otakar Smrz) — Thu, 26 Oct 2006 14:10:00 +0000

The Challenge of Arabic for NLP/MT conference organized in London by The British Computer Society is over. Let us give a couple of new links:

Tips and Tricks of the Prague Arabic Dependency Treebank: [paper] [slides] Otakar Smrž
An Ambiguity-Controlled Morphological Analyzer for Modern Standard Arabic Modeling Finite State Networks: [www] Mohammed A. Attia
Standard Arabic formalization and linguistic platform for its analysis: [www] Slim Mesfar

In this context, the work on morphology by Columbia's Arabic Dialect Modeling Group should also be noted:

MAGEAD: A Morphological Analyzer and Generator for the Arabic Dialects: [pdf] Nizar Habash and Owen Rambow
Morphological Analysis and Generation for Arabic Dialects: [pdf] Nizar Habash, Owen Rambow and George Kiraz

Information Structure with the Prague Arabic Dependency Treebank

noreply@blogger.com (Otakar Smrz) — Tue, 19 Sep 2006 04:41:00 +0000

Written for the conference on Communication and Information Structure in Spoken Arabic, our contribution is now online:

Information Structure with the Prague Arabic Dependency Treebank: [paper] [slides]

The article is being reviewed and edited for the proceedings of the conference. Your suggestions and comments are very welcome. Thank you.

Encode Arabic

noreply@blogger.com (Otakar Smrz) — Sun, 04 Jun 2006 10:03:00 +0000

Encoding the Arabic language in the ArabTeX transliteration, a non-trivial and multi-purpose notation for Arabic orthographies and phonetic transcriptions, brings improved possibilities for natural language processing systems.

Relying on the advanced techniques of functional parsing in Haskell, we present an implementation of an interpreter for this notation. The developed programming library will be published shortly, complementing the current Encode Arabic implementation in Perl.

Encode Arabic: Exercise in Functional Parsing: [submission version]

This paper has been submitted to the Haskell Workshop of the International Conference on Functional Programming 2006.

Open Source Arabic Grammars

noreply@blogger.com (Otakar Smrz) — Wed, 01 Mar 2006 13:31:00 +0000

Ali El Dada from Chalmers University of Technology in Gothenburg, Sweden, is developing an interesting thing:

Implementing an Open Source Arabic Resource Grammar in Grammatical Framework: [paper] [abstract] Ali El Dada and Aarne Ranta

To this topic, but from the view point of other distinct projects, the following links can be given:

Topologische Dependenzgrammatik fürs Arabische: [report] [slides] Marwan Odeh
PAPPI: A Multilingual Parser for the Principles-and-Parameters Framework: [www] Sandiway Fong

Treebanking and Advanced Processing of Arabic

noreply@blogger.com (Otakar Smrz) — Sun, 26 Feb 2006 14:54:00 +0000

In early December 2006, the Institute of Formal and Applied Linguistics, Charles University in Prague, is organizing a two-week series of workshops and invited lectures, which include the Fifth Workshop on Treebanks and Linguistic Theories (TLT 2006), the Vilem Mathesius Courses (VMC 2006), and the Partnerships for International Research and Education (PIRE) group meeting.

The proposed TAPA 2006 workshop should open this series of events on November 30, 2006. Its purpose is to bring together people from different areas of the Natural Language Processing community, who are either interested in the problem of multi-level linguistic description of Arabic, or concerned with the resources, tools and methods used recently in the study of this language.

The organizers would like to see this workshop on Treebanking and Advanced Processing of Arabic both as an opportunity for the invited research teams to promote their relevant scientific projects, and as an open opportunity for other teams of the community to report on their original approaches or derived applications. The workshop should become a forum for explanation and informed discussion.

Combining one's participation in TAPA 2006 with the other events is expected and highly recommended, of course.

More information on the website ...

Communication and Information Structure in Spoken Arabic

noreply@blogger.com (Otakar Smrz) — Mon, 30 Jan 2006 17:41:00 +0000

The Center for Advanced Study of Language, University of Maryland, organize a conference on Communication and Information Structure in Spoken Arabic, which we would like to join:

Information Structure with the Prague Arabic Dependency Treebank: [abstract]

The Other Arabic Treebank: Prague Dependencies and Functions

noreply@blogger.com (Otakar Smrz) — Fri, 20 Jan 2006 19:04:00 +0000

There is a new overview of the theories behind the Prague Arabic Dependency Treebank:

The Other Arabic Treebank: Prague Dependencies and Functions: [draft]

It is to appear in Arabic Computational Linguistics: Current Implementations (edited by Ali Farghaly), CSLI Publications, 2008.

Reviews and comments to the draft version are very welcome. Thank you.

CoNLL-X Task: Multi-lingual Dependency Parsing

noreply@blogger.com (Otakar Smrz) — Tue, 03 Jan 2006 14:05:00 +0000

PADT 1.0 is going to be used by CoNLL-X, the Tenth Conference on Natural Language Learning, within their shared task of multi-lingual dependency parsing.

We recommend the users to consult the following documents, should they need further explanation to our treebank:

Learning to Use the Prague Arabic Dependency Treebank: [paper]
Feature-Based Tagger of Approximations of Functional Arabic Morphology: [paper] [slides]

CoNLL-X have their own wiki page and a discussion forum, too.

PADT ++

noreply@blogger.com (Otakar Smrz) — Mon, 02 Jan 2006 20:58:00 +0000

This website provides information on Prague Arabic Dependency Treebank and other projects in Arabic computational linguistics and natural language processing.

`VI VP VC`	imperfect, perfect, and imperative verb forms	`N- A- D-`	nouns, adjectives, and adverbs
`C- P- I-`	conjunctions, prepositions, interjections	`G- Q- Y-`	graphical symbols, numbers, abbreviations
`F- FN FI`	particles, esp. negative and interrogative	`S- SD SR`	pronouns, esp. demonstrative and relative
`--`	isolated definite articles	`Z-`	proper names

Mood	`I`ndicative, `S`ubjunctive, or `J`ussive of imperfect verbs, with `D` if undecided between `S` and `J`
Voice	`A`ctive or `P`assive
Person	`1` speaker, `2` addressee, `3` others
Gender	morphologically overt 'gender', `M`asculine or `F`eminine
Number	morphologically overt 'number', `S`ingular, `D`ual, or `P`lural
Case	`1` nominative, `2` genitive, `4` accusative
Defin	morphological 'definiteness', `I`ndefinite, `D`efinite, `R`educed, or `C`omplex
MemberOf	member of syntactic `Co`ordination or `Ap`position
ClauseHead	the token is the head of the given type of a subordinate clause
GramCoref	the pronoun `S-` is a grammatical coreferent, unlike other pronouns that are textual coreferents
InputForm	the token is the first of all that analyze the given orthographical word in the original input
TokenGloss	a clue to the morphological structure of the token

Pred	verbal predicate	Coord	coordination
Pnom	nominal predicate	Apos	apposition
PredE	existential predicate	Ante	anteposition
PredC	conjunction as the clause's head	AuxC	conjunction
PredP	preposition as the clause's head	AuxP	preposition
Sb	subject	AuxE	emphasizing expression
Obj	object	AuxM	modifying expression
Adv	adverbial	AuxY	auxiliary, part of compound
Atr	attribute	AuxG	graphical symbol
Atv	complement	AuxK	sentence separator
ExD	ellipsis, no actual dependency	_	excessive token, esp. due to typo