Recently, these two research trends have converged into Distributional Compositionality; a number of works have been carried out on how to incorporate compositionality in DSMs, in order to construct vectorial representations for linguistic constituents above the word (e.g.: Baroni and Zamparelli, 2010; Clarke et al., 2011; Erk et al., 2010; Grefenstette and Sadrzadeh, 2011; Mitchell and Lapata, 2010; Thater et al., 2010). Moreover, grammatical words and in particular logical words (quantifying determiners, coordination, negation) that for long have been considered to be part of the formal semantics realm only have became of interest within the DSMs framework too.

DSMs have been evaluated on different semantic tasks. For instance, they have proved to be very successful at modeling a wide range of lexico-semantic phenomena by geometric methods applied to the distributional space (Turney and Pantel, 2010). Sentential entailment has been the starting point of the logical view on natural language but again it is a valid task for any theory which aim to capture the semantics of a language. Hence, predicting entailment is a good test-bed for Compositional DSMs too.

In the course, after a brief general introduction to FS and DS models, I will introduce various state-of-the-art approaches to composition in DS and I will discuss ways in which they have been evaluated and point to on-going work within this research line.

Outline:

First Lecture: Formal Semantics Models

• Brief introduction: Model, Domain, Function interpretation
• Meaning of words, sentential entailment
• Syntax-semantics and meaning of phrases/sentence

Reading:

• Portner, P. and Partee, B. (eds.) (2002) Formal Semantics: The essential readings. Blackwell

Second Lecture: Distributional Semantics Models

• Brief introduction to DSMs
• From content words to grammatical words
• Composing DS representation: state-of-art methods

Reading:

• Peter D. Turney, and P. Pantel (2010) From Frequencyto Meaning: Vector Space Models of Semantics. Journal of Artificial Intelligence Research. 37, p. 141-188
• G. Strang (2009) Introduction to Linear Algebra. Wellesley Cambridge Press
• S. Evert and A. Lenci (2009) Distributional Semantic Models. Advanced course at ESSLLI 2009
  http://wordspace.collocations.de/doku.php/course:esslli2009:start
• K. Gimpel (2006) Modeling Topics

Third Lecture: Evaluation Tasks

• Evaluation of state-of-the-art compositional DSMs

Reading:

• M. Baroni and R. Zamparelli (2010) Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. Proceedings of EMNLP
• M. Baroni, R. Bernardi, Q. Do, Ch. Shan (2012) Entailment above the word level in distributional semantics. Proceedings of EACL
• E. Grefenstette and M. Sadrzadeh (2011) Experimenting with transitive verbs in a DisCoCat. Proceedings of GEMS
• E. Guevara (2010) A regression model of adjective-noun compositionality in in distributional semantics. Proceedings of GEMS
• W. Kintsch (2001) Predication. Cognitive Science, 25(2): 173-202.
• J. Mitchell and M. Lapata (2008) Vector-based models of semantic composition. Proceedings of ACL
• J. Mitchell and M. Lapata (2010). Composition in distributional models of semantics. Cognitive Science 34(8): 1388-1429

The lecture series will discuss not only (a) DeepDict's uses and perspectives, but also (b) the annotation system and (c) the Constraint Grammar parsing technology behind it, dedicating one session to each of these issues.

Since DeepDict is available for 10 different languages, it will be possible to accomodate participants' individual language interests to a certain degree. Also, given the modular and language-independent architecture of Constraint Grammar systems, workshop-style discussion of individual project ideas is strongly encouraged.

1. Websites

• http://gramtrans.com/deepdict/ (The online DeepDict interface)
• http://gramtrans.com/deepdict/reference/ (Its reference page)
• http://visl.sdu.dk (SDU university site with the CG parsers etc. that are behind DeepDict),
  e.g. http://beta.visl.sdu.dk/visl/en/parsing/automatic/ (English live parses at various levels).
  For other languages, navigate -> sentence analysis -> machine analysis -> language; or -> language flag -> machine analysis -> flat structure / dependency
• http://beta.visl.sdu.dk/constraint_grammar.html (Constraint Grammar, short introduction, CG lab and manual, language overview and references)
• http://corp.hum.sdu.dk (CorpusEye, with corpus search interfaces for the corpora behind DeepDict)

2. Publications

• Bick, Eckhard (2009). DeepDict - A Graphical Corpus-based Dictionary of Word Relations. Proceedings of NODALIDA 2009. NEALT Proceedings Series Vol. 4. pp. 268-271. Tartu: Tartu University Library. ISSN 1736-6305 (http://beta.visl.sdu.dk/~eckhard/pdf/nodalida2009_deepdict.pdf)
• Karlsson et al. (1995). "Constraint Grammar - A Language-Independent System for Parsing Unrestricted Text". Mouton de Gruyter (The seminal book on CG, especially initial chapters, accessible in http://books.google.com)
• Further, interest-based reading: The Eckhard Bick publications page with articles on various aspects of Constraint Grammar and its applications, targeting a range of languages:
  http://beta.visl.sdu.dk/Artikeloversigt.html

3. Annotation, category definitions

• http://beta.visl.sdu.dk/tagset_cg_general.html (Cross language Constraint Grammar tags)
• http://beta.visl.sdu.dk/semantic_prototypes_overview.pdf (Semantic prototypes)
• Depending on individual interests, language-specific definition overviews from http://beta.visl.sdu.dk/lecture_notes.html,
  e.g. Portuguese: http://beta.visl.sdu.dk/visl/pt/info/symbolset-manual.html

Outline:

Lecture One: Big Data, Economics and Obstacles

This class will look at the problems and challenges associated with processing massive amounts of data, using commodity machines (ie cloud computing). We will touch upon questions of trust, economics as well those aspects of Big Data which make it hard to deal with.

Reading:

• http://www.wired.com/science/discoveries/magazine/16-07/pb_theory
• http://queue.acm.org/detail.cfm?id=1563874
• http://en.wikipedia.org/wiki/Power_law
• http://perspectives.mvdirona.com/2010/07/13/HighPerformanceComputingHitsTheCloud.aspx
• http://www.webhostingunleashed.com/features/server-meltdowns-millions-020309/
• http://www.google.com/governmentrequests/
• http://net.tutsplus.com/articles/general/supercharge-website-performance-with-aws-s3-and-cloudfront/

Lecture Two: Map Reduce and Hadoop

Given the background material presented in L1, L2 will give an overview of one popular way to problem solve using large numbers of unreliable machines. This class will introduce Hadoop (the open source version of Map Reduce), the Map Reduce programming models, efficiency concerns and will end with a critique.

Reading:

• http://research.google.com/archive/papers/mapreduce-sigmetrics09-tutorial.pdf
• http://database.cs.brown.edu/projects/mapreduce-vs-dbms/
• http://hadoop.apache.org/

Lecture Three: Randomised Algorithms

Sometimes we need to deal with problems that are just to large for our machines. Randomised algorithms allow us to tackle such problems and can be amazingly fast (or compact). However, unlike conventional approaches, they can make mistakes. This class will show how two problems in natural language processing --representing large language models and finding breaking news in Twitter-- can be solved using Bloom Filters and Locality Sensitive Hashing.

Reading:

• http://en.wikipedia.org/wiki/Universal_hashing
• http://en.wikipedia.org/wiki/Randomized_algorithm
• http://en.wikipedia.org/wiki/Bloom_filter
• http://en.wikipedia.org/wiki/Locality_sensitive_hashing
• http://homepages.inf.ed.ac.uk/miles/papers/naacl10a.pdf
• http://homepages.inf.ed.ac.uk/miles/papers/acl07.pdf

Outline:

Lecture One: Introduction, Inputs & Outputs

Reading:

• Pieraccini et al. (2009) Are We There Yet? Research in Commercial Spoken Dialog Systems. In Text, Speech and Dialogue, 12th International Conference, TSD 2009, Pilsen, Czech Republic
• F. Mairesse et al. (2009) Spoken language understanding from unaligned data using discriminative classification models. In ICASSP 2009, Taiwan
• F. Jurcicek et al. (2009) Transformation-based Learning for Semantic Parsing. In Interspeech 2009, Brighton, UK
• Luke S. Zettlemoyer and Michael Collins (2009) Learning Context-dependent Mappings from Sentences to Logical Form. In Proceedings of the Joint Conference of the Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACL-IJCNLP)

Lecture Two: Dialogue belief modeling

Reading:

• C. Bishop (2006). Pattern Recognition and Machine Learning. Chapter 8.
  http://research.microsoft.com/en-us/um/people/cmbishop/prml/Bishop-PRML-sample.pdf
• B. Thomson and S. Young (2010) Bayesian update of dialogue state: A POMDP framework for spoken dialogue systems. Computer Speech & Language, vol. 24, no. 4, pp. 562-588
• B. Thomson et al. (2010) Parameter learning for POMDP spoken dialogue models. In IEEE SLT'10: Spoken Language Technology Workshop, pp. 271-276

Lecture Three: Dialoge policy learning

Reading:

• Sutton & Barto (1998) Reinforcement Learning
• E. Levin et al (1998) Using Markov decision process for learning dialogue strategies. In ICASSP
• N. Roy, J. Pineau and S. Thrun (2000) Spoken Dialog Management for Robots. In Association for Computational Linguistics. Hong Kong, Oct. 2000
• S. Young et al. (2010) The Hidden Information State model: A practical framework for POMDP-based spoken dialogue management? Computer Speech & Language, vol. 24, no. 2, pp. 150-174
• J. Williams and S. Young (2007) Partially Observable Markov Decision Processes for Spoken Dialog Systems. Computer Speech and Language21(2):231-422.

The participants can benefit from the course in two ways:

• General experience with experimenting in Unix environment. (Unlike e.g. Experiment management system distributed with Moses, eman is versatile and task independent. It is only the specific set of 'seed scripts' that cover Moses training and evaluation pipeline.)
• Exposure to phrase-based MT and exploration of some relevant parameters.

Outline:

(1) Lecture:

• The inherent limitations of phrase-based vs. syntactic machine translation.
• Factored models to include linguistic annotation in phrase-based MT.
• Wild experimenting (with eman) and how to make sense of results.

(2) Lab:

• Working with eman.
• Using eman for Moses experiments.
• Baseline and factored phrase-based English->Tamil translation.
• Collecting and understanding scores.

(3) Lab:

• Including Treex into eman.
• Applying Treex pre-processing for English->Tamil MT.
• (Time-permitting:) Visualisation of MT errors.

Reading:

• Moses Manual ... especially Section 4 Background
  http://www.statmt.org/moses/manual/manual.pdf
  You may want to get Moses running on your laptop, but you will be provided with Unix servers and Moses pre-installed.
• Bash
  • (For Windows users: Console Crash-Course
    http://www.ibm.com/developerworks/linux/library/l-roadmap2/index.html)
  • Introductory slides with some examples:
    http://www.cv.nrao.edu/~jmalone/talks/bash.pdf
  • Bash manual page
    http://www.gnu.org/software/bash/manual/html_node/index.html
  • Console text editors to your taste:
    a) nano: http://www.howtogeek.com/howto/42980/the-beginners-guide-to-nano-the-linux-command-line-text-editor/
    b) vim: http://blog.interlinked.org/tutorials/vim_tutorial.html
    c) whichever else you like

Outline:

(1) Lecture: Introduction to Treex and its main features
(2) Lab: Installing Treex, using it for tagging and parsing
(3) Lab: Implementing new Treex blocks for SMT preprocessing

Treex (formerly called TectoMT) is described in Popel and Žabokrtský (2010). If you plan to install it on your notebook, we recommend to learn basics of Perl and follow the installation guide.