Tutorial 2: Usability and Performance Evaluation for Advanced Spoken Dialogue Systems
Kristiina Jokinen and Michael McTear
This tutorial will focus on methods, problems and challenges in the evaluation of advanced spoken dialogue systems. It is grounded in research that combines various speech and language technology components into an integrated system, and surveys the issues related to the design, evaluation and comparison of such systems. A number of different approaches to robust and efficient interaction management will be reviewed, together with various performance and user evaluation methods in academic and industrial environments. A closer look will be taken at the different metrics and usability criteria, as well as automatic design and evaluation methods. Practical requirements for dialogue systems, such as robustness, scalability and portability will also be discussed and exemplified from the point of view of performance evaluation and usability. Special attention will be paid to user evaluation, and to the user’s expectations and experience of the system.
The tutorial is targeted at researchers and system developers who wish to learn more about theoretical and practical issues concerning development and evaluation of spoken dialogue systems. The tutorial will consist of two parts. The first part will survey techniques in spoken dialogue management and motivate the need for evaluation studies by discussing novel challenges and requirements for interactive systems. Issues concerning conversational dialogue modelling, such as knowledge representation, context, adaptation, learning, error handling, multimodality, ubiquitous computing, as well as practical requirements for commercial dialogue systems, such as robustness, scalability and portability, will be discussed and exemplified from the point of view of performance evaluation, usability, and user evaluation. The second part of the tutorial will focus on methodologies and practices of evaluation, and will deal with basic concepts, methods and metrics for system performance evaluation and usability. In particular, challenges for the evaluation of advanced dialogue systems will be discussed.
Tutorial Outline
-
Introduction: state-of-the-art and motivation
- Basic concepts and techniques in spoken interaction management
- Conversational dialogue models and commercial systems
- User requirements and need for evaluation
-
Evaluation methodologies and practices
- Basic concepts in system evaluation
- Usability, usefulness, and human factors in speech user interfaces
- Evaluation of advanced dialogue systems
- Questions and discussion
Michael McTear is Professor of Computer Science at University of Ulster. His main research interest is spoken dialogue technology. Other interests include: user modelling, natural language processing, language acquisition. He is the author of the widely used text book "Spoken dialogue technology: toward the conversational user interface" (Springer Verlag, 2004), and co-author of the Queen’s Communicator dialogue system. He has given several invited talks, and was the recipient of the Distinguished Senior Research Fellowship of University of Ulster in 2006. He is a member of the Scientific Advisory Group of SIGDial, the ACL/ISCA Special Interest Group for Discourse and Dialogue.
Kristiina Jokinen is Adjunct Professor of Interaction Technology at the University of Tampere, and a Visiting Professor of Intelligent User Interfaces at the University of Tartu, Estonia. Her research concerns human-computer interaction, spoken dialogue systems, and cooperative response planning, and her current interests include adaptation in speech interfaces, multimodal communication, and evaluation methodology for interactive systems. She has played a leading role in several academic and industrial research projects, and is currently writing her book "Constructive Dialogue Management – Speech Interaction and Rational Agents" (John Wiley & Sons, forthcoming). She is the secretary of SIGDial, the ACL/ISCA Special Interest Group for Discourse and Dialogue.
