An evaluation of the chat and knowledge delivery components of a low-level dialog system:The AZ-ALICE experiment

Robert P. Schumaker, Mark Ginsburg, Hsinchun Chen, Ying Liu

Research output: Contribution to journalArticlepeer-review

22 Scopus citations


An effective networked knowledge delivery platform is one of the Holy Grails of Web computing. Knowledge delivery approaches range from the heavy and narrow to the light and broad. This paper explores a lightweight and flexible dialog framework based on the ALICE system, and evaluates its performance in chat and knowledge delivery using both a conversational setting and a specific telecommunications knowledge domain. Metrics for evaluation are presented, and the evaluations of three experimental systems (a pure dialog system, a domain knowledge system, and a hybrid system combining dialog and domain knowledge) are presented and discussed. Our study of 257 subjects shows approximately a 20% user correction rate on system responses. Certain error classes (such as nonsense replies) were particular to the dialog system, while others (such as mistaking opinion questions for definition questions) were particular to the domain system. A third type of error, wordy and awkward responses, is a basic system property and spans all three experimental systems. We also show that the highest response satisfaction results are obtained when coupling domain-specific knowledge together with conversational dialog.

Original languageEnglish (US)
Pages (from-to)2236-2246
Number of pages11
JournalDecision Support Systems
Issue number4
StatePublished - Jan 2007


  • AIML
  • Dialog platform
  • Domain-specific knowledge
  • Knowledge delivery evaluation
  • XML
  • chatterbot

ASJC Scopus subject areas

  • Management Information Systems
  • Information Systems
  • Developmental and Educational Psychology
  • Arts and Humanities (miscellaneous)
  • Information Systems and Management


Dive into the research topics of 'An evaluation of the chat and knowledge delivery components of a low-level dialog system:The AZ-ALICE experiment'. Together they form a unique fingerprint.

Cite this