Evaluation of an integrated multi-task machine learning system with humans in the loop

Aaron Steinfeld, S. Rachael Bennett, Kyle Cunningham, Matt Lahut, Pablo Alejandro Quinones, Django Wexler, Dan Siewiorek, Jordan Hayes, Paul Cohen, Julie Fitzgerald, Othar Hansson, Mike Pool, Mark Drummond

    Research output: Contribution to conferencePaperpeer-review

    15 Scopus citations

    Abstract

    Performance of a cognitive personal assistant, RADAR, consisting of multiple machine learning components, natural language processing, and optimization was examined with a test explicitly developed to measure the impact of integrated machine learning when used by a human user in a real world setting. Three conditions (conventional tools, Radar without learning, and Radar with learning) were evaluated in a large-scale, between-subjects study. The study revealed that integrated machine learning does produce a positive impact on overall performance. This paper also discusses how specific machine learning components contributed to human-system performance.

    Original languageEnglish (US)
    Pages182-188
    Number of pages7
    StatePublished - 2007
    Event2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07 - Gaithersburg, MD, United States
    Duration: Aug 28 2007Aug 30 2007

    Other

    Other2007 Performance Metrics for Intelligent Systems Workshop, PerMIS'07
    Country/TerritoryUnited States
    CityGaithersburg, MD
    Period8/28/078/30/07

    Keywords

    • Evaluation
    • Intelligent systems
    • Machine learning
    • Mixed-initiative assistants

    ASJC Scopus subject areas

    • Artificial Intelligence
    • Computer Science Applications
    • Control and Systems Engineering

    Fingerprint

    Dive into the research topics of 'Evaluation of an integrated multi-task machine learning system with humans in the loop'. Together they form a unique fingerprint.

    Cite this