Toward AI Research Methodology: Three Case Studies in Evaluation

Paul R. Cohen; Adele E. Howe

doi:10.1109/21.31069

Toward AI Research Methodology: Three Case Studies in Evaluation

Paul R. Cohen, Adele E. Howe

Research output: Contribution to journal › Article › peer-review

30 Scopus citations

Abstract

The roles of evaluation in empirical artificial intelligence (Al) research are described, in an idealized cyclic model and in the context of three case studies. The case studies illustrate pitfalls in evaluation and the contributions of evaluation at all stages of the research cycle. Evaluation methods are contrasted with those of the behavioral sciences, and it is concluded that AI must define and refine its own methods. To this end, several experiment “schemas” and many specific evaluation criteria are described; recommendations are offered in the hope of encouraging the development and practice of evaluation methods in AI.

Original language	English (US)
Pages (from-to)	634-646
Number of pages	13
Journal	IEEE Transactions on Systems, Man and Cybernetics
Volume	19
Issue number	3
DOIs	https://doi.org/10.1109/21.31069
State	Published - 1989

ASJC Scopus subject areas

Engineering(all)

Access to Document

10.1109/21.31069

Cite this

@article{c1dffd844eb140c1bd0eb6af14a3811a,

title = "Toward AI Research Methodology: Three Case Studies in Evaluation",

abstract = "The roles of evaluation in empirical artificial intelligence (Al) research are described, in an idealized cyclic model and in the context of three case studies. The case studies illustrate pitfalls in evaluation and the contributions of evaluation at all stages of the research cycle. Evaluation methods are contrasted with those of the behavioral sciences, and it is concluded that AI must define and refine its own methods. To this end, several experiment “schemas” and many specific evaluation criteria are described; recommendations are offered in the hope of encouraging the development and practice of evaluation methods in AI.",

author = "Cohen, {Paul R.} and Howe, {Adele E.}",

note = "Funding Information: Manuscript received March 2, 1988; revised August 10. 1988. This research was funded by the Office of Naval Research under a University Research Initiative Grant, contracts N00014-86-K-0764 and ONR AFOSR 4331690-01. P. R. Cohen and A. E. Howe are with the Experimental Knowledge Systems Laboratory, Department of Computer and Information Science, University of Massachusetts, Amherst. MA, 01003. IEEE Log Number 8927057.",

year = "1989",

doi = "10.1109/21.31069",

language = "English (US)",

volume = "19",

pages = "634--646",

journal = "IEEE Transactions on Systems, Man and Cybernetics",

issn = "0018-9472",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

number = "3",

}

TY - JOUR

T1 - Toward AI Research Methodology

T2 - Three Case Studies in Evaluation

AU - Cohen, Paul R.

AU - Howe, Adele E.

N1 - Funding Information: Manuscript received March 2, 1988; revised August 10. 1988. This research was funded by the Office of Naval Research under a University Research Initiative Grant, contracts N00014-86-K-0764 and ONR AFOSR 4331690-01. P. R. Cohen and A. E. Howe are with the Experimental Knowledge Systems Laboratory, Department of Computer and Information Science, University of Massachusetts, Amherst. MA, 01003. IEEE Log Number 8927057.

PY - 1989

Y1 - 1989

N2 - The roles of evaluation in empirical artificial intelligence (Al) research are described, in an idealized cyclic model and in the context of three case studies. The case studies illustrate pitfalls in evaluation and the contributions of evaluation at all stages of the research cycle. Evaluation methods are contrasted with those of the behavioral sciences, and it is concluded that AI must define and refine its own methods. To this end, several experiment “schemas” and many specific evaluation criteria are described; recommendations are offered in the hope of encouraging the development and practice of evaluation methods in AI.

AB - The roles of evaluation in empirical artificial intelligence (Al) research are described, in an idealized cyclic model and in the context of three case studies. The case studies illustrate pitfalls in evaluation and the contributions of evaluation at all stages of the research cycle. Evaluation methods are contrasted with those of the behavioral sciences, and it is concluded that AI must define and refine its own methods. To this end, several experiment “schemas” and many specific evaluation criteria are described; recommendations are offered in the hope of encouraging the development and practice of evaluation methods in AI.

UR - http://www.scopus.com/inward/record.url?scp=0024668625&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0024668625&partnerID=8YFLogxK

U2 - 10.1109/21.31069

DO - 10.1109/21.31069

M3 - Article

AN - SCOPUS:0024668625

VL - 19

SP - 634

EP - 646

JO - IEEE Transactions on Systems, Man and Cybernetics

JF - IEEE Transactions on Systems, Man and Cybernetics

SN - 0018-9472

IS - 3

ER -

Toward AI Research Methodology: Three Case Studies in Evaluation

Abstract

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this