Evaluating Artifacts Using Experiments as Part of the Design Science Framework

Design science is of increasing importance in information systems (IS) with top journals and funding agencies recognizing it as an important component of IS research. One critical element of good design science is a solid evaluation of the artifacts, i.e., of individual algorithms and completed information systems. This tutorial focuses on the design and execution of experiments to evaluate such artifacts in an efficient manner. The artifact's life cycle is taken into account to ensure the evaluation is appropriate, effective and efficient through optimal use of resources. Evaluations can range from automated evaluations of algorithms using gold standards to user studies of an entire information system with representative users in situ. The tutorial will focus on the controlled experiment for artifacts, appropriate statistical analysis, and tips on common errors that can be avoided.