Designing experiments to test and improve hypothesized planning knowledge derived from demonstration

A number of techniques have been developed to effectively extract and generalize planning knowledge based on expert demonstration. In this paper we consider a complementary approach to learning in which experiments are designed to test hypothesized planning knowledge. In particular, we describe an algorithm that automatically generates experiments to test assertions about plan-step ordering. Experimenting with plan-step ordering can identify asserted ordering constraints that are in fact not necessary, as well as uncover necessary ordering constraints previously not represented. The algorithm consists of three parts: identifying the space of step-ordering hypotheses, efficiently generating ordering tests, and planning experiments that use the tests to identify order constraints that are not currently represented. This method is implemented in the CMAX experiment design module and is part of the POIROT integrated learning system. We discuss the role of experimentation in planning knowledge refinement and some future directions for CMAX's development.