Understanding planner behavior

As planners and their environments become increasingly complex, planner behavior becomes increasingly difficult to understand. We often do not understand what causes them to fail, so that we can debug their failures, and we may not understand what allows them to succeed, so that we can design the next generation. This paper describes a partially automated methodology for understanding planner behavior over long periods of time. The methodology, called Dependency Interpretation, uses statistical dependency detection to identify interesting patterns of behavior in execution traces and interprets the patterns using a weak model of the planner's interaction with its environment to explain how the patterns might be caused by the planner. Dependency Interpretation has been applied to identify possible causes of plan failures in the Phoenix planner. By analyzing four sets of execution traces gathered from about 400 runs of the Phoenix planner, we showed that the statistical dependencies describe patterns of behavior that are sensitive to the version of the planner and to increasing temporal separation between events, and that dependency detection degrades predictably as the number of available execution traces decreases and as noise is introduced in the execution traces. Dependency Interpretation is appropriate when a complete and correct model of the planner and environment is not available, but execution traces are available.