Evaluating the model activity systems: General methodological considerations

Our highest ambition was to find ways to conduct diagnostic evaluations in situ-that is, in activity. To some extent this proved possible, but showing development in situ does not meet the criteria of causal social science analysis. Although in situ evaluations may be an effective tool locally, they are unlikely to be considered appropriate under contemporary views of what constitutes "real science" (Lyon 1999; Slavin 2003). We sought to solve these problems by adopting a multi-method approach that takes advantage of the analytic potential in the broad variety of settings provided by the project. In some cases, we approached, or even met, the standard that takes the psychological experiment of individual college sophomores as the model. In other cases, we provided alternative, quasi-experimental evidence. In still others, we used a qualitative longitu dinal approach that focused on interactions of a quality that, according to conclusions drawn from "proper" scientific research, should underpin the quantitative results. In the chapters that follow, readers must reach their own conclusions about the adequacy of the evidence that the Fifth Dimension is an effective program. Some may find our strategic mixing of methods, locations, and opportunities a useful tool for thinking about their own efforts at program evaluation and analysis of psychological development. Others may argue that the work should be dismissed because we simply did not succeed in getting our local communities to adhere to scientific criteria. We hope that most readers will come to the former conclusion, but expect at least some to draw the latter.