Did I Buy the Right Gadget? A Methodological Replication Study

This study is a methodological replication of Valacich et al.’s (2018) experiment, which investigated the evaluability hypothesis in the context of technology product assessment. The evaluability hypothesis suggests that the evaluation context significantly influences how users assess and perceive technology features. More specifically, it posits that the perception of technology features, as well as the ultimate selection of a technology product, is likely to differ depending on whether a single technology is evaluated in isolation or multiple technologies are evaluated simultaneously. Our replication study consisted of 310 participants who evaluated two wireless internet products with two product features – connection speed and security level – in either a joint or separate evaluation context. The product features were operationalized in two ways: (1) a hard/hard context where both connection speed and security level were hard-to-evaluate and (2) a hard/easy context where connection speed was hard-to-evaluate, but security level was easier-to-evaluate. Although the original study found that participants’ preferences between the products did not reverse in the hard/hard context but did reverse in the hard/easy context, our results diverged from both of these findings. In our study, the preference reversal occurred in the hard/hard context but not in the hard/easy context. Notably, our study suggests a shift in preferences toward greater emphasis on security and less emphasis on speed, regardless of the evaluation context. These results align with recent trends in the IT marketplace and may offer new practical insights into technology design, usability assessments, and product acceptance.