Reliability of MUSE 2 and Tobii Pro Nano at capturing mobile application users' real-time cognitive workload changes

Introduction: Despite the importance of cognitive workload in examining the usability of smartphone applications and the popularity of smartphone usage globally, cognitive workload as one attribute of usability tends to be overlooked in Human-Computer Interaction (HCI) studies. Moreover, limited studies that have examined the cognitive workload aspect often measured some summative workloads using subjective measures (e.g., questionnaires). A significant limitation of subjective measures is that they can only assess the overall, subject-perceived cognitive workload after the procedures/tasks have been completed. Such measurements do not reflect the real-time workload fluctuation during the procedures. The reliability of some devices on a smartphone setting has not been thoroughly evaluated. Methods: This study used mixed methods to empirically study the reliability of an eye-tracking device (i.e., Tobii Pro Nano) and a low-cost electroencephalogram (EEG) device (i.e., MUSE 2) for detecting real-time cognitive workload changes during N-back tasks. Results: Results suggest that the EEG measurements collected by MUSE 2 are not very useful as indicators of cognitive workload changes in our setting, eye movement measurements collected by Tobii Pro Nano with mobile testing accessory are useful for monitoring cognitive workload fluctuations and tracking down interface design issues in a smartphone setting, and more specifically, the maximum pupil diameter is the preeminent indicator of cognitive workload surges. Discussion: In conclusion, the pupil diameter measure combined with other subjective ratings would provide a comprehensive user experience assessment of mobile applications. They can also be used to verify the successfulness of a user interface design solution in improving user experience.