DR-AIR: A data repository bridging the research gap in AI reliability

Artificial intelligence (AI) technology and systems have been advancing rapidly. However, ensuring the reliability of these systems is crucial for fostering public confidence in their use. This necessitates the modeling and analysis of reliability data specific to AI systems. A major challenge in AI reliability research, particularly for those in academia, is the lack of readily available AI reliability data. To address this gap, this article focuses on conducting a comprehensive review of available data and establishing DR-AIR: a data repository for AI reliability. Specifically, we introduce key measurements and data types for assessing AI reliability, along with the methodologies used to collect these data. We also provide a detailed description of the currently available datasets with illustrative examples. Furthermore, we outline the setup of the DR-AIR repository and demonstrate its practical applications. This repository provides easy access to datasets specifically curated for the research. We believe these efforts will significantly benefit the research community by facilitating access to reliability data and promoting collaboration across various academic domains. We conclude our article with a call to action, encouraging the research community to contribute and share AI reliability data to further advance this critical field of study.