Understanding formal aggregation of expert assessments in safety decision making

Expert assessment is critical to safety decision making. Expert assessment is used to interpret, qualify, or complement historical data when characterizing uncertainties related to safety. When historical records are not available, uncertainty characterization relies solely on the experts’ assessment. Because safety of engineered systems is a multidisciplinary problem, decision makers need to reason through the information provided by several experts. Past work on expert aggregation in engineering management has focused on using multiple behavioral and mathematical aggregation techniques that are subjective in nature and lack the rigor to reason about the individual assessments in a mathematically consistent manner. In this paper, we leverage work from the fields of business management, economics, and cybernetics to present different techniques to mathematically aggregate expert assessment. We differentiate between expertise provided through three different information mechanisms: belief distributions, opinions, and judgements. We show that these three forms of expert information inherently require different aggregation. In fact, we demonstrate that most of the existing techniques for expert aggregation can actually lead to misusing expertise.