Uncertainty estimation
Three types of uncertainty were identified during the development of the
risk ranking framework. First, the uncertainty associated with the
criteria weights assigned by the stakeholder-experts was characterized
by a Beta-PERT distribution (Vose, 2008). For each pathogen, a total
weighted risk score was obtained by adding each individual risk
criterion score multiplied by values from the expert’s weight
distribution for each criterion using the following equation (adapted
from ECDC 2017):
\(Weighted\ risk\ score=\ \sum_{i=1}^{7}{Wi*S_{\text{ij}}}\) (2)
\begin{equation}
W_{i}\sim\text{BetaPERT}(a,b,c)\nonumber \\
\end{equation}where Sij is the score for pathogen j on
criterion i as in Equation 1, and Wi is
the probability distribution of the expert-designated weights for each
criterion i . The Beta-PERT distribution was characterized by a
minimum (a), most likely (b) and maximum value (c). Latin hypercube
sampling (LHS) was performed in @Risk (Palisade, Inc.) to iterate over
Equation 2 and sample stratified random numbers from each probability
distribution of the expert-designated weights defined in the model
(Vose, 2008). Significant correlations between input values were
included in the model (Supporting Information). The LHS was repeated for
10,000 iterations to generate the final distribution of total weighted
risk scores with mean and standard deviation values that accurately
accounted for all possible weighted risk scores for a given set of
parameters defined. Pairwise t-tests with a Bonferroni correction and
nonparametric Kolmogorov-Smirnov tests (Arnold & Emerson, 2011) were
applied to test for significant differences in mean total risk scores
and overall total weighted risk distributions between pathogens,
respectively.
The second type of uncertainty was related to the amount of published
evidence supporting the risk score assigned to each criterion. A
normalized scale (0-2, Table 3) was developed to estimate the evidence
uncertainty associated with the total weighted and unweighted risk
scores for each pathogen. If we were unable to find published
information about a particular criterion for a particular pathogen, the
risk score was extrapolated from similar pathogens and was assigned a
high uncertainty score (2) for that criterion. Total evidence
uncertainty score for each pathogen was estimated using Equation 3:\(Total\ evidence\ uncertainty=\sum_{i=1}^{7}U_{\text{ij}}\) (3)
where Uij is the normalized uncertainty score for
pathogen j on criterion i. Total evidence uncertainty
scores for each pathogen are reported in Table 4.
A third type of uncertainty was related to the ‘confidence level’ of the
stakeholder-experts in assigning the weight values. Experts indicated
their confidence in the assigned weights by a score between 1 (low) and
10 (high, integer number). The confidence scores were intended to
illustrate the range and variety of confidence from various experts and
not used in the final calculations of the risk ranking.