Unifying framework for the classification and calculation of robustness metrics

There are many metrics in the literature that provide some measure of robustness. Previous categorisations of these metrics have focused on the metrics as a whole, accentuating their differences and sometimes masking their similarities. We proposed breaking down the calculation of robustness metrics into three transformations, which provides a unifying approach (see figure) to calculating a wide range of metrics and enables deeper insights to be gained into the similarities and differences between them. As shown in the figure, the form of the first transformation determines the type of information the robustness metric utilises (e.g. absolute performance values, regret, or satisfaction of constraints), the second transformation reflects the degree of optimism or pessimism of the decision-maker, and the form of the third transformation reflects the decision-maker’s preference towards maximising average performance, minimising variance, or some other higher-order moment. We demonstrate the wide range of applicability of the proposed framework by using it to categorise a number of commonly used metrics, such as the maximin and maximax metrics, Hurwicz optimism-pessimism rule, Laplace’s principle of insufficient reason, several regret metrics, and Starr’s domain criterion. The unifying framework demystifies robustness metrics for decision-makers, providing guidance as to which metric might be most appropriate in their decision context, and, unlike previous categorisations, provides a computational framework, rather than a conceptual one, allowing users calculate different robustness metrics using three transformations that are common to all metrics.