Predictive Multiplicity in Probabilistic Classification

There may exist multiple models that perform almost equally well for any given prediction task. We examine how predictions change across these competing models. In particular, we study predictive multiplicity -- in probabilistic classification. We formally define measures for our setting and develop optimization-based methods to compute these measures for convex empirical risk minimization problems. We apply our methodology to gain insight into why predictive multiplicity arises. We demonstrate the incidence and prevalence of predictive multiplicity in real-world risk assessment tasks. Our results emphasize the need to report multiplicity more widely.