Towards a Just Theory of Measurement: A Principled Social Measurement Assurance Program

While formal definitions of  fairness in machine learning (ML)  have been proposed,  its  place within a broader institutional model of fair decision-making remains ambiguous. In this paper we interpret ML as a tool for revealing when and how measures fail to capture purported constructs of interest, augmenting a given institutions understanding of its own interventions and priorities. Rather than codifying ”fair” principles into ML models directly, the use of ML can thus be understood as a form of quality assurance for existing institutions, exposing the epistemic fault lines of their own measurement practices. Drawing from Friedler et al.s recent discussion of representational mappings and previous discussions on the ontology of measurement, we propose a social measurement assurance program (sMAP) in which ML encourages expert deliberation on a given decision-making procedure by examining unanticipated or previously unexamined covariates. As an example, we apply Rawlsian principles of fairness to sMAPand produce a provisional just theory of measurement that would guide the use of ML for achieving fairness in the case of child abuse in Allegheny County.