CAT 2021 Slot 2 VARC: The Impossibility of Algorithmic Fairness

When a bank denies a loan application, a hospital triages patients, or a court sets bail, the decision was once made, for better or worse, by a human being exercising judgment within institutional constraints. Increasingly, such decisions are made, or substantially shaped, by statistical models trained on historical data. Proponents argue that algorithms, unlike people, are free of conscious bias, apply consistent criteria across cases, and can be audited in ways that an individual official's private reasoning cannot. This optimism has collided, over the past decade, with a body of research showing that algorithmic systems can reproduce, and sometimes amplify, the very discriminatory patterns they were meant to eliminate, and with a further, more unsettling finding: that different, individually reasonable definitions of fairness are mathematically incompatible with one another, so that a system cannot satisfy all of them simultaneously.

The reproduction problem is by now well documented. A model trained on historical arrest data will, if arrests have historically been racially skewed by discriminatory policing, learn to associate certain neighbourhoods or demographic groups with elevated risk, regardless of whether that association reflects true underlying offending rates or merely the historical pattern of who was targeted for enforcement. The model, in a formal sense, may be accurately describing the data it was given; the injustice lies not in mathematical error but in an unexamined assumption that yesterday's enforcement patterns are a neutral proxy for genuine risk. This is not a failure that better engineering alone can fix, since the flaw lies partly in the data-generating process, the world the data was drawn from, rather than in the model's internal logic.

More surprising to many observers was the discovery, formalized independently by several groups of computer scientists around 2016 in the context of criminal risk-assessment tools, that competing notions of fairness cannot generally all be satisfied at once. Consider two intuitively appealing criteria. 'Calibration' requires that, among people assigned a given risk score, the actual outcome rate should be the same across demographic groups, so that a score of 'seven out of ten' means the same thing for every group. 'Balance' (also called equalized error rates) requires that the model's errors, wrongly flagging low-risk people as high-risk, or wrongly clearing high-risk people, should be distributed equally across groups. When the underlying base rates of the outcome being predicted differ between groups, as they typically do given unequal social and economic conditions, it is a mathematical certainty, not a matter of engineering effort, that a model achieving calibration will violate balance, and vice versa, except in the trivial case of a perfect predictor. This impossibility result means that the choice among fairness criteria is not a technical problem awaiting a clever solution but an ethical and political choice about which kind of unfairness a system-designer is willing to accept.

This has significant implications for how algorithmic systems ought to be governed. If perfect, criterion-satisfying fairness is unavailable, the legitimate response is not to abandon quantitative decision-making, since unaided human judgment carries its own, often less visible, biases and offers far weaker means of accountability. The more defensible response is to make the choice among fairness criteria explicit, contestable, and subject to public and democratic input, rather than leaving it to be settled implicitly by a vendor's default configuration or an engineer's unexamined preference. Some jurisdictions have begun requiring algorithmic impact assessments before high-stakes automated systems are deployed, and some researchers now argue for context-specific fairness criteria, treating a lending algorithm and a bail algorithm as posing distinct ethical questions rather than instances of one general fairness problem to be solved once.

The broader lesson extends beyond any single technical domain. Algorithms do not remove value judgments from consequential decisions; they relocate them, from the discretion of an individual official to the design choices embedded, often invisibly, in a model's training data and its chosen fairness metric. Recognizing that relocation, rather than mistaking automation for neutrality, is the necessary first step toward governing these systems responsibly.