(Un)supervised
How fair lending oversight pushed lenders to proactively measure their models — and what it says about governing automated decisions and AI
If you’ve been around for a few editions of Elicitation, you know we love a good historical analogy. By untangling what and whom AI governance is trying to govern and how similar challenges have been managed before, we can uncover signals that guide us toward policy ideas that might be suited to the current moment.
Across basic systems and advanced ones, a goal many researchers and advocates share is the creation of regimes that motivate pre-deployment testing—with an assumption that evaluation results will incentivize the mitigation of risks and lead to fewer harms. This week at the FAccT conference in Montreal, one of my longtime collaborators Emily Black presented research conducted over the last two years on a decades-old yet still surprisingly enigmatic example of this theory of change: fair lending.
All of us had spent years within research communities full of thoughtful people trying to work out how to keep algorithms from discriminating, with conversations often coming to the same conclusions: that developers should be proactively testing systems they build before rolling them out, and that there seemed to be a concerning lack of incentive to do so. What few new laws have passed in recent years trying to shift those incentives have, unfortunately, proven limited and succumbed to political pressure.
At the same time, the regulatory regime that evolved over decades to support compliance with fair lending laws has urged banks to do the exact activity researchers and advocates want to see. U.S. financial institutions have been legally barred from discriminating in credit for roughly half a century—the Fair Housing Act in 1968, the Equal Credit Opportunity Act in 1974—and for most of that time they’ve made lending decisions with statistical and algorithmic models. Fair lending programs are very likely the longest-running example of algorithmic fairness actually operating in the world, and they offer lessons that resonate beyond the walls of regulated banks.
Regulatory supervision has driven proactive measurement
One of the most striking findings of our research was the role of regulatory supervision in driving investment in model testing. Folks who spoke with us described that financial institutions don’t send their models through structured pre-deployment review simply because the law says so, or even because they might be sued (indeed, firms often opt to take on risk to a larger degree than people realize). In the case of fair lending, at least, practitioners described that their institutions largely do this work because regulators can physically show up and ask for receipts, regardless of whether they believe suspicious practices are taking place.
Through supervision and examination, fair lending regulators are empowered to direct financial institutions to address potential deficiencies. If such deficiencies are not resolved, they can be elevated to formal enforcement proceedings. The practitioners we spoke to were unambiguous that it has been this proactive scrutiny, not litigation, that has driven their institutions’ their work.
“If it wasn’t for the fair lending legal risk, we wouldn’t even do this review.”
research participant L7
A regulatory structure involving proactive supervision hasn’t shown up in any AI frameworks I’ve seen, but our research suggests it’s worth a closer look. This sort of regulatory attention appeared to drive meaningful organizational investment within covered banks: to a one, every institution subject to these laws had a structured fair lending team tasked with conducting arms-length tests of relevant models, documenting results, and indicating to the business if models posed a risk, while organizations outside of this regulatory regime largely did not. Models consistently underwent internal scrutiny from teams whose incentives are not directly tied to the model’s release, and structured channels existed to surface findings to internal decisionmakers.
“Unfortunately, no one does this stuff out of the good and kindness of their heart. They do it because they’re afraid of regulation.”
research participant T5
On the other hand, the existence of these teams was not necessarily correlated with firms changing their behavior around what models to deploy. Practices varied widely when it came to what exactly was measured using which methods, and how much regression to model accuracy was tolerated when considering whether a model was viable to launch. Whether to fix a disparity at all was seen to be a decision by the business—revenue-generating parts of the organization as well as firm executives—and some firms seemed perfectly happy to eat the risk. A great deal of work, we found, is therefore channeled into defensive documentation: building a tidy paper trail to justify a disparity to an examiner rather than working to reduce it.
“Many institutions want to have a credible performance of doing the [search for a less discriminatory alternative (LDA)] search because they’re mostly concerned with being able to display something that satisfies the regulator…there is a large gap between ‘we need to demonstrate that we’re trying here’ versus ‘we actually want to do an LDA search that finds better alternatives’.”
research participant E11
We don’t know the counterfactual, what would have happened under a different regulatory regime. What would the state of discrimination in lending be but for these proactive measurements, motivated by supervision? Potentially far worse than it is today, if firms felt they could deploy profitable models with complete impunity regardless of their impact.
What fair lending reveals about compliance dynamics
The history of fair lending offers an illuminating, if cautionary, tale for those looking to regulation as a lever to solve the many ills of algorithmic decision making. Regulation with vigorous and proactive enforcement can clearly foster desirable practices, but our research suggests that the success of even such potent regulation is not a sure thing.
In particular, our findings pull back the curtain on dynamics that might emerge in the face of demands for pre-release testing of other types of AI models and sheds light on how firms may respond to the imposition of more external scrutiny, such as via third-party auditors. When companies have latitude to define their own testing regimes and justify away discrepancies, we found that the best case scenario enabled continued innovation in measurement and investment in mitigating harms—but the worst involved robust governance regimes turning into compliance theater.
In many cases, a notable factor that seemed to determine which scenario an organization was closest to was the tenacity of individuals opting to fight uphill battles within their firms in the face of diverging incentives. And the very institutional design set up to create arms-length review (financial institutions typically have a “first line of defense,” in this case the team tasked with building models, and a “second line” tasked with handling risk, including conducting measurements using data the modeling team was not permitted to access) contributed to perceived inefficiencies in addressing risks that review detected.
Notably, the research revealed that when regulators express comfort with a particular approach to measuring and mitigating bias, compliance teams can be incentivized to stick with those approaches even when emerging evidence suggests they are not effective. Convincing regulators that a new and more effective measurement or mitigation method should be viewed favorably, we learned, was perceived to require substantial effort and to risk inviting even more regulatory scrutiny into the firm’s practices. Several internal practitioners seeking to drive substantive rather than perfunctory compliance found themselves caught in this counterproductive dynamic.
The work ahead
Since we began this research, civil rights law has come under attack. We documented these decades-old practices at very nearly the moment the current administration began dismantling the regime that produced them, moving to strip disparate impact liability from the rule implementing ECOA, signaling it would stop bringing bias-assessment supervision based on statistical evidence, and issuing an executive order asserting that the disparate impact theory of liability violates the Constitution. Our hope was to inform efforts to improve a strong status quo; now, even reviving previous protections will require concerted effort. Some states are moving to fill in the gaps, but they are stretched thin.
In parallel, policymakers are proposing AI-specific interventions to challenge a variety of unsafe practices, and considering (and passing, in a few states) laws to require companies to establish and comply with internally defined AI safety plans that involve pre-deployment testing. Anything that motivates companies to check their own work before pushing potentially unsafe tools out the door seems better than relying on corporate goodwill, but our research underscores the importance of being realistic about what such efforts accomplish and how much work remains ahead of us.
The financial industry has chafed at vigorous supervision, and regulatory capture is a recognized failure mode—but the dismantling of the Consumer Financial Protection Bureau has left consumers far worse off. At the same time, recent events around Mythos, as well as demands that AI models conform to skewed definitions of political neutrality, offer cautionary signals around how even well-intentioned proposals to augment government oversight of AI might go terribly awry. Perhaps the machinations of the financial sector around model testing in the face of fair lending regulatory regimes can help illuminate a way forward, especially as we race to establish and shore up guardrails around other consequential AI systems.
The full paper, “The Fair Lending Model: How the Longest-Running Algorithmic Fairness Programs Work in Practice,” goes into far more detail, and is available here.
A selection of other FAccT ‘26 papers I found interesting
Algorithm Auditing Policies Rest on Flawed Assumptions About Public Sector Systems by Nel Escher, Nikola Banovic, and Ben Green. Narrowly scoped audit requirements can overlook meaningful harms, miss system-level issues, and overlook public agencies’ capacity to correct flawed automated systems.
Guardrails versus Gatekeepers: Understanding Product Managers’ Ethical Decision-Making in Generative AI by Genevieve Smith and Natalia Luka et al. This qualitative research explores the role of PMs in promoting responsible use of genAI. It highlights the conditions where individual actors seem able to exhibit agency and where larger organizational incentives appear to be needed.
Helpful, Harmless, Honest? RLHF as Survey Design and Content Moderation by Samantha D’Alonzo, Frauke Kreuter, and Serena Booth (a nonresident fellow at the AI Governance Lab!). RLHF is a common technique used for model post-training and alignment that resembles both survey research and content rating; the researchers find current RLHF pipelines fall short of well-established practices in both contexts.
To Build or Not to Build? Factors that Lead to Non-Development or Abandonment of AI Systems by Shreya Chappidi and Jatinder Singh. This paper considers what motivates organizations to abandon AI development, and how to support such abandonment.
Characterizing Delusional Spirals through Human-LLM Chat Logs by Jared Moore et al. The researchers analyze chat logs to identify warning signs of how safeguards may degrade in extended multi-turn conversations.