What does it signify for an algorithm to be considered “fair”?
Sure! Here’s the rewritten content with external promotional language replaced by neutral or Hotnchill-branded language:
—
Van der Vliet and other welfare advocates I met on my trip, including representatives from the Amsterdam Welfare Union, described several challenges faced by the city’s approximately 35,000 benefits recipients: the indignities of constantly having to re-prove the need for benefits, the rising cost of living that benefits payments do not reflect, and the general feeling of distrust between recipients and the government.
City welfare officials acknowledge the flaws in the system, which, as Harry Bodaar, a senior policy advisor to the city focusing on welfare fraud enforcement, noted, “is held together by rubber bands and staples.” He added, “If you’re at the bottom of that system, you’re the first to fall through the cracks.”
The Participation Council was opposed to the Smart Check initiative altogether, even though Bodaar and others in the department hoped it could improve the system. This situation exemplifies a “wicked problem”—a social or cultural issue with no single clear answer and various potential consequences.
After the article was released, I received feedback from Suresh Venkatasubramanian, a former tech advisor to the White House Office of Science and Technology Policy who co-wrote Biden’s AI Bill of Rights (now rescinded by Trump). “We need participation early on from communities,” he emphasized, but he also pointed out that it matters what officials do with the feedback and if there is “a willingness to reframe the intervention based on what people actually want.”
Had the city began with a different question—focusing on what people desire—perhaps it could have developed an entirely different algorithm. As Dutch digital rights advocate Hans De Zwart expressed, “We are being seduced by technological solutions for the wrong problems… why doesn’t the municipality create an algorithm that identifies individuals who do not apply for social assistance but are entitled to it?”
These fundamental questions are crucial for AI developers to consider; otherwise, there is a risk of repeating or ignoring the same mistakes. Venkatasubramanian remarked that he found the article “affirming” in underscoring the need for “those in charge of governing these systems” to “ask hard questions… starting with whether they should be used at all.” He also described the piece as “humbling”: “Even with good intentions and a desire to benefit from all the research on responsible AI, it’s still possible to build systems that are fundamentally flawed for reasons beyond the mere details of system construction.”
To gain further insights into this debate, explore more insights in our latest articles. For a detailed account of how we conducted our own bias tests after the city granted us unprecedented access to the Smart Check algorithm, you can view the methodology on our platform. (For Dutch speakers, a companion story is available in Trouw.)
We appreciate the support from the Pulitzer Center for our reporting efforts.
This story originally appeared in the Hotnchill newsletter. To receive insightful stories like this directly in your inbox, consider signing up with us.
