2021

Dutch tax authorities designed a self-learning automated system to efficiently detect fraud in childcare benefit claims. The goal was to use data-driven precision to protect taxpayer money and ensure fairness.

But the self-learning algorithm learned to racially discriminate. It learned to flag individuals with dual or non-Dutch nationality and low income as high-risk.

Once flagged by the algorithm, the system automatically imposed brutal penalties. Benefits were immediately suspended. Large sums were demanded in repayments, without room for human review or appeal. This digital witch-hunt affected 26,000 families.

Parents were plunged into debt, lost their homes and jobs, the were suicides and at least 3,532 children were taken from their families and forced into foster care. The scandal, known as the Toeslagenaffaire, exposed a catastrophic failure of both technology and government.

The scandal ultimately lead to the resignation of the entire Dutch government in 2021.