British Grading Debacle Shows Pitfalls of Automating Government

LONDON — Even after a final term with schools closed for the pandemic, Sam Sharpe-Roe was optimistic about the coming school year. Teachers from his West London school had given him grades — three A’s and one B — that were strong enough to secure him a spot at his first choice of university next month.

But after the British government used a computer-generated score to replace exams that were canceled because of the coronavirus, all his grades fell and the college revoked his admission.

Mr. Sharpe-Roe, along with thousands of other students and parents, had received a crude lesson in what can go wrong when a government relies on an algorithm to make important decisions affecting the public.

Experts said the grading scandal was a sign of debates to come as Britain and other countries increasingly use technology to automate public services, arguing that it can make government more efficient and remove human prejudices.

But critics say the opaque systems often amplify biases that already exist in society and are typically adopted without sufficient debate, faults that were put on clear display in the grading disaster.

Nearly 40 percent of students in England saw their grades reduced after the government re-evaluated the exams, known as A-levels, with the software model. It included in its calculations a school’s past performance on the tests and a student’s earlier results on “mock” exams.

Government officials said the model was meant to make the system more fair, balancing out potentially inflated scores given by some teachers. But students and their parents, particularly those from lower-income areas with struggling schools, were outraged that their futures had been turned over to lines of code that favored students from private schools and wealthy areas.

Even after the government apologized and threw out the computer scores, many students had already lost their slots at their preferred universities, sending the admission process into further chaos.

“These algorithms are obviously not correct,” said Mr. Sharpe-Roe, 18, whose home borough of Ealing is enormously diverse but also divided by race, ethnicity and income. “I know a load of other people who are in a similar situation.”

The outcome, experts say, was entirely predictable. In fact, the Royal Statistical Society had for months warned the test administration agency, Ofqual, that the model was flawed.

“It’s government trying to emulate Silicon Valley,” said Christiaan van Veen, director of the digital welfare state and human rights project at New York University. “But the public sector is completely different from private companies.”

As an investigator for the United Nations, Mr. van Veen studies how Britain and other countries use computers to automate social services. He said the techniques were being applied to policing and court sentencing, health care, immigration, social welfare and more. “There are no areas of government that are exempt from this trend,” he said.

Britain has been particularly aggressive in adopting new technology in government, often with mixed results. Earlier this month, the government said it would stop using an algorithm for weighing visa applications after facing a legal complaint that this was discriminatory. A few days later, a British court ruled against the use of some facial-recognition software by the police.

The country’s automated welfare system, Universal Credit, has faced years of criticism, including from the United Nations, for making it harder for some citizens to obtain unemployment benefits. Britain’s contact-tracing app, which the government had said would be key to containing the coronavirus, has been delayed by technical problems.

“There is an idea that if it has an algorithm attached to it, it’s novel and interesting and different and innovative, without understanding what those things could be doing,” said Rachel Coldicutt, a technology policy expert in London who is working on a book about responsible innovation.

Those who have called for more scrutiny of the British government’s use of technology said the testing scandal was a turning point in the debate, a vivid and easy-to-understand example of how software can affect lives.

Cori Crider, a lawyer at Foxglove, a London-based law firm that filed a complaint against the grading algorithm, said the problem was not the use of technology itself but the lack of transparency. Little is known about how the models work before they are introduced.

“There has been a tendency to compute first and ask questions later,” said Ms. Crider, who also brought the legal challenge against the visa algorithm. “There’s been a refusal to have an actual debate about how these systems work and whether we want them at all.”

For years, Britain has heralded technology as a way to modernize government and provide social services more efficiently. The trend has spanned several administrations but has been given fresh momentum under Prime Minister Boris Johnson.

His top political adviser, Dominic Cummings, has argued forcefully that Silicon Valley thinking is needed to create high performance government, including new workers in areas like data science and artificial intelligence. He has expressed admiration for the “frontiers of the science of prediction.”

In response to the coronavirus, Britain has sought help from companies like Palantir, a Silicon Valley analytics firm that was hired to manage data for the country’s National Health Service. A London-based artificial intelligence firm, Faculty, is working on predictive systems to help track the virus.

In another embarrassing misstep, the government decided to build its own contact-tracing app rather than using technical standards set by Apple and Google, despite warnings that it would have limitations. The release has been delayed for months.

Britain is not alone in turning some decisions over to computer systems. In the United States, algorithms are used by police departments to determine where officers patrol and by courts to set prison sentences. In Spain, the monitoring group Algorithm Watch identified a system being used to predict households at risk of domestic violence. The Netherlands abandoned the use of a system to detect welfare fraud after a judge said it was unlawful.

The techniques are often pitched as apolitical, but researchers say they disproportionately affect lower-income and minority groups.

“One of the great benefits of these tools for governments is it allows them to portray the decisions they are making as neutral and objective, as opposed to moral decisions,” said Virginia Eubanks, an associate professor at the State University of New York at Albany, whose book, “Automating Inequity,” explores the topic.

In Britain, the political fallout of the grading mishap dominated the news and led to calls for the country’s education minister to resign. Students protested outside Parliament, chanting expletives at “the algorithm.”

Critics say the experience shows the risks ahead as more sophisticated tools like artificial intelligence become available and companies pitch them to public agencies.

Mr. Sharpe-Roe said “there’s a lot of anger” at having his fate set by an algorithm. After struggling to regain his lost spot at college, he decided to defer for a year to work.