An opaque algorithm that has detected errors decides who receives public aid from the social bonus | Technology

Since 2017, the Government of Spain has granted the most vulnerable families a discount on the electricity bill, the so-called social bonus. In order to access this benefit, a series of conditions must be met: prove a low income level, collect the minimum pension, be unemployed or ERTE or have a large family. Applications are not evaluated by a group of officials, but by an algorithm, BOSCO. And that algorithm is wrong. The Civio organization demonstrated in 2019 that the automatic application had denied aid to people who were entitled to it. The Government corrected the error, but did not agree to open its source code to be able to search for other possible mismatches. Civio went to court and last February, three years later, the justice system agreed with the Executive, arguing reasons of national security.

The case of BOSCO is illustrative of an unresolved and increasingly relevant debate: how societies should articulate algorithmic governance. Do we citizens have the right to know how the automatic systems that make decisions about the allocation of public resources work? The main motivation of the Civio team in following the case is, in fact, to prepare the ground for possible future complications with other algorithms. “We want to pave the way in case we later ask for information about another more complicated system. The foundations must be laid so that next time the Administration does not take four years to open the source code of its algorithms”, explains David Cabo, co-founder of Civio.

The error they detected was gross. Until 2020, when being unemployed was incorporated, there were three entry routes to access the social bonus discount: have low income (a maximum of 24,000 euros if other special conditions are met), be a large family or benefit from a pension minimum disability or retirement without additional income. “This last route leaves out other types of pensioners, such as widows, and pensioners who receive some income or, after the last reform, income that exceeds 500 euros per month. Of course, in many cases, these pensioners can access through the other entrance door, the rent one, since they do not exceed the maximum income, ”says Eva Belmonte, Cabo’s partner and head of the BOSCO investigation, on the Civio blog. . “But if those people check the pensioner box, which they are, the application rejects their application. He does it despite checking his income and knowing that they do have the right to rent.

They also verified that when any of the members of the family unit did not give their consent for their income data to be examined, they were denied. Even if they belonged to a large family and therefore were entitled to help regardless of their income.

Tracking down a problem

“We realized that something was wrong when we saw that much fewer people applied for the social bonus than were entitled to it,” says Cabo. According to calculations by the Government itself, in 2019 some 5.5 million households were entitled to aid, but the Executive estimated that only 2.5 would request it. In the end, 1.1 million families did, a low figure. Civio prepared, together with the CNMC, a guide so that those interested knew how to fill out the application, a complex task. Because there are many papers to present and not precisely before the Administration. Consumers have to ask the electricity companies for the social bonus; these transfer the data of their clients to the Government, which in turn inserts them into BOSCO. It is the computer system that then tells the operators which of their clients can enjoy the discounts.

After a while, some of those who used the guide and who knew they were eligible complained that they had not been granted the aid. The Civio team reviewed your application and saw that everything was in order. Then he asked the Ministry of Ecological Transition for access to the technical documentation with which BOSCO had been prepared. “We saw that there had been problems in the way they had translated the Decree Law into technical specifications for automatic application. What we don’t know is if there are more errors in the translation of those specifications in the source code”, says Cabo.

That’s why they asked to be shown the guts of the algorithm. The Central Contentious Administrative Court number 8 ruled in February that releasing the code would violate the government’s right to intellectual property and would affect both public security and national defense. Sources from the Ministry emphasize this aspect: facilitating access to the source code, they maintain, could lead to hacking of the system. From Civio they counter that they do not ask for passwords at all, but to be able to study the architecture of the system to verify that it does not harm anyone. The organization has already announced that it will appeal the sentence and that it will reach the Supreme Court.

Algorithmic governance and security

Gemma Galdon, director of the Eticas consultancy, believes that the sentence was foreseeable. “When access to algorithmic code is requested, the owner always hides behind intellectual property. I am not aware of any legal case in which the company’s right to protect it has been outweighed by the right to access”, she stresses.

Galdon and his team, who have also been on the trail of BOSCO for some time, propose a different strategy of action. “We are going to make a formal claim to find out if the legal conditions surrounding the application of an algorithm in public affairs have been met. National and community law says that when a public matter is submitted to automated decisions, that algorithm must be audited and subjected to an impact assessment, ”he adds.

For Ana Valdivia, a researcher at King’s College London specializing in the social impact of algorithms, this case exposes the so-called efficiency paradox, in reference to the faith that governments have in the infallibility of these computer programs to manage public affairs. “Some people think that algorithmic systems make decisions more efficiently because they process information faster than humans and are cheaper than labor. But implementing an algorithm in real life is the opposite of efficient”, he writes in an article published at the beginning of the month together with the jurist Javier de la Cueva. “You have to digitize the process, gather the data, clean it, train the algorithm, evaluate it, implement it in the system and test that it works in real life, among other tasks. Also, mistakes like BOSCO’s can seriously hurt efficiency.”

These two experts believe that the case of the social bond illustrates the friction between the law and the algorithms. “While citizens should have the right to know if a decision has been made by an algorithm, the courts rule in favor of algorithmic opacity,” they argue.

The impossibility of accessing the source code of an algorithm that makes decisions on public matters has, in his opinion, several relevant implications. Mainly it prevents knowing “if the final result has been reached through reasoning that does not contemplate the law”. In the source code where you can see the conceptual categories and delimitations applied by the engineers, which in the case of BOSCO led to error. “A black box system, in which the rules determining why one case is preferable to another are unknown, affects the ability of the parties to appeal against the decision,” the scholars conclude.

You can follow THE COUNTRY TECHNOLOGY in Facebook Y Twitter or sign up here to receive our weekly newsletter.