Register › Concepts › Decisions made by machine
Decisions made by machine
An automated system may assist a decision about a person, and in no case may it take one.
The rule
A decision producing legal effects upon a person is taken by a human being who is identified and accountable, under Article 51(5). Systems may compute, sort, flag and advise. The signature, and the liability that goes with it, belongs to a named person who can be asked why.
Reasons, on demand
Article 51(6) entitles any person to the reasons for a decision affecting them, including the logic, the inputs and the weightings of any system that assisted it. The same paragraph gives a right to review by a person who had no part in the original decision. An unexplainable model is therefore unusable for the purpose, whatever its accuracy.
Three uses prohibited outright
Scoring, ranking and profiling of persons for the allocation of provision, of housing, of education or of liberty are prohibited by Article 51(7). The prohibition attaches to the technique, so it survives consent, statutory authority and any finding that the system performs well.
Where it is difficult
The plan is a pressure point, and Article 90(3) answers it: decisions of the plan are taken by assemblies and councils composed of persons, computation assists them, and in no case does it decide. Allocation is the other, and Article 113(4) applies the rules of fair allocation through identified persons.
Who keeps it under control
A division of the Defender of Rights under Article 254 keeps a public register of every automated system used in administration, planning, allocation and provision, stating its purpose, its inputs, its logic and the person accountable for each decision it assists. The division can order a prohibited system to stop, and only a court can set that order aside.
The data behind it
Processing runs on a lawful basis, for a stated purpose, to the minimum extent that purpose requires, and for a limited period. Consent extracted as the price of an essential service is void. Trade in personal data is prohibited, and data generated by public services and planning systems pass to the knowledge commons of Article 10 in aggregated and anonymised form.