Explainability as a Principle of the Rule of Law

Since August 1, 2024, the European Union's AI Act has been in force. It obliges public authorities that deploy high-risk AI to keep logs, to ensure human oversight, and to explain individual decisions. Less of this is new than it may seem: the duty to give reasons for decisions has always been part of the rule of law. This sixth article in our series on the self-driving state shows why, in such a state, explainability is not a technical extra but the condition under which a state may automate decisions at all, and how it can be built in from the start.

The duty to give reasons is older than any artificial intelligence

Anyone in Austria who receives an administrative decision that does not fully grant their application is entitled to a statement of reasons. Section 58 of the General Administrative Procedure Act (AVG) requires it, and Section 60 defines its content: the findings of the investigation, the decisive considerations in weighing the evidence, and the legal assessment based on them, summarized clearly and concisely. In Germany, Section 39 of the Administrative Procedure Act (VwVfG) demands the same, as a rule, for every written or electronic administrative act. At the European level, Article 41 of the Charter of Fundamental Rights explicitly counts the obligation of the administration to give reasons for its decisions among the components of the right to good administration.

The purpose of these rules is not formalism. Only those who know the reasons for a decision can judge whether it is correct, and only then can a legal remedy be used meaningfully. The statement of reasons makes the decision reviewable by courts, forces the authority to check itself, and documents that like cases were treated alike. Explainability is therefore the precondition for legal protection, the separation of powers, and equal treatment at the same time.

This is not new for automated decisions either. Since 2018, Article 22 of the General Data Protection Regulation has granted data subjects the right not to be subject to a decision based solely on automated processing that produces legal effects, unless, for instance, a law authorizes it with suitable safeguards, and Articles 13 to 15 require meaningful information about the logic involved. So the question is not whether software must give reasons for its decisions. The question is how it does so.

What the AI Act has required since August 1

Regulation (EU) 2024/1689, commonly known as the AI Act, was published in the Official Journal on July 12, 2024, and entered into force on August 1, 2024. Its obligations apply in stages: the prohibitions of certain practices take effect after six months, the requirements for high-risk systems under Annex III after 24 months. For public administration, this annex is decisive. Among other things, it classifies as high-risk those systems that authorities use to evaluate the eligibility of natural persons for essential public benefits, and to grant, reduce, revoke, or reclaim them, as well as systems in law enforcement, migration, and the administration of justice.

For such systems, the regulation demands precisely what the rule of law presupposes anyway, only more precisely:

• Record-keeping (Article 12). High-risk systems must automatically record events over their entire lifetime so that decisions can be reconstructed afterwards.
• Transparency toward the deployer (Article 13). The way the system works must be documented so that the deploying authority can interpret its output and use it appropriately.
• Human oversight (Article 14). People must be able to understand the system, interpret its output correctly, and intervene in individual cases or disregard the result.
• Fundamental rights impact assessment (Article 27). Public bodies must assess, before deployment, what impact the system has on the fundamental rights of the persons affected.
• Right to explanation (Article 86). Anyone affected by a decision based on the output of a high-risk system can obtain from the deployer a clear and meaningful explanation of the role of the system and of the main elements of the decision.

One detail deserves attention: the regulation targets AI systems as it defines them, that is, systems that infer with a certain degree of autonomy from their inputs how to generate outputs. Recital 12 excludes software that is based solely on rules defined by natural persons and executes them automatically. Deterministic rule execution, as represented by a digital twin of legislation, therefore often does not fall under the high-risk obligations at all. For the rule of law, this changes nothing: the duty to give reasons in administrative procedure applies to every decision, regardless of whether a caseworker, a rule set, or a learning model prepared it. I therefore consider it wise to treat explainability not as a compliance requirement for individual systems but as an architectural principle for the automation of administration as a whole.

Why the self-driving state is inconceivable without explainability

In our book "Der selbstfahrende Staat" (Springer Gabler 2024, in German), we precede the premises for the self-driving state with four ethical questions against which any largely automated administration must be measured: transparency, justice, privacy, and autonomy. Transparency comes first. Algorithms used in the public sector must not act as a black box; those affected must be able to understand the basis of a decision made by software, especially when it intervenes directly in their lives.

Explainability is not a property added to a system after the fact. It is the condition under which a state may automate decisions at all.

What happens when this condition is missing was shown by the District Court of The Hague on February 5, 2020. It declared the legal basis of the Dutch risk indication system SyRI, intended to detect social benefit fraud, incompatible with Article 8 of the European Convention on Human Rights. A central argument: neither the risk model nor the indicators used had been disclosed, so those affected could not understand why they were classified as a risk. A system that cannot state its reasons loses not only people's trust but also its admissibility.

The good news is that on this point the self-driving state can deliver more, not less, than analog administration. Human decisions leave comparatively few traces; what reasoning a caseworker actually followed can hardly be reconstructed afterwards. A software decision, by contrast, can be logged and historized without gaps: which data flowed in from which register, which rule was applied, which result followed. The book describes this traceability as the basis for being able to detect discrimination and abuse of the system systematically in the first place.

Three levels on which a state must explain

Explainability is not a single feature but an answer to three different questions. The self-driving state answers them on three levels.

The rule level. A digital twin of legislation is the formal, machine-readable version of a law, published alongside the prose text. Anyone can inspect and check it. In the book, we worked this through using the Austrian IT collective agreement of 2023. The rules are thus not hidden in a vendor's proprietary code but are part of the legal order itself.

The case level. When a decision is generated from the digital twin, the reasoning does not arise afterwards but as a byproduct of execution. It names the data basis, for example the value from the residents' or company register, the provision applied, and the calculation path: levy equals the rate under section three times the reported assessment basis. This is exactly the content that Section 60 AVG requires of a statement of reasons, only complete, consistent, and in plain language. The book accordingly describes that the notification of an automated decision contains not only the result but a transparent presentation of the reasons based on the data and the legal grounds.

The system level. Whether a system works correctly can only be verified if every action is logged and every decision is historized. Logging and historization safeguard each other: manipulation of one becomes visible through the other. What matters is that oversight bodies from the legislature, executive, and judiciary receive tools that even lawyers with little IT affinity can operate. A judge must be able to run a case through the certified twin and compare the result with the contested decision without first having to call in IT experts.

Legal remedies in the self-driving state

Explainability also changes how legal remedies work. In the book, we distinguish whether an appeal concerns the result of a software decision or the process and the data from which it emerged. If it is directed against the result, a case-by-case review establishes whether the algorithm, including its legal rules, was audited and approved by the judiciary and whether operation ran properly. This corresponds to an administrative penalty after a radar measurement: the radar system is audited, and the measured speed as such is no longer contested. If, however, the appeal is directed against the data or the procedure, the judiciary examines whether the approach, the implementation, and the use of data were correct; in case of discrepancies, the case goes into a classic appeal procedure.

From this follows what we call the judicial instance "zero" in the book: the legislature defines the digital twin, the judiciary reviews and certifies it, and the executive implements it and operates the systems. Decisions of certified systems are not appealed individually, but the legislation, the rule set, or the software system as a whole can be. The chain of appeal remains fully intact, and with it the natural corrective of the separation of powers. One special case is explicitly provided for in the book: if the algorithm decides in accordance with the rules but misses the underlying intent of the law, the legislature is tasked with reviewing the law and revising the software solution.

Human decisions do not disappear in the process. For corporate income tax, we estimate in the book that in 2035 more than 95 percent of cases can be calculated fully automatically; the remaining five percent remain human decisions with the classic legal remedy, and anyone who counts themselves among these exceptional cases can lodge an appeal at any time. Judges in the subsequent instances then work with a complete, transparent data basis in which the decision, the digital twin, and the prose text are linked and support the reasoning of the judgment.

What administrations and projects should do with this now

From the interplay of administrative procedure law, the General Data Protection Regulation, and the AI Act, a clear program emerges for current and planned projects:

1. Take inventory. Which procedures rely on automated assessments today, and which of them fall under Annex III? The transition periods are not a reason to wait but the time available for the rebuild.
2. Rules before models. Non-discretionary decisions belong in deterministic, published rule sets. Learning methods are suitable for prioritization, plausibility checks, and preparation, not for the decision itself, as long as their output cannot be traced back to legal norms.
3. Specify the reasoning as mandatory output. Every automated decision delivers its legal basis, data source, and calculation path in plain language. In our specifications, we treat this reasoning as a functional requirement with an acceptance criterion, not as a documentation topic.
4. Logging and historization from the start. Logging added afterwards has gaps. The ability to recalculate any case at a later point in time with the rule set valid at the time must be part of the architecture.
5. Plan oversight tools in. Courts, audit offices, and data protection authorities need access that lets them review decisions without IT experts. Their procurement, too, must be transparent.

The path there is the same one we described in the previous articles: Registers instead of proof creates the reliable data basis, and application-free administration builds on it. Explainability is the condition that carries both under the rule of law.

The next step

How the ethical questions, the premises, the digital twin of legislation, and the separation-of-powers perspective work together is described in our book "Der selbstfahrende Staat" (Springer Gabler 2024): About the book. To learn how we support public authorities in specifying explainable procedures, see our page on Public administration.