Where Autonomy Reaches Its Limits

The previous posts in this series were about what the self-driving state can automate: applications, evidence, registers, entire life situations. This post turns the question around. It describes three limits where autonomy in public administration deliberately stops: discretion, judicial decisions, and interference with fundamental rights. Drawing these limits precisely does not mean less automation; it means more trust in the cases that may be automated.

Three limits that no maturity level moves

The evolutionary stages from our book "Der selbstfahrende Staat" (Springer Gabler 2024, in German) describe how much an administration automates: analog, digital, automated, self-driving. In the automated state, 80 percent of decisions are made according to predefined algorithms, while the remaining 20 percent are special cases handled by people. The self-driving state shifts this ratio further because intelligent software can prepare many special cases as well. What the stages do not answer is which decisions fundamentally stay with humans, no matter how mature the technology becomes.

Our answer in the book is unambiguous: people in politics, legislation, and the judiciary retain the central role in decisions that have significant consequences for the lives of individuals or for society. Three concrete limits follow from this principle, and I will go through them one by one in this post. They are not technical restrictions that will fall tomorrow, but design decisions that a constitutional state makes before it lets software decide.

In doing so, final responsibility will continue to remain in human hands.

That is how the chapter on the DNA of the self-driving state puts it, and no generation of models has changed it. For people to actually exercise this responsibility, however, they need tools that deliver complete, comprehensibly prepared data that can be verified back to its source, instead of piles of files and callbacks.

Limit one: discretion is not an arithmetic problem

Administrative law drew the first limit before AI systems arrived at scale. In Germany, Section 35a of the Administrative Procedure Act (Verwaltungsverfahrensgesetz) permits a fully automated administrative act only where a legal provision allows it and neither discretion nor a margin of judgment exists. In his guest contribution to the book, Jörn von Lucke draws the same dividing line from a technical perspective with a view to legislation: decisions without discretionary or decision-making leeway are legally rather unproblematic because they can already be automated today; everything else cannot be automated without further ado.

What does that mean in practice? A law rarely consists only of discretion or only of calculation rules. The question of whether someone is entitled to a subsidy contains thresholds, deadlines, and calculations that can be checked unambiguously, alongside indeterminate legal terms such as "particular hardship" or "unreasonableness" that require a weighing of interests. The self-driving state fully automates the unambiguous part and hands the weighing part over to a caseworker. What matters is that this handover is not left to chance: where the law provides no executable rule, the software must not invent one. It must visibly mark the case as not automatable and pass it on, together with all the facts it has already verified.

Software is nevertheless far from useless in discretionary cases. In his guest contribution, von Lucke describes what matters: the better AI systems recognize and measure discretionary and decision-making leeway, understand it in light of its consequences, and present options transparently, the more people will trust them. The human then decides on a complete, traceable basis. I draw one further conclusion from this: as trust grows, legislators will deliberately narrow individual margins so that they become automatable. That is the right path: the legislator moves the limit, not the software.

Limit two: the judiciary certifies the systems and keeps the verdict

The second limit concerns the courts. In the book, we propose a model that renowned legal experts dismissed as unworkable in the first review rounds and that we nevertheless consider the only consistent one: the legislature defines the digital twin of the law, the judiciary reviews and certifies it together with its data structures before it makes a single decision, and the executive operates the software. Software decisions can be determined in advance, so they can also be validated in advance. We call this review "instance zero."

The consequence is precisely delimited. Every appeal is examined to determine whether it concerns the result of the software decision or the process and the data behind it. In the first case, a case-specific check confirms that the algorithm, the software, and the legal rules were audited and approved by the judiciary, comparable to a calibrated speed camera whose measurement is not renegotiated. In the second case, the judiciary validates the procedure, the implementation, and the use of data, and if inconsistencies appear, the case goes to the next instance in a classic appeal procedure. The chain of appeals remains fully intact; the highest courts have full access to all software solutions, algorithms, and data sets in order to uncover wrong decisions. Anyone who fears that software escapes the courts should see the opposite: it becomes more verifiable than any administrative practice on paper.

The verdict itself remains human. In the book, we illustrate this with an insolvency procedure: at the highest stage, software analyzes the application completely, draws on historical company data, creditor data, and market data, and issues recommendations for the court and the insolvency administrator. The final decision is made by a judge who evaluates the proposed measures. The same applies to proceedings initiated ex officio: procedural steps run automatically, while judgments continue to be handed down by judges and arguments continue to be presented in person.

What changes for the judiciary are the tools. A judge can load the certified twin of the law, enter the facts of the case, and compare the expected outcome and its derivation with the contested decision. If the two differ, either the implementation was wrong or the input was, and both become visible within minutes. For that, the tools must be as simple as a weather app: a supreme court judge or a public prosecutor cannot be expected to have a computer science degree, and it would be a step backward if the judiciary had to call in IT experts for every review. Human judgments flow back into improving the rules, but through legislation and certification, not through silent retraining of the software.

Limit three: interference with fundamental rights requires approval, not automation

The third limit is the sharpest. The self-driving state is a "transparent state": every action of the administration is documented and traceable. That is exactly what protects against a surveillance state, but only if linking data about people is bound to the same hurdles as an intervention in the physical world. In the book, we draw this line explicitly: viewing the data of a mobile device or all linked data of a person's digital twin is on the same level as a house search. It requires judicial approval that the system enforces technically, not merely provides for organizationally. Data may be used exclusively for defined purposes, and an independent body monitors compliance.

The European legal framework confirms this limit. The AI Act, Regulation (EU) 2024/1689, which entered into force on August 1, 2024, prohibits certain practices such as social scoring by public authorities and classifies, in Annex III, access to public services, law enforcement, migration, and the administration of justice, among others, as high-risk areas, with obligations for logging, documentation, and human oversight. Article 22 of the General Data Protection Regulation has given data subjects since 2018 the right not to be subject to a decision based solely on automated processing that produces legal effects, unless one of the exceptions applies, such as a law with appropriate safeguards that expressly permits it. In his foreword to the book, Jörn von Lucke rightly points out that the AI Act will act as a driver, regulator, and corrective, and that some of the deliberately far-reaching ideas in the book may not be implemented in Europe. I consider that the right division of labor: models of thought may reach further than what a constitutional state permits, so that society draws the limit deliberately rather than by accident.

A special case of the third limit is discrimination through data. A learning system built on historical enforcement data inherits the prejudices that shaped those data. In the book, we use the example of the colorful VW bus that was stopped in traffic far more often than the gray Škoda: the software would overestimate the risk posed by VW bus drivers because people did. That is why safety-related decisions must not rest on experience data without review. We propose an interdisciplinary committee of experts from IT, law, ethics, and social affairs whose members are appointed with an independence similar to that of judges. Finally, the third limit includes analog access: for people who have no technical or cognitive access to digital services, analog alternatives must be provided or maintained so that they can still use the services of the state.

The three limits at a glance

What this means for administrations starting now

Naming limits is easy; anchoring them in systems is work. From our projects in the public sector, five rules emerge that can be implemented regardless of maturity level:

1. Spell out the automation boundary for each process. For every service, it is set down in writing when a decision is made automatically and when it is escalated, for example: automatically when all statutory conditions are met and all required data is present; otherwise to a human. This rule is part of the specification, not the implementation.
2. Design discretionary cases instead of preventing them. The case arrives at the human fully prepared, with verified facts, options, and consequences. The human decision is documented together with its reasoning so that it remains verifiable and the legislator can see where leeway is systematically exercised in the same way.
3. Involve the judiciary before go-live. A set of rules that produces administrative decisions is reviewed and approved before deployment, not only in the appeal procedure. The review tools must be usable without computer science knowledge.
4. Wire interventions to approvals. Any linking of personal data beyond the purpose of the procedure requires a logged judicial approval that the system verifies technically before it releases data.
5. Safeguard learning components. Wherever software learns from data, an independently staffed committee reviews the data basis for bias, and decisions affecting fundamental rights remain deterministically traceable.

These rules are not a brake. They are the precondition for citizens to accept the many automated decisions that save them time, trips, and paperwork. How explainability becomes a principle in this context is described in our post Explainability as a Rule-of-Law Principle; which registers make the unambiguous rule-based part decidable in the first place is covered in Registers Instead of Evidence.

The next step

The three limits are derived in detail in the book "Der selbstfahrende Staat" (Springer Gabler 2024, in German) in the sections on the premises and on the separation-of-powers perspective: About the book. How we support administrations in anchoring automation boundaries in specifications and aligning specialized procedures with them is described on our page Public Administration. In an expert consultation, we will gladly walk through a specific process with you and draw the line together.