Measuring Maturity Instead of Estimating It

In most organizations, the question of where they stand on the road to agentic software development is answered by estimates: the number of Copilot licenses distributed, satisfaction surveys, or the success stories of individual teams. None of that is reliable. This article describes which metric counts at which maturity level, how a maturity assessment locates an organization domain by domain, which system of metrics steers the agentic Software Development Lifecycle, and what is monitored in day-to-day operations after the 90-day measurement.

Why estimates end in pilot islands

In the first article of this series I introduced the maturity model with six levels from L0 to L5; in the second, the quality gates at which people decide. Both remain theory as long as an organization does not know where it stands. And this is where we observe the same pattern in almost every organization: licenses have been distributed, developers report time savings, the acceptance rate of suggestions is being tracked, and at portfolio level neither lead time nor quality nor cost structure changes measurably.

The reason is not that the tools do not work, but that the wrong metric is being watched. The acceptance rate of a copilot measures how often a person adopts a suggestion. It says nothing about whether orders reach production readiness faster, whether the defect rate stays stable, or whether the organization delivers the same output with less staff. Whoever steers by this metric optimizes tool usage and overlooks the operating model. That is the first of the four reasons agentic transformations fail: tools without an operating model. And it is the reason behind one of our principles: steer with evidence, not with anecdotes.

A metric that fits the maturity level says more about where you stand than any self-assessment.

One leading metric per maturity level

The maturity model we use, modeled on the Capability Maturity Model (Paulk et al. 1993) and the DORA research on delivery performance (Forsgren, Humble, Kim 2018), assigns a leading metric to each level.

Two things stand out in this table. First, at the tipping point between L3 and L4 the metric changes its subject: up to L3, what is measured is the interaction between a person and a tool; from L4 onward, it is the outcome of an order an agent has processed end to end. An organization that cannot capture cycle time per order, because there are no delimited orders with machine-readable gate criteria, is not yet at L4, regardless of how capable its models are. Second, the metric at L5 is no longer a speed metric but an efficiency metric with a quality condition: throughput per FTE counts only at stable quality.

The model thus replaces anecdotes with a rule: an organization is at the level whose leading metric it can extract from its delivery systems and actually uses to make decisions.

The assessment: locating each domain, not the organization as a whole

An organization does not have a single maturity level. A bank may be at L2 in CI/CD and release engineering, at L0 in its data and knowledge foundation, and at L3 in individual teams using copilots. An aggregate score would hide these differences and lose the decisive information: which domain limits the progress of the others? Without a consolidated platform there are no reliable agents, without quality gates no autonomy, without governance no regulatory approval.

That is why the maturity assessment works with a capability model that breaks the delivery organization down into domains and rates each capability individually against the six levels. In a current engagement with a large European bank (anonymized, project status first quarter of 2026), the model comprises 41 capabilities in nine domains, mapped to banking supervision, DORA, and the AI Act: engineering practices, CI/CD and release engineering, architecture and platform, resilience and observability, delivery governance and flow, data and knowledge foundation, security, risk and compliance, business and customer alignment, and culture, people and leadership. The position was L2; the program target for the first year is L3 across the board, with agent assistance following from year two.

Three rules make such an assessment robust:

• Evidence instead of questionnaires. The rating of each capability rests on what can be demonstrated in repositories, pipelines, ticket systems, and operational data, not on the self-assessment of the teams. A team that claims to automate its tests is measured by its test coverage.
• Baseline before the first pilot. The baseline metrics are collected during the diagnosis, from the same systems and with the same definitions used for measurement later. Without a baseline, an effect after 90 days of pilot operation can only be claimed, not demonstrated.
• Regulatory mapping. Every capability is assigned to the requirements it satisfies: the AI Act, Regulation (EU) 2024/1689, for human oversight and traceability; Regulation (EU) 2022/2554 (DORA) for ICT risk management and resilience. The gap to supervisory readiness is visible from the start, not only before go-live.

The diagnosis, completed in four to eight weeks, delivers four things: the position per domain, the quantified efficiency potential including the baseline, a prioritized starting portfolio with quick wins, and a draft roadmap in three horizons. It is the fact base on which the board and supervisory bodies decide on the target picture.

The metrics system: one board metric, phase KPIs, guardrails

Once the position is established, steering needs a system of metrics that carries beyond the pilot phase. We work with three levels.

At board level, one metric is enough: delivery efficiency, measured as completed units of work per FTE, quarterly per division, extracted directly from the delivery systems and therefore not open to manipulation. The target picture is sober: output stays stable, and headcount declines through natural attrition because agents absorb the repetitive work. In our illustrative target picture, a division delivers 120 units of work with 25 FTE in the first quarter, that is 4.8 units per FTE, and the same 120 units with 20 FTE in the fourth quarter, that is 6.0. This corresponds to 25 percent more efficiency at 20 percent less headcount over four quarters, without disruption.

Phase KPIs measure progress in detail: the time from idea to validated specification, the time from commit to production readiness, and the share of automated tests. They show in which phase of the lifecycle the agents are having an effect and where handovers still cost time. At L4, cycle time per order is added, measured from acceptance of the specification at the Definition of Ready gate to production release.

Quality guardrails decide whether efficiency counts at all: change failure rate, priority one and two production incidents, defect rate, and test coverage must not deteriorate. The established standard for this comes from the four metrics of DORA research (DevOps Research and Assessment; Forsgren, Humble, Kim 2018): deployment frequency, lead time for changes, change failure rate, and time to restore service. They are not to be confused with the EU regulation of the same name.

The same discipline applies to the interpretation of our own figures. Delivery up to five times faster applies in the agentically orchestrated implementation model, that is, to the pure coding share, with coding costs up to 80 percent lower. Across entire projects including strategy, specification, and steering, the efficiency gain is 20 to 45 percent, and quality rises by around 30 percent through systematic verification. All four values are empirical values from ReqPOOL projects 2023 to 2026, verifiable per project and not scientific evidence. An assessment that promises a fivefold acceleration across the entire project has confused the metrics.

What should not be measured

Equally important is what stays out of the metrics system. Generated lines of code, the number of prompts, tokens consumed, or license utilization measure activity, not effect. They rise with every agent that is added and say nothing about whether the organization delivers faster, better, or cheaper. I consider them the most dangerous metrics in agentic development, because they are so easy to collect and look so convincing.

After the 90-day measurement: monitoring as a permanent state

The pilot phase with two to three lighthouse cells ends with the 90-day measurement against the baseline. It answers the question of whether the operating model demonstrably works in the organization's own environment, and it provides the blueprint for the first rollout wave. Scaling happens only when the metrics are met; quick wins must show their measurable effect within 180 days; deferred initiatives are reassessed in the next planning cycle.

After the pilot, measurement does not stop; it changes character. Three mechanisms carry monitoring in ongoing operations:

1. Metrics from the process, not from reports. Because gate criteria are stored in machine-readable form and every agent step is logged, cycle time, test coverage, and change failure rate arise as a by-product of the work. Nobody has to collect them afterward, and nobody can embellish them.
2. The Definition of Operability as the fourth gate. In operations, agents take over monitoring, anomaly detection, and self-healing proposals; verification agents produce SLA reports, root-cause analyses, and quality trends; people decide on escalation, carry incident responsibility, and record the lessons learned. Insights from operations flow back into backlog and specification.
3. Maturity as a recurring measurement. Scaling follows portfolio logic: every domain is measured against the maturity model and rolled out in waves, and every wave supplies the evidence for the next. The north star becomes binding through board sponsorship, budget, and exactly this one metric in corporate steering.

In reqCoder, the complete agentic Software Development Lifecycle of ReqPOOL, which is already in use internally, this monitoring is built in. Each of the four stages leaves evidence at its gate: scope proof and estimate after the prototype, reqChecker review and security scan in the beta, Signoff acceptance report and audit trail before deployment, monitored operations in production. From the same evidence we read the metrics by which we steer our own delivery. Whoever commissions a stage sees the measurements before deciding on the next.

What this means for your organization

• Measure your position before you buy tools. An assessment per domain in four to eight weeks delivers the baseline without which no licensing program can demonstrate its effect, and prevents pilots from becoming islands.
• Collect the leading metric of the target level early. An organization aiming for L4 should measure cycle time per order in its pilot cells from the start, even if it looks poor at first.
• Fix baseline and guardrails before the pilot. Otherwise the 90-day measurement becomes a matter of interpretation.
• One metric for the board, not twenty. Delivery efficiency per FTE with four stable guardrails is decision-ready; a dashboard with forty metrics is not.
• Drop activity metrics. Lines of code, prompts, and tokens belong in cost accounting, not in steering.

The next step

How we set up the maturity assessment, the metrics system, and the 90-day measurement in your delivery organization is described on the page Agentic Software Development Lifecycle. How evidence arises at the gates of our own agentic pipeline is described on the page reqCoder. In an expert meeting, we determine together which leading metric your organization actually collects today and what that says about your position on the maturity path.