Measuring the Loop: Cycle Time, Yield, Re-test Pass Rate and the Metrics That Mislead

A loop that is run well and measured badly will drift towards what its measures reward. This chapter asks what the purple loop should report about itself, to the people who run it and to the board that is accountable for it, and which numbers to refuse. It concerns three stages. P3 Observe produces the evidence ages, the generalisation gap and the findings that most measures start from. P4 Improve produces the changes whose cycle time and debt the loop must show. P5 Verify decides what may be counted as closed, and it is the stage whose records make the best measures because a re-test cannot be argued with. The chapter defines six measures of the loop, names the measures that mislead, applies Goodhart's law to a loop with AI on both sides, and fixes the rule that keeps all of it on the right side of employee representatives' rights: the loop is measured, and the people in it are not.

Measures from P3, P4 and P5 feed a board page of four to six measures with age, with gaming checks and the rule on people attached and the misleading measures kept off the page.

Figure 18.1: Loop measures by stage, the gaming checks and the rule on people attached to the board page, with misleading measures kept off it.

The chapter builds on conclusions that earlier volumes recorded. Architecture as Code (Chapter 15A) concluded that evidence is machine-collected and that attestation, a person asserting, differs from evidence, the system's own record (87, Nordqvist, 2025). AI Red Team, Chapter 26, fixed the closure rule that every measure here depends on: a finding closes when the re-test passes and the evidence is attached (90, Nordqvist, 2026). AI Threat, Chapter 30, classified controls as verified, declared or failed (91, Nordqvist, 2026). AI Blue, Chapter 20, replaced the speed measures of the old SOC with four families, speed, coverage, quality of judgement and learning, each with a gaming check, and set the rule that measures about people are agreed first, reported in aggregate and never per named analyst (94, Nordqvist, 2026). AI Boss, Chapters 13 and 23, supplied the law and practice behind that rule (93, Nordqvist, 2026). Chapter 11 of this book defined the closure rule and the evidence expiry, and Chapter 15 defined the generalisation gap and seeded gaps. This chapter does not repeat them. It decides which of their numbers go on a page, in what form and for whom.

What to measure

A measure of the loop is a number computed from the loop's records. That sounds obvious and has two consequences. First, a measure that cannot be traced to records (a finding, a regression case, an evidence manifest, a coverage cell) is an opinion, and the page must say so. Second, because the records are about techniques, scenarios, detections and findings, the natural grain of every measure is the technique, the class or the team. Nothing in the loop's records needs a person's name to be measured.

NIST's measurement guide for information security gives the general frame. Its first volume describes how an organisation develops, selects and prioritises measures that show whether its policies, procedures and controls are working, and its second volume describes a measurement programme around them (46, Schroeder et al., 2024) (47, Schroeder et al., 2024). Neither volume is specific to a purple loop. The inference to the loop is the book's own: the question that NIS 2 Article 21(2)(f) puts to essential and important entities, whether they assess the effectiveness of their cybersecurity risk-management measures (3, European Union, 2022), is answered by measures that come from tests and not from activity, and the loop is where those tests are run.

Six measures meet that standard. Each has a stage, a definition, a denominator and a reading that cannot be improved by running more scenarios.

Measure Stage Definition Why running more scenarios does not improve it
Finding-to-control cycle time P4 Improve Days from a finding opened to a control, detection or playbook change in production; median and 90th percentile, individual values when fewer than ten Counts only findings; a scenario that passes produces none
Finding half-life P5 Verify Days after which half of the findings opened in a quarter have a passing re-test Counts only verified closure, so a merge without a re-test does not shorten it
Re-test pass rate P5 Verify Share of re-tests passed at first attempt, with held-out variants included, reported with its count Easy scenarios are not in the denominator; held-out variants are
Verified share within expiry P5 Verify Priority techniques with a passing result younger than their expiry, over the priority list, with reasons for the rest The list and the expiry are fixed and any change to them is reported
Yield per exercise P3 Observe Findings per exercise that no catalogued run could have produced, and how many reach verified control within 90 days A scenario already in the catalogue cannot yield a novel finding
Detection debt P4 Improve Entries in the debt register of AI Blue, Chapter 11 (no owner, no passing test, stale data source, overdue review), with the age of the oldest Debt rises with every detection added without its test

Three notes on these definitions. The finding-to-control cycle time is the measure AI Blue recorded for Hansa Mutual, a median of 19 days in BTP-HM-2027-Q3 (AIDR-0778) (94, Nordqvist, 2026). It is useful and easy to improve by merging faster, which is why it never appears alone. The finding half-life, introduced in Chapter 10 as the book's own measure, counts only verified closure; the gap between the cycle time and the half-life is the stock of changes that are implemented and not verified, and the register reports that stock as open. The re-test pass rate is the least intuitive of the six, because a high rate looks like success. A first-attempt pass rate near 100 per cent over a stable library says that the fixes fit the scenarios, which is the overfitting of Chapter 8. The measure is informative only when the denominator includes variants and a held-out variant written by someone other than the author of the fix (Chapter 11), and when its count is shown: 11 passes from 16 re-tests and 110 from 160 are different statements.

Verified share within expiry is the loop's headline measure and the one most easily misread. Chapter 11 set the rule that the report states the denominator, the reasons and the age. "Twenty-seven of 34 priority techniques have a passing result younger than 90 days" names what was tested and when, and it is the form that carries the rule of this book that coverage is a claim with an age. A percentage without the list behind it does not.

Yield per exercise needs the most care, because it is the only one of the six that measures what the people on the red side add. Chapter 8 proposed to count findings that no catalogued scenario could have produced, per 100 hours of human-led emulation, and to report them beside the pipeline's regression results, since the two answer different questions. Yield counts at raising and again at 90 days: a novel finding that never reaches a verified control has produced a document and not an improvement. Yield is reported for the exercise and the team and never for the operator, for reasons given below.

Detection debt is borrowed from the blue team's register and applied to the loop's own output. A pipeline that produces detections faster than the team can test them creates debt by design, and AI Blue concluded that the rate of new detections is bounded by the rate at which they can be tested (94, Nordqvist, 2026). The loop measures the debt for the detections it has caused: those drafted from a finding and merged without a regression case.

What to avoid

Four measures are common on dashboards of continuous emulation and breach-and-attack simulation, and each rewards activity. They are not wrong as operational data. They are wrong as targets and as board figures.

Measure What it rewards How it is gamed, with or without AI What to report instead
Scenarios run Volume of emulation Run the cheap S1 scenarios on a schedule; split one scenario into several; re-run a passing scenario Verified share within expiry, with the share of the priority list that has only scenarios above S1
Findings opened Output of the red side Report many low-value findings; split one root cause into several records Yield: novel findings that reach verified control
Findings closed Output of the blue side Close on merge; close by acceptance; reclassify a repeat as new Finding half-life and recurrence
Green percentage A pass rate over the library Tune detections to the scenarios; keep the library stable; omit the techniques that cannot run in production Pass rate on held-out variants, beside the pass rate on known forms, each with its count

The counts that these dashboards show still have uses. Scenarios run is a measure of cost and of load on the SOC, and it belongs on the capacity plan. Findings opened and closed describe the register's throughput. The rule is that an activity count is never the numerator or the denominator of a figure that the board reads as a statement about security. AI Blue, Chapter 11, drew the same line for the blue team: the number of findings opened or closed is not a measure of learning (94, Nordqvist, 2026).

Goodhart and the loop

The general failure has a name. The formulation usually quoted is that when a measure becomes a target, it ceases to be a good measure, a restatement of Goodhart's law that appears in Strathern's 1997 paper on audit in universities, where she follows Hoskin (80, Strathern, 1997). This is the book's one explanation of the effect; Chapters 8, 17, 25 and 26 refer to it.

Manheim and Garrabrant distinguish at least four mechanisms by which optimising a measure undermines it: regressional (selecting hard on a proxy also selects for its noise), extremal (the proxy and the goal part company in the extreme cases that optimisation reaches), causal (acting on a proxy that is correlated with the goal but does not cause it) and adversarial (other agents respond to the measure and produce the effect) (82, Manheim et al., 2018). The taxonomy is a conceptual one. It does not report a study of security loops, and the mapping below is the book's own reading of it.

  • Regressional. Choosing, for the board page, the techniques whose results look best selects partly for luck. A technique that passed once, on a day when the sample was small, is not the same as a technique that is detected. The response is the evidence age and the count beside every figure, and the rule that the denominator is the whole priority list.
  • Extremal. A green percentage pushed to 100 per cent on a stable library describes a regime that the real adversary never visits. Chapter 8 called this dashboard comfort. The response is the generalisation gap, the difference between the pass rate on known-form scenarios and on held-out variants, which at Hansa Mutual in the first quarter of 2028 was 90 per cent on 41 known-form scenarios (37 detected) against 71 per cent on 14 held-out variants (10 detected), a gap of 19 points.
  • Causal. Findings closed correlate with security work and do not cause security. A team can close findings at will by accepting them or by closing on merge. The response is the closure rule of Chapter 11, which a person signs, and a risk acceptance that carries a named authority, a reason and an expiry (Chapter 4).
  • Adversarial. The agents that respond to a measure include people and, in this book, software. A detection drafter that is rewarded for passing scenarios will produce detections that pass scenarios, and a drafter that is a model does so at machine speed and without malice. The independence principle of Chapter 15 is the structural answer: the side that is measured does not control the instrument, the held-out set is hidden from it, and a person who was not in the chain reproduces a sample of results.

AI sharpens the adversarial case in two ways, both of which are the book's own conclusions. A Mode 4 or Mode 5 agent that optimises a pass rate has no reason to prefer the behaviour the organisation wants over the behaviour the number rewards, so a measure used as an agent's objective must be one for which the wanted and the rewarded behaviour are the same. And AI makes the cheap activity cheaper: a curator that drafts scenarios and a runner that executes them can double the scenarios run in a month at the same cost, and a dashboard that counts scenarios run would call that progress.

Four properties make a measure safe to use as a target, and the six measures above were chosen for them. The behaviour that improves the measure is the behaviour the organisation wants (verified closure). The measure carries a denominator that the one who is measured cannot change without the change being reported (the priority list, the expiry). A part of the instrument is hidden from the one who is measured (the held-out set, the seeded gap). And a person outside the chain can reproduce a sample of the results. A measure that lacks all four stays on the operational dashboard.

Leading and lagging measures

Some measures report what has happened; others predict what will. A board needs both, and a loop that reports only lagging measures learns of a problem a quarter late.

Lagging measures report outcomes. The finding half-life, the recurrence rate (the share of findings in a class that return within 12 months), the verified share and the yield at 90 days are lagging: each is known only after the work it describes. The most important lagging measure comes from outside the loop. After a real incident, the post-incident review (PIR-*) asks whether the technique used was on the priority list, whether it had a verified detection at the time and how old the evidence was. A loop whose priority list repeatedly misses the techniques of real intrusions is measuring the wrong thing, whatever its pass rate. That is the book's own conclusion, and the count of incidents will be small enough in most organisations that it is read as a signal and reported with the count.

Leading measures predict. The evidence-age distribution predicts the verified share: the number of priority techniques whose evidence expires in the next 30 days is the share that will fall out of the verified column unless someone acts. The generalisation gap predicts the first-attempt re-test pass rate on the next real change. The age of the oldest entry in the detection-debt register predicts a declared control. The seeded-gap catch rate predicts whether the loop's observation would find a real one. The open drafts per reviewer predict the review quality that Chapter 10 caps at six at Hansa Mutual.

The distinction decides what is shown to whom. The loop's operators read the leading measures weekly. The board reads a small selection each quarter, with at most three decisions, each traced to a measure (the rule AI Blue set for the blue team's page and AI Threat, Chapter 32, set for the threat register) (94, Nordqvist, 2026) (91, Nordqvist, 2026). A page that asks for no decision informs but does not govern.

AI-specific measures

A loop with AI in it needs measures of the AI's part that do not depend on the AI's own report. Four are enough, and each is a measure of the system and not of a person.

The independence measures. The catch rate of seeded gaps, the pass rate on held-out variants and the age of the independence statement together say whether the loop's checks can fail. Chapter 4 set the freeze rule, that the loop profile freezes if the seeded-gap catch rate falls below 0.80 (DLG-HM-LOOP-01), and Chapter 15 set the seeded gaps and the rule that the independence statement for a priority agent expire after 90 days. On the page they appear with their counts. Six seeded gaps with five found is 0.83, and the page says that six is too few to read as more than a signal.

Agent work and person verification, separately. A measure that includes AI-run work separates what the agent did from what a person verified. A figure such as "re-tests passed" is split into those run by the re-test runner (MND-HM-EMU-06) and the share of them reproduced by a person under the rule of Chapter 15. The first is activity at machine speed; the second is the part that is evidence.

Corrections to the agents' proposals. The observation correlator (MND-HM-EMU-03) proposes a classification for each step of a session. The measure is the number of proposals, the number a person re-did and the number corrected, with the seeded wrong joins and how many were caught. In PRP-HM-2028-01 the correlator proposed classifications for 96 tagged steps, people re-did 24 and corrected 3, and four wrong joins seeded by the white team were all caught. The measure belongs to the mandate and the review process. It is not a measure of reviewers, and it is reported for the team.

Drift. Detections, connectors, mandates and models change, and a result obtained before the change is not evidence about the system after it. The drift measure is the count of changes to dependencies of verified techniques in the period (a detection version, a connector, a model behind a triage agent, a mandate) and the share of them followed by a re-test within the stated time. It is the measure that tells the board whether the expiry condition, and not only the expiry clock, is working. At Nordvik Region the provider's 20 working days' notice of a model change under annex MS-2 is the input; a model change without a re-test within the annex's window is a finding against the provider's process.

Aggregation and works-council constraints

The loop produces records that could be turned into a ranking of people overnight. Who raised the most findings, whose detections were merged and later failed a re-test, which analyst's closures were reopened, which tester's scenarios yielded little: every one of these can be computed from the registers, and every one is a measure of a worker. Chapter 9 recorded the rule that the sessions apply, that no analyst is named in an outcome table and no outcome enters an appraisal. This chapter generalises it to every measure.

AI Boss gave the grounds. Monitoring and scoring of workers falls within co-determination and data-protection law (F86) (93, Nordqvist, 2026). In Germany, § 87(1) no. 6 of the Works Constitution Act gives the works council co-determination over technical devices designed to monitor employees' behaviour or performance (11, Bundesrepublik Deutschland, n.d.), and the Federal Labour Court holds that a device is designed for that purpose when it is objectively suitable to collect and record such information, whatever the employer intends (16, Bundesarbeitsgericht, 2022) (121, Bundesarbeitsgericht, 2024). In Sweden the union's right to information and the employer's duty to negotiate before an important change in activity apply (15, Sveriges riksdag, 1976), and the Region's agreements follow that pattern (AI Boss, Chapter 23). Under the GDPR the data about identified employees is personal data, data minimisation under Article 5(1)(c) requires it to be adequate, relevant and limited to what is necessary for the purpose (7, European Union, 2016), and Article 88 allows Member States to provide by law or by collective agreement for more specific rules on the processing of employees' personal data in the employment context, with suitable and specific measures to safeguard dignity, legitimate interests and fundamental rights, with particular regard to transparency of processing and monitoring systems at the work place (7, European Union, 2016). These are organisational implications, not legal advice, and the position differs by Member State and by agreement. The quiet harms that AI Boss described, a cost of pace (Chapter 11) and a chilling effect on the willingness to speak (Chapter 13), bear on the purple loop with particular force, because the loop depends on people saying that a detection did not fire and that their own earlier change was wrong (93, Nordqvist, 2026).

Five rules follow, and they are written into the measure catalogue so that they can be inspected and changed only by the route the agreement sets.

  1. Grain. Every measure is computed per technique, scenario, class, service or team. No measure is computed per named person, and no field in the page or its source query identifies one.
  2. Minimum group size. A figure for a group so small that it identifies a person is an individual figure. Hansa Mutual's floor is five people, as agreed for the SOC; the red team of five is exactly at it, so its yield is reported for the team and never split by campaign lead. Nordvik Region's unit has six analysts, so its figures are reported for the unit and never by shift, week or reviewer.
  3. Roles held by one person. A role held by one person is that person. At Nordvik Region the detection owner is Karin Lund, so a figure reported "by owner" is a figure about her. The Region reports by service and technique, and the owner field names the role in the record and never appears on the page.
  4. Agreement before collection. The agreement precedes the data. A measure that already exists in a ticket field does not become a measure of performance because it is available. Adding a measure to the catalogue is a change of the kind that the works agreement governs by class and not by software version, the approach that AI Boss, Chapter 23, recommended for systems that change with every release: a new team-level measure from existing loop records is notified; a measure that would identify a person is outside the catalogue and not eligible for agreement.
  5. No collection from assistants for the loop. Individual-level data from Modes 1 to 3 assistants (prompts, accepted suggestions, query histories) is not collected for the loop's measures. A person's use of AI in a loop record is declared in the record, which Chapter 14 fixed, and the declaration is not a performance measure.

The contracted tester sits slightly outside this rule and under another. Elin Berglund and the firm are not the Region's employees, but the same logic applies. The call-off agreement NR-2026-071 measures the engagement: the share of findings that the Region's detection owner could reproduce and the share that were rejected as duplicates, reported for the engagement. The quality of an individual consultant's work is a matter between the firm and its employee.

The constraint carries a price, and it is better stated than hidden. Aggregate measures cannot tell a manager which person needs support, and a team of six cannot hide a bad quarter inside an average. The answer to the first is the one the earlier volumes gave: look at the class, the tooling and the load before looking at the person, and handle a person's difficulty through ordinary line management and not through a dashboard. The answer to the second is to report counts and to say, on the page, which figures are too small to read.

The board page

The board needs a view that it can decide from. AI Blue gave its page five parts and at most three decisions. The loop's page is shorter, because the loop supports the blue team's page and does not replace it: four to six measures, each with its count and its age, chosen from the six above, and a decisions block.

The page answers one question, the one NIS 2 Article 20 and DORA Article 5 place with the management body (3, European Union, 2022) (2, European Union, 2022): would the organisation have seen and stopped this, and how does anyone know? Its form is that of Chapter 11, a denominator, reasons and an age, extended with the generalisation gap and the cycle time. Every figure on it is generated from the registers and not typed, so that a reader can follow it to the finding, the regression case and the evidence manifest.

Practical guide: Designing a board page for the loop that cannot be improved by running more scenarios

Purpose. To produce a page of four to six measures of the loop that a team cannot improve by activity alone, that carries its own gaming checks and that stays on the right side of the works agreement.

Roles. CISO (owner of the page); purple-team coordinator (compiles and signs the facts); white-team lead (holds the held-out set and the seeded gaps); head of detection engineering; data protection officer; employee representatives (agree the catalogue); internal audit (samples).

Steps.

  1. Start from the six measures of this chapter and the decisions the board can take: fund work, extend or hold a delegation, accept a named gap until a date. Remove every measure that supports none of them.
  2. For each measure, write a catalogue entry (below): definition, source records, denominator, grain, gaming check and the figure's count. A measure without a gaming check does not enter the catalogue.
  3. Fix the denominator in code: the priority list, the expiry and the held-out set are referenced by identifier and version, and any change appears on the page in the period in which it occurs. Default: a change to the list or the expiry is shown as a line "list changed: techniques added or removed, expiry from X to Y".
  4. Attach the age. For each coverage figure show the age of the oldest passing result and the number of techniques that expire within 30 days. Default: expiry 90 days for priority techniques and 180 days for others at Hansa Mutual, and 180 days for all at Nordvik Region.
  5. Add the generalisation gap and the first-attempt re-test pass rate side by side with the count of each, so that a high pass rate cannot be read without its gap.
  6. Agree the catalogue with the employee representatives before the first collection. State the grain, the minimum group size and that no result enters an appraisal. At Hansa Mutual this extends KBV-IT-2024-SOC-05 to the page; at Nordvik Region it is the union's agreement that results are reported in aggregate and never by name.
  7. Generate the page from the registers by a pipeline and have the coordinator read it before it is sent. Default: a 10 per cent sample of re-test passes, and every high-severity closure, is reproduced by a person (Chapter 15) before the quarter's figures are signed.
  8. Limit the decisions block to three, each traced to a measure. Record the page and the decisions as an AI Decision Record.
  9. Test the page once a year by trying to improve it by activity: double the scenarios run in a test quarter and check that none of the six figures moves. If one does, remove or redefine it.

Definition of done. The page has four to six measures, each with count, age and gaming check; no figure is computed per person; the catalogue is agreed; the page is generated from the registers; at most three decisions are requested, each traced to a figure; and the step 9 test has been run once.

A catalogue entry is an artefact for the repository:

# reporting/measure-catalogue.yaml  (illustrative excerpt)
catalogue: MSC-LOOP-01
version: 1.0.0
agreed_with: [konzernbetriebsrat, polish-service-centre-representatives]
grain_allowed: [technique, scenario, class, service, team]
grain_forbidden: [person, role-held-by-one-person, shift]
min_group_size: 5
no_collection_from: [mode-1-to-3-assistants]
measures:
  - id: verified-share
    stage: P5
    numerator: priority techniques with a passing result younger than expiry
    denominator: priority list (id and version stated on the page)
    show: [count, reasons, oldest_passing_days, expiring_in_30_days]
    gaming_check: changes to list or expiry reported in the period
  - id: generalisation-gap
    stage: P3
    numerator: pass rate known-form minus pass rate held-out
    show: [count_known, count_held_out]
    gaming_check: held-out set hidden from drafter; rotated after each failure
    owner_of_instrument: white-team-lead
  - id: finding-half-life
    stage: P5
    cohort: findings opened in the quarter
    gaming_check: closure requires passing re-test; repeat linked, not reclassified

Worked example: LPG-HM-2028-Q1 and LPG-NR-2028-Q1

Hansa Mutual's page for the first quarter of 2028 compares itself with BTP-HM-2027-Q3, which reported 25 of 34 priority techniques verified, an oldest passing test of 74 days and a finding-to-control median of 19 days from 23 findings. The figures below are illustrative and new in this chapter, except those that Chapters 8, 9 and 15 established and that are marked.

# reporting/loop-page.yaml  (illustrative)
page: LPG-HM-2028-Q1
period_end: 2028-03-31
baseline: BTP-HM-2027-Q3
owner: group-ciso
reporting_level: team            # never by person; minimum group size 5
catalogue: MSC-LOOP-01
verified_share:
  priority_techniques: 34
  verified_within_expiry: 27     # baseline 25
  not_verified: {evidence_expired: 3, retest_failed: 2, never_tested: 2}
  oldest_passing_days: 81        # expiry 90
  expiring_in_30_days: 4
  list_changed: "AIDR-0851 re-ranking, five tier changes, 34 unchanged"
generalisation_gap:              # Chapter 8
  known_form: {run: 41, detected: 37}
  held_out: {run: 14, detected: 10}
  gap_points: 19
retest_first_attempt: {runs: 16, passed: 11, rate: 0.69}
learning:
  cohort_2027_q3: {findings: 23, median_days_to_control: 19, half_life_days: 46}
  cohort_2027_q4: {findings: 22, median_days_to_control: 18, half_life_days: 37}
debt: {entries: 23, past_threshold: 5, oldest_days: 118}
ai_work:
  seeded_gaps: {planted: 6, found: 5, rate: 0.83, freeze_below: 0.80}
  correlator: {proposals: 96, redone_by_person: 24, corrected: 3,
               seeded_wrong_joins: 4, caught: 4}   # PRP-HM-2028-01
  independence: {statement: IND-HM-2028-Q1, age_days: 38, valid_to: 2028-05-22}
  reproduced_by_person: {high_severity_closures: all, sample_rate: 0.10}
decisions_requested:
  - {measure: verified_share.not_verified.never_tested,
     request: fund red-team range runs for T1486 and T1490 in 2028-Q2}
  - {measure: generalisation_gap,
     request: extend the held-out set from 13 to 20 scenarios}
decision_record: AIDR-0868

The page reads as follows. Two more techniques are verified than at the baseline, but not by the margin the raw count suggests: the independence review of 22 February moved two cells to failed, so the gain would otherwise have been four. The reasons are given, and two of the seven are techniques that cannot run in production in a reversible form, which is a design problem and not a backlog. The cycle time has barely moved while the half-life has shortened from 46 to 37 days, which means that the gain came from verifying faster and not from merging faster; the gap between the two, the stock of changes implemented and not verified, fell from 27 days to 19. The generalisation gap of 19 points is the figure that the board question of January (AIDR-0850) most needs, and the page does not interpret it. The held-out set stood at 13 of 86 scenarios at the start of the year (15.1 per cent) and at 13 of 87 after SCN-HM-0087 (14.9 per cent, one short of the 15 per cent rule of Chapter 15); the requested extension to 20 restores it. Whether the continuous pipeline makes the quarterly session redundant is answered in the report due on 2028-04-30, to which Chapters 26 and 27 return, and the page supplies its evidence. The decision record AIDR-0867 (2028-03-13) replaced the scenarios-run tile and the single pass-rate tile on the board's page with the measures of the catalogue and fixed the reporting level; AIDR-0868 (2028-04-04) issues the first page and records the two requests.

Nordvik Region's page has four measures, because four are what a team of six and a contract can produce from records the Region holds.

# reporting/loop-page.yaml  (illustrative)
page: LPG-NR-2028-Q1
period_end: 2028-03-31
owner: ciso
reporting_level: unit            # six analysts; never by shift, week or reviewer
agreed_with: [union-representatives]
verified_share:
  priority_list: PRI-NR-2028-01  # nine techniques
  verified_within_expiry: 2      # COV-NR-0001 and -0002, both 2028-02-09 (Chapter 17)
  tested_and_failed: 3           # COV-NR-0003 (2028-02-20), -0004 and -0005 (2028-03-22)
  asserted_by_provider: 2        # COV-NR-0006 and -0007
  not_tested: 2                  # T1190 and T1486, past their expiry
  expiry_days: 180
cycle_time:                      # n = 2, so individual values, no median
  FND-NR-2027-011: {opened: 2027-10-20, first_closed: 2027-11-17,
                    reopened_as: FND-NR-2028-002, verified: 2028-02-09, days: 112}
  FND-NR-2028-002: {opened: 2028-01-19, verified: 2028-02-09, days: 21}
provider_closure:                # MS-2 round 2028-Q1, AIDR-0813
  seeded: 15
  handled_correctly: 14
  escape: bulk-mail seed in class MS2-C1
evidence_age:
  oldest_passing_days: 51        # 2028-02-09 at period end, expires 2028-08-07; 77 on 2028-04-26
  open_finding: FND-NR-2028-003  # T1053.005, DET-NR-0012, injected test due 2028-05-02
decisions_requested:
  - {measure: evidence_age,
     request: turn the two asserted techniques into injected tests under MS-2 and re-run T1190 and T1486 in the mirror}
decision_records: [AIDR-0817, AIDR-0818]

The cycle time of FND-NR-2027-011 is the lesson of this page. The first closure on 17 November 2027 was 28 days after the finding and would have been the headline. The variant re-test of 19 January 2028 showed that the base case had passed and the hours-of-day variant had not, the finding was reopened as FND-NR-2028-002, and the closure that the register accepts is 9 February 2028, 112 days after the session. A page that counted the first closure would have understated the cycle time by 84 days. The page shows both findings as individual values because two is too few for a median. AIDR-0817 (2028-03-06) adopted the four-measure page and the grain agreed with the union; AIDR-0818 (2028-04-12) issued the first page.

Across the usage modes and degrees of delegation

The mode describes how the AI is deployed and whose identity it uses; the level describes who decides each action. The page and its measures sit under the same ceilings as the rest of the loop, and Level 4 Autonomous is never used for any step that would score or rank a named person.

Activity in measuring the loop Usual mode Default ceiling Condition
Draft commentary on a page Mode 1 to Mode 3 under the coordinator's identity Level 1 Assisted A person reads every figure against the register
Compile the figures from the registers Mode 4 report compiler Level 3 Bounded Read-only; generated from records; reversible; definitions fixed by the catalogue
Propose a new measure or change a definition Mode 3 or Mode 4 Level 0 Advisory Agreement with representatives before the measure exists
Flag evidence about to expire Pipeline Level 3 Bounded Rule-based; reported
Compute a figure per named person None Not built Outside the catalogue and not eligible for agreement
Collect prompts or query histories of assistants None Not built Modes 1 to 3 individual-level data is not collected for the loop
Send the page to the board Person Not delegated The CISO signs; the board's decisions are recorded

A Mode 5 emulation runner or re-test runner generates events that become the measures, and its activity is reported as activity; only a person-reproduced share is reported as verified.

Nordvik Region and Hansa Mutual

Nordvik Region buys the loop, and its leverage is the contract. Its page has four measures because those are the ones that can be computed from records it holds or can obtain by annex: verified share, cycle time, provider closure rate and evidence age. The provider's own figures enter the page only after the Region has traced them to a record (annex MS-2), and a figure it cannot trace is declared, not verified. The grain is the unit of six; the owner role of Karin Lund never appears on the page; the union has agreed that results are aggregate and never by name. The risk is a loop that stops after the first report, and the evidence-age figure, with two techniques already past their expiry listed beside it, is the mechanism that makes the absence of a re-test visible.

Hansa Mutual runs the loop, and its leverage is the pipeline. It has more data than a board can read, so the page selects: six measures, the generalisation gap beside the pass rate, the half-life beside the cycle time and the AI-specific counts. Its risk is a dashboard that nobody questions, and it answers with the step 9 test, a hidden held-out set owned by the white team and a minimum group size of five that its red team of five meets exactly. Under KBV-IT-2024-SOC-05 no exercise result enters an appraisal, a rule that now covers red-team and detection-engineering staff as it already covered the analysts. The two organisations differ in what they can measure and not in what they should refuse.

Observable, expected, speculation

Observable. Goodhart effects are described in the literature as a general pattern (80, Strathern, 1997) (82, Manheim et al., 2018), published guidance on information-security measurement asks organisations to select measures that show whether controls work (46, Schroeder et al., 2024), and the activity counts of emulation platforms are available on every dashboard. No source examined for this book measured Goodhart effects in a purple loop, and the mapping above is the book's own.

Expected. Measuring yield and verified share rather than activity is expected to become the norm for threat-led testing, because the instruments already require closure and remediation (DORA Article 24(5) and the delegated regulation on threat-led testing) (2, European Union, 2022) (8, European Commission, 2025).

Speculation. Fully automated board reporting without a person reading the figures against the registers is plausible and unsupported by present evidence, and the page of this chapter is built so that it need not be believed.

Summary

This chapter treated the measures of the loop, concerning P3 Observe, P4 Improve and P5 Verify. Six measures describe the loop and none can be improved by running more scenarios: finding-to-control cycle time, finding half-life, first-attempt re-test pass rate with held-out variants, verified share within expiry, yield per exercise and detection debt. Four measures mislead when they become targets: scenarios run, findings opened, findings closed and a green percentage without a denominator. Goodhart's law applies to the loop in four ways, regressional, extremal, causal and adversarial, and AI makes the adversarial form faster, so a measure is safe as a target only if the improving behaviour is the wanted one, the denominator is fixed and reported, part of the instrument is hidden and a person can reproduce a sample.

Four AI-specific measures carry the independence principle onto the page: the seeded-gap catch rate, agent work reported apart from person-verified work, corrections to the agents' proposals and drift. All measures are of the system. They are computed per technique, class, service or team, never per person; a role held by one person is that person; the minimum group size is five at Hansa Mutual and the unit as a whole at Nordvik Region; the catalogue is agreed with representatives before collection; and individual-level data from Modes 1 to 3 assistants is not collected for the loop. The board page has four to six measures with count and age, at most three decisions, and a yearly test that doubles the scenarios run and checks that nothing moves.

Hansa Mutual's LPG-HM-2028-Q1 shows 27 of 34 priority techniques verified within expiry against 25 at BTP-HM-2027-Q3 (the series ran 25 at the baseline, 23 on 2028-02-22 when the independence re-test of Chapter 15 moved two cells to failed, and 27 at 2028-03-31 after four techniques were re-verified; on 2028-02-22 the 11 not verified were 4 expired, 5 failed and 2 never tested), a half-life shortened from 46 to 37 days and a generalisation gap of 19 points. Nordvik Region's LPG-NR-2028-Q1 shows 2 of 9 verified within expiry and 3 more tested and failed, a cycle time of 112 and 21 days for two findings, 14 of 15 seeded cases handled correctly by the provider and an oldest passing result 51 days old. The reader can begin by removing one activity count from a board page and replacing it with a verified share that carries its list, its reasons and its age.

Chapter 19 turns to who runs the loop: the operating models, the roles and the procurement of testing and simulation.

Sources

  1. European Parliament and Council of the European Union. Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA). (2, European Union, 2022)
  2. European Parliament and Council of the European Union. Directive (EU) 2022/2555 on measures for a high common level of cybersecurity across the Union (NIS 2 Directive). (3, European Union, 2022)
  3. European Parliament and Council of the European Union. Regulation (EU) 2016/679 (General Data Protection Regulation). (7, European Union, 2016)
  4. Nordqvist, G. (2025). Architecture as Code. (87, Nordqvist, 2025)
  5. Nordqvist, G. (2026). AI Red Team. (90, Nordqvist, 2026)
  6. Nordqvist, G. (2026). AI Threat. (91, Nordqvist, 2026)
  7. Nordqvist, G. (2026). AI Boss. (93, Nordqvist, 2026)
  8. Nordqvist, G. (2026). AI Blue. (94, Nordqvist, 2026)
  9. Strathern, M. (1997). 'Improving ratings': audit in the British University system. European Review 5(3). (80, Strathern, 1997)
  10. European Commission (2025). Commission Delegated Regulation (EU) 2025/1190 (regulatory technical standards on threat-led penetration testing under DORA). (8, European Commission, 2025)
  11. Schroeder, K., Trinh, H. & Pillitteri, V. (2024). Measurement Guide for Information Security: Volume 1. NIST SP 800-55 Vol. 1. (46, Schroeder et al., 2024)
  12. Schroeder, K., Trinh, H. & Pillitteri, V. (2024). Measurement Guide for Information Security: Volume 2. NIST SP 800-55 Vol. 2. (47, Schroeder et al., 2024)
  13. Manheim, D. & Garrabrant, S. (2018). Categorizing Variants of Goodhart's Law. arXiv:1803.04585. (82, Manheim et al., 2018)
  14. Bundesrepublik Deutschland. Betriebsverfassungsgesetz, § 87. (11, Bundesrepublik Deutschland, n.d.)
  15. Sveriges riksdag. Lag (1976:580) om medbestämmande i arbetslivet. (15, Sveriges riksdag, 1976)
  16. Bundesarbeitsgericht. Beschluss vom 8. März 2022, 1 ABR 20/21. (16, Bundesarbeitsgericht, 2022)
  17. Bundesarbeitsgericht (2024). Beschluss vom 16. Juli 2024, 1 ABR 16/23. Bundesarbeitsgericht. (121, Bundesarbeitsgericht, 2024)