Verify: Re-test, Regression and Evidence

A finding is a claim about the past: on a given day, a technique reached its objective, or a detection did not fire, or an alert was closed by a system that should not have closed it. A change is a claim about the future: this will not happen again. Between the two claims sits P5 Verify, the stage that decides whether the second claim may be believed. It is where the loop either closes or merely appears to. The chapter defines closure, designs the re-test, explains why re-running the exact scenario proves less than it seems, shows how a re-test becomes a regression case that runs for as long as the control exists, and treats the evidence as code that is signed, versioned and, above all, dated. It concerns P5 Verify and takes its input from P4 Improve (Chapter 10) and its tests from P2 Emulate (Chapters 7 and 8); the comparison of result with expectation is P3 Observe (Chapter 9) run a second time.

A finding passes the exact re-test, then the variants and the negative control, becomes a regression case with a signed evidence manifest, and is closed by a named person; the evidence then expires and the case runs again. Until the re-test passes the finding is reported as open and unverified.

Figure 11.1: A finding passes the exact re-test, the variants and the negative control, becomes a signed regression case and is closed by a named person; the evidence then expires.

The chapter builds on conclusions that earlier volumes recorded. Architecture as Code (Chapter 15A) concluded that evidence is machine-collected, signed and versioned rather than assembled in audit season (87, Nordqvist, 2025). AI Governance, Chapter 11, added that attestation is a person asserting and evidence is generated by the system (88, Nordqvist, 2026). AI Red Team, Chapter 26, defined the purple loop as five record states, finding, control, detection, re-test and evidence, and fixed the exit condition that this chapter develops: a finding is closed only when the re-test passes and the evidence is attached (90, Nordqvist, 2026). AI Blue, Chapter 11, set out the blue team's half of the loop and added that a finding is closed with a pointer to the passing re-test and that the date of the last passing re-test is part of the record (94, Nordqvist, 2026). Those chapters state the rule. This one asks what it takes to apply the rule honestly when a machine can run the re-test every night and another machine can write the fix.

The closure rule

The rule has four clauses, and each answers a way that closure goes wrong.

  1. A finding is closed only when the re-test and its variant pass. Not when the change is merged, not when the owner says it is deployed, not when the ticket is moved. A deployed change that nobody has tested is an implemented, unverified control, and the register reports the finding as open.
  2. The evidence is attached. The re-test leaves artefacts a person can inspect: the scenario that ran, the telemetry it produced, the detection output, the timing, the version of everything involved. A pass without artefacts is an assertion.
  3. A named person signs. Closure is a decision about the organisation's risk, and the signature records who took it. A runner at Level 3 Bounded may execute the re-test and assemble the evidence; it never closes.
  4. Closure has an age. The signature carries a date and a condition, and when the condition lapses the finding is not closed any more, it is unverified again (the section on expiry).

The register shows a priority technique in one of four states: verified (a re-test has passed within the expiry and the evidence is attached), implemented, unverified (a change exists and no passing re-test does), failed (the latest re-test did not pass) and not tested (untested, or the evidence has expired). These are the states of a coverage cell, a projection of the five finding states of Chapter 4 and of the coverage statuses of Chapter 17, and not a fourth taxonomy: verified is closed and verified within its expiry, and failed is re-test failed. The states of a finding are those of Chapter 4. They extend the classification of AI Threat, Chapter 30, to a technique: a control counts as verified only when a test has passed recently, and otherwise it is declared or failed (91, Nordqvist, 2026). "Not tested" is a legitimate entry and a far better one than a green mark with no date (Chapter 17 develops the coverage record that holds it).

The rule is also what regulators already ask of the most intensive testing. TIBER-EU ends a test in a closure phase in which the stakeholders reflect on the outcome, the red and blue teams replay the test, and the test concludes with remediation planning (20, ECB, 2025). The delegated regulation that completes DORA's rules on threat-led penetration testing has its own stages for results, closure and remediation, including a remediation plan (8, European Commission, 2025). For a designated financial entity such as Hansa Mutual's German entity, the architectural implication is that the evidence of remediation must be producible on request from the programme's own records. For a public authority subject to NIS 2, which requires policies and procedures to assess the effectiveness of cybersecurity risk-management measures (Article 21) (3, European Union, 2022), the re-test and its evidence are the assessment. Neither instrument prescribes a regression suite; the suite is the cheapest way to have the evidence when it is asked for. This is architectural reasoning, not legal advice.

Designing the re-test

A re-test is an experiment with a hypothesis, and it is designed before the change is merged, not after. The hypothesis is the expected outcome of the original finding, restated for the changed estate: the scenario that reached its objective unseen is now prevented, or detected within a stated time, or both. The same discipline that chaos engineering applies to steady-state hypotheses holds here: state what the system should do, run the condition, and compare (78, Principles of Chaos Engineering community, n.d.) (77, Basiri et al., 2016).

Five decisions go into the design.

What outcome counts as a pass. Chapter 9 defined seven outcomes for a step: prevented, detected, alerted but not triaged, triaged but not acted on, missed, no telemetry and not tested. A re-test states which outcome passes and in what time. For a technique where prevention is realistic, prevented passes. For one where detection is the control, detected and alerted is not enough: the pass criterion names the layer (triaged by a person, or acted on by a playbook), and the time bound. The Nordvik finding in the worked example below fails if the alert exists but the provider's AI closes it; a rule that fires is not a defence that works.

Who runs it. The test should not be run by the party that wrote the fix, and where an agent drafted the fix, the agent that drafted it does not run the re-test. This is the independence principle at its smallest scale (Chapter 15): the author of a change is the party least able to see what it missed. Nordvik gets this by role, since the contracted tester is not the detection owner. Hansa has to design it between agents and teams that share a repository, and does so by mandate: the emulation runner that executes the re-test has a different mandate, a different identity and no write access to detection content, and the fix is drafted elsewhere.

Where it runs. Re-tests run against the same environment class as the original finding. A pass in a pre-production mirror proves the mirror. If the original finding was in production, the re-test is a catalogued, reversible safety-class S1 scenario in production, under the signed rules of engagement (Chapter 6); if it cannot safely run in production, the evidence says so, and the entry in the register states the environment that was tested.

What can be undone, and who can stop it. Because a re-test is emulation, the P2 rules apply to it. Every scenario in a regression suite carries its safety class, is tagged so that its telemetry is recognisable, leaves nothing behind that cannot be removed by a named cleanup step, and can be stopped by a holder outside the system being stopped: the white-team lead and the SOC shift lead at Hansa Mutual, the managed provider's shift lead at Nordvik Region for injected tests that arrive through its intake. A re-test that has no stop path is not added to the suite, however attractive its coverage.

A negative control. A detection can pass its re-test by alerting on everything. Each case therefore includes at least one benign look-alike that must not alert, because the change that closes a finding must not open a new false-positive problem (Chapter 10 treated the cost). The negative control is part of the pass criterion, not an optional extra.

Variants

Re-running the exact scenario is necessary and insufficient. It shows that the specific path that was found is now seen. It does not show that the technique is seen, and the difference is the one that matters to the organisation: an adversary does not repeat the tester's path.

A detection written from a finding tends to fit that finding. The author, human or model, reads the telemetry the scenario produced and writes a rule that matches it. The exact re-test then passes, trivially, because the rule was shaped by the data in the test. That is overfitting in the plain sense, and it is the failure that makes a green re-test the most reassuring and the least informative signal in the loop. The principle that real-world events should be varied is part of the published discipline of chaos engineering for the same reason (78, Principles of Chaos Engineering community, n.d.).

A variant changes one thing about the scenario and keeps the technique. The book uses three kinds, each defined at the level of the technique and the telemetry, never of the tooling:

  • Path variants change how the technique is carried out while the ATT&CK technique stays the same (17, MITRE, n.d.): another tool class, another API route, another account type.
  • Context variants change where it happens: a server instead of a workstation, a different business unit, a different network zone, out of hours instead of in office hours, a different jurisdiction of the source.
  • Rate and shape variants change the volume and timing: slower, spread over more sources, interleaved with benign activity.

A held-out variant is a fourth kind, defined by who wrote it, not by what it changes. It is authored by someone who did not write the fix and is not visible to the author of the fix before the run: the contracted tester at Nordvik, the red team at Hansa Mutual. It is the nearest thing the loop has to an unseen test set. Chapter 15 develops held-out scenario sets and seeded gaps as part of the independence principle; this chapter only asks that the regression case has a place for one. AI can generate variants cheaply, and that is a gain: it widens the neighbourhood of a finding at the cost of compute. It also shifts the risk. If the same model family proposes the variants that wrote the detection, the variants will tend to sit where the detection already looks. For that reason, AI-proposed variants are labelled as such in the case, a person reviews them before they enter the suite (Level 1 Assisted for the planner), and a share of variants in every case is human-authored.

How many variants is enough has no universal answer. The defaults in the practical guide below are a floor for priority techniques, to be raised for techniques where the finding was severe and lowered, with a recorded reason, for techniques where the environment offers only one path.

Regression suites in the pipeline

A regression case is a re-test that has been kept. The exact scenario, its variants and the negative control become a REG-* record in the repository, tied to the finding it came from, the detection or control it protects and the technique it concerns. From then on the case runs for as long as the control exists. The practice is the one that AI Platform and Security applied to AI red-teaming findings and AI Red Team, Chapter 26, extended to every finding: each re-tested finding becomes a case that runs continuously, so that the fix cannot silently erode (89, Nordqvist, 2026) (90, Nordqvist, 2026).

Three triggers run a case.

  • On change. A change to the detection, the connector or log source that feeds it, the parser, the playbook, the agent mandate or the provider's configuration runs the cases that depend on it. This is the case that catches the commonest cause of regression, which is not the adversary but the estate: a field renamed, a log source silent, a threshold tuned for another reason.
  • On schedule. A cadence independent of change, because some regressions come from changes nobody connected to the control. Hansa Mutual exercises each priority technique at least every 30 days; Nordvik Region's cadence is the quarterly injected test of annex MS-2.
  • On intelligence. A new variant of the technique reported by the threat-intelligence cell or the sector group adds a case and runs the suite.

The suite is a ratchet: a case once added is not removed because it is inconvenient. A case may be retired only by a recorded decision when its control is retired, and a failing case blocks the change that broke it or reopens the finding, whichever applies. Without a ratchet, regression suites decay into the same state as the detections they test, and the loop of Chapter 8 becomes a set of green runs that guard nothing.

Two cautions apply when machines run the pipeline. First, a green run is itself an automation output that people learn to trust. Research on human use of automation describes complacency, the reduced monitoring of an automated function that is reliable most of the time (101, Parasuraman & Manzey, 2010), and a nightly suite that is green for months invites exactly that. The remedy is design rather than exhortation: seeded failures (Chapter 15), a periodic human replay of a case, and a report that shows the age of the evidence instead of the colour of the last run. Second, the suite runs emulation, so it is an offensive capability inside the estate. It is a candidate for the same protection as the scenario library (Chapter 16), and the credentials its runner uses are the narrowest the scenarios need.

Practical guide: Turning a closed finding into a regression case with variants and a cadence

Purpose. Convert one finding that has passed its exact re-test into a REG-* case that protects the technique rather than the one path, with evidence that expires and is refreshed.

Roles. The finding owner (detection owner or control owner), the red-team lead or contracted tester (author of the held-out variant and the signer of the re-test), the detection engineer who wrote the fix, the white-team lead or the CISO's office (rules of engagement and stop path), and the person who signs closure.

Steps.

  1. Restate the hypothesis from the finding: the technique, the environment and the pass criterion with its time bound and layer (prevented, detected, triaged by a person or acted on by a playbook).
  2. Record the exact scenario (SCN-*) as the base case, and run it against the changed estate. Do not proceed if it fails; the finding returns to P4 Improve.
  3. Write at least three variants for a priority technique and one for any other: one path, one context, one rate or shape. A person reviews any AI-proposed variant before it enters the case.
  4. Ask someone who did not write the fix to author one held-out variant. The fix author does not see it before the run.
  5. Add a negative control: one benign activity that resembles the technique and must not alert.
  6. Confirm for every scenario its safety class, the tag that marks its telemetry, the cleanup step and the stop-path holders. Reject any scenario that lacks one.
  7. Run the full case once. If all scenarios and the negative control pass, produce the evidence manifest and have it signed; if any fails, the finding stays open.
  8. Set the triggers: on change to the listed dependencies, and a schedule.
  9. Set the evidence expiry and enter it in the register.

Default values. Priority techniques: a re-run at least every 30 days and on change, evidence expiry of 90 days. Other techniques: every 90 days, expiry of 180 days. Detected within 15 minutes for a priority technique where the detection is the control. Pass requires all scenarios and the negative control. Adapt to the contract (an annex may fix the sample), to the works agreement (results in aggregate, never by person) and to the organisation's risk appetite; record the reason for any default that is relaxed.

Definition of done. The finding has a REG-* case with an exact scenario, at least one variant, a negative control and (for a priority technique) a held-out variant; an EVD-* manifest is signed by a named person; the triggers and expiry are in the register; and the case runs in the pipeline.

Evidence as code

The evidence manifest is the unit of proof. It is the EVD-* record that ties together what ran, against what, when, with what outcome, under which version of every dependency, and who signed. Architecture as Code (Chapter 15A) made the case that evidence should be generated by operation, signed and versioned (87, Nordqvist, 2025). In the loop that means five properties.

  • Generated, not written. The runner writes the manifest from the run; a person adds a signature and a comment but not the results.
  • Content-addressed. Each artefact (the scenario run log, the telemetry extract, the detection output) is referenced by a digest, so that a later reader can check that what is cited is what was recorded.
  • Versioned with its dependencies. The manifest records the version of the detection, the scenario, the runner, the mandate and the estate snapshot it concerns. A pass against detection version 14 says nothing about version 15.
  • Signed by a person who can be named. The signature binds a role to the decision. For findings on a stop path or a containment, the signer is never an agent.
  • Minimised. Evidence extracts contain telemetry from the estate, sometimes with personal data. The manifest keeps what the reader needs to reproduce the result and not more, with exercise telemetry classified as the Region's and Hansa's profiles require. Chapter 16 treats the store as a tier-0 asset because it is a map of the estate.

The test of a manifest is whether a person who was not in the room can reproduce the claim from it. If the person can re-run the case from the manifest and arrive at the same result, the evidence is credible; if the manifest points to a dashboard that has since changed, it is not.

Worked example: Regression case and evidence manifest for one finding

The illustrative example below follows a finding at Nordvik Region. In the autumn 2027 catalogue run under the call-off agreement NR-2026-071 with the testing firm, the contracted tester, Elin Berglund, found that a slow password-spraying pattern against the benefits-triage sign-in (ATT&CK T1110.003, with T1078 as the technique that followed) produced an alert, and that the managed provider's AI closed the alert out of hours as low severity. The detection DET-NR-0001 existed; the finding was against the provider's closure of the alert. The Region changed the provider's runbook in annex MS-1 and required, before the finding could be reported closed, a pass of an injected test through the intake of annex MS-2.

# Illustrative: Nordvik Region, regression case for FND-NR-2027-011
kind: RegressionCase
id: REG-NR-0004
finding: FND-NR-2027-011
source: autumn-2027-catalogue-run-PRP-NR-2027-02
technique: {attack: T1110.003, followedBy: T1078}
protects: [DET-NR-0001, MS-1/runbook-low-severity-closure]
owner: ROLE-NR-SEC-DET
environment: benefits-triage-signin      # production, safety class S1, injected record
passCriterion:
  layer: triaged-by-person               # an alert that is closed by the provider's AI fails
  maxMinutes: {inHours: 15, outOfHours: 30}
scenarios:
  - {id: base,    kind: exact,    ref: SCN-NR-0001}   # S1 injected-record form; the live form is S2
  - {id: v1-rate, kind: variant,  change: rate-below-threshold,   author: model-proposed-person-reviewed}
  - {id: v2-zone, kind: variant,  change: source-network-class,   author: karin-lund}
  - {id: v3-hours, kind: variant, change: out-of-hours-delivery,  author: karin-lund}
  - {id: h1,      kind: held-out, author: elin-berglund, visibleToFixAuthor: false}
negativeControl: {id: n1, case: bulk-password-resets-after-holiday, mustAlert: false}
safety: {class: S1, cleanup: remove-injected-record, stopPath: [mssp-shift-lead, nr-duty-analyst]}
triggers: {onChange: [DET-NR-0001, MS-1, provider-ai-config], schedule: quarterly-MS-2}
evidenceExpiryDays: 180

The first manifest run passed the exact case and failed a variant.

# Illustrative: Nordvik Region, evidence manifest EVD-NR-2028-0003
kind: EvidenceManifest
id: EVD-NR-2028-0003
regression: REG-NR-0004
finding: FND-NR-2027-011
versions: {detection: DET-NR-0001@v3, runbook: MS-1@2027-11-03, provider-ai-config: 2028-02-01}
runs:
  - {date: 2027-11-17, scenario: base,     outcome: triaged-by-person, minutes: 6,  result: pass}
  - {date: 2028-01-19, scenario: v3-hours, outcome: closed-by-provider-ai, minutes: 41, result: fail,
     reopenedAs: FND-NR-2028-002}
  - {date: 2028-02-09, scenario: [base, v1-rate, v2-zone, v3-hours, h1, n1], result: pass,
     minutes: {base: 6, v1-rate: 9, v2-zone: 8, v3-hours: 17, h1: 21}}
artefacts:
  - {path: evidence/REG-NR-0004/2028-02-09/runlog.json,   sha256: 3f9c1e07a2d4}
  - {path: evidence/REG-NR-0004/2028-02-09/telemetry.json, sha256: b81d55e0c6a9}
reproducedBy: elin-berglund
signedBy: {role: ROLE-NR-SEC-DET, person: karin-lund, date: 2028-02-10}
expires: 2028-08-07
status: closed-verified

Three features carry the lesson. The exact re-test of 2027-11-17 passed, and under a rule that closed findings on a passing re-test the finding would have been closed that day; the out-of-hours variant on 2028-01-19 failed, because the provider's AI closed an alert delivered at night. The case caught a weakness the exact scenario could not, and the failure was recorded as a new finding, not buried in the old one. The held-out variant h1 passed in 21 minutes, within the out-of-hours bound of 30. And the expiry runs from the date of the last full pass, 2028-02-09, not from the date of signature, so that the signature cannot extend the life of the evidence. The next quarterly injected test under MS-2 refreshes it well before 2028-08-07. (The digests are shortened for the illustration.)

Expiry

Evidence is a photograph of a moment. The estate changes, detections drift, a log source is replaced, an agent's configuration is updated, a provider changes its model, and the adversary changes too. A pass that was true in February says nothing certain about August. The book's conclusion, which Chapter 17 carries into the coverage record, is that evidence must expire, and that evidence which never expires is not credible. The reasoning is arithmetic: if no limit is set on the age of a pass, the share of controls reported as verified can only rise, whatever happens to the estate.

An expiry has two parts. A clock sets a number of days after the last full pass. A condition ends the evidence earlier than the clock when something it depends on changes: a new version of the detection, a change to the connector, a change to the model behind a triage agent, an update to the mandate. The condition does not require a person to notice; the manifest lists its dependencies, and the pipeline compares them.

The clock is set by the technique's priority and by how quickly the control is expected to decay. Hansa Mutual's defaults, taken from its profile, are 90 days for a priority technique and 180 days for others. Nordvik Region, whose cadence is quarterly, sets 180 days for the case above, long enough to survive one missed quarterly run and short enough that two missed runs turn the finding back into an open one. These numbers are choices. Their value lies in being written down, and in being reported: an expiry that is set and then silently extended is the same as none.

When the clock runs out or the condition triggers, the status changes by rule, from verified to unverified, and the register shows the age at which it changed. Nobody has to decide that a control has stopped being verified. A person decides only whether to re-run the case, to accept the risk for a stated period or to retire the control, and the decision is recorded.

Reporting

What the register reports is where the loop meets management, and the rule is plain: the report states what was verified, when, and with what result, and it reports an implemented-but-unverified control as open. Three further rules keep the report honest.

Report the denominator. "Twenty-five of 34 priority techniques are verified" tells the board more than "74 per cent". It names what was tested and what was not, and it prevents a rise in the share from hiding an expanded list or a shrunk one.

Report the reasons. Unverified is not one condition. Hansa Mutual's nine unverified priority techniques split into four whose evidence had passed its expiry, three that failed a re-test and two that had never been tested. These need different responses: the first need a re-run, the second need work in P4 Improve, and the third need a scenario. A single red cell hides which.

Report the age. An evidence-age distribution is more useful than the last run's colour. Chapter 18 develops the measures (re-test pass rate, share of priority techniques verified within their expiry, finding half-life) and the way they mislead when they reward activity instead of yield.

Hansa Mutual's board page for the verification stage reads, in code, as follows.

# Illustrative: Hansa Mutual, verification status behind BTP-HM-2027-Q3
kind: VerificationStatus
board: BTP-HM-2027-Q3
priorityTechniques: 34
verified: 25
notVerified:
  - {reason: evidence-expired, count: 4, action: re-run-case}
  - {reason: retest-failed,    count: 3, action: back-to-P4-Improve}
  - {reason: never-tested,     count: 2, action: author-scenario}
reportedAs: {verified: 25, open: 9}
expiryDays: {priority: 90, other: 180}
ratchet: {casesRemovedWithoutDecision: 0}

For a supervisor, the same records answer a different question: whether remediation after a threat-led test was carried out and shown to work. Observable, expected, speculation. That a finding is closed after a re-test and a remediation stage is observable in the instruments cited above. Whether supervisors will accept machine-generated re-test evidence in place of a tester's attestation is expected rather than observable: the instruments described above require closure and remediation stages, and the book found no published source in which a supervisor states that automated re-test evidence suffices. The architectural response is to produce evidence that a person can reproduce, which is the same test that the manifest already meets. A closure signed by an agent is speculation, and this book's rules do not allow it.

Across the usage modes and degrees of delegation

The mode describes how the AI is deployed and whose identity it uses; the level describes who decides each action. The ceiling for catalogued re-tests is Level 3 Bounded, defined by a catalogued scenario and a rules-of-engagement manifest enforced by a policy decision point outside the agent. Level 4 Autonomous is never used for a re-test that is irreversible, that leaves the signed scope, that touches personal data beyond what the exercise is authorised to touch or that targets a named person.

Activity in P5 Verify Usual mode Default ceiling Condition
Draft a regression case from a finding Mode 1 to Mode 3 under the engineer's identity Level 1 Assisted A person reviews every scenario and the pass criterion
Propose variants Mode 3 or Mode 4 Level 1 Assisted Labelled as model-proposed; a person reviews; a share human-authored
Run a catalogued S1 re-test Mode 5 emulation runner Level 3 Bounded Inside the signed scope; tagged; stop path outside the runner; not the agent that drafted the fix
Run a re-test that needs judgement or touches production data Mode 3 or Mode 5 Level 2 Supervised A person approves each run
Assemble and hash the evidence manifest Pipeline Level 3 Bounded Generated from the run, never edited
Compare an evidence manifest with its dependencies and expire it Pipeline Level 3 Bounded Rule-based; reversible; reported
Sign closure Person Not delegated A named role; never an agent, and never the author of the fix alone
Close a finding on a stop path or a containment Person Not delegated Needs a demonstration that the stop holds without the cooperation of the system it stops
Retire a regression case Person Not delegated Recorded decision, tied to retirement of the control

A re-test run by the same agent that drafted the fix is not independent. Chapter 15 treats what independence means when both sides share a model family, data or a vendor; for this chapter it is enough that the author of a change neither runs the re-test nor signs closure alone.

Nordvik Region and Hansa Mutual

Nordvik Region. The Region buys the loop, and its leverage in P5 Verify is the contract. It requires the managed provider to pass an injected test before it reports a finding closed, using the five seeded incidents per class per quarter of annex MS-2, and it treats the provider's own statement that a runbook is updated as an assertion until the test has passed. The contracted tester is not the detection owner, which gives independence by role; Karin Lund, the detection owner, signs closure, and Elin Berglund reproduces the result. With six analysts in office hours, the Region cannot maintain a large suite, so it keeps a small number of cases, each with a held-out variant, and sets long expiry (180 days) on a quarterly cadence. Its risk is a loop that stops after the first report, and the expiry clock is the mechanism that makes the absence of a re-test visible. Findings and evidence are held in the separate, restricted store described in its profile.

Hansa Mutual. The group runs the loop, and its leverage is the pipeline. HM-SEC-004 runs the regression cases on a rolling schedule so that each priority technique is exercised at least every 30 days and after any change to a detection, a connector or a mandate. At BTP-HM-2027-Q3 the group reported 25 of 34 priority techniques verified, and the other nine as open with the reasons given above: four with expired evidence, three failed, two never tested. The risk at Hansa is the opposite of Nordvik's: a pipeline that runs every night and is read by nobody. The remedy is the held-out variant authored by the red team under Mareike Hartmann, the periodic human replay, a report by evidence age, and a rule that an agent which drafted a detection (HM-SEC-002) is not the agent that verifies it. Lena Vogel's detection-engineering team signs closure; the red team signs the re-test. Under the exercise clause KBV-IT-2024-SOC-05, no re-test result enters an appraisal.

Summary

This chapter treated P5 Verify, the stage at which the loop closes or fails to. The closure rule has four clauses: a finding is closed only when the re-test and its variant pass, the evidence is attached, a named person signs, and the closure carries an age. An implemented but unverified control is reported as open. The register shows a priority technique in four states, verified, implemented-unverified, failed and not tested, which project the finding states of Chapter 4. It takes its input from P4 Improve, runs its tests as P2 Emulate and repeats the comparison of P3 Observe.

The exact re-test is necessary and insufficient, because a detection shaped by a finding passes the scenario that shaped it. Variants of path, context and rate, a held-out variant written by someone other than the fix author, and a negative control turn a re-test into evidence about the technique. Regression cases keep the result alive by running on change, on schedule and on intelligence, and the ratchet prevents their quiet removal. Evidence is code: generated by the run, hashed, versioned with its dependencies, signed by a person and minimised. It expires by clock and by condition, and a manifest that a person who was not in the room can reproduce is credible, while one that never expires is not. These conclusions, including the choice of expiry defaults, are this book's own.

Re-tests are emulation, so they carry the P2 Emulate rules: catalogued and reversible, with a stop path outside the system that is stopped, run by an agent at no more than Level 3 Bounded and never by the agent that wrote the fix. The reader can begin by choosing one finding that the register calls closed, asking for its re-test date, its variants and its signer, and reporting it as open if any of the three is missing.

Chapter 12 turns from the stages of the loop to the organisation's own AI systems and agents as targets of the loop, the detections that catch attacks on them and the regression cases they add.

Sources

  1. European Parliament and Council of the European Union (2022). Directive (EU) 2022/2555 on measures for a high common level of cybersecurity across the Union (NIS 2 Directive). Official Journal of the European Union. (3, European Union, 2022)
  2. MITRE Corporation. MITRE ATT&CK: Adversarial Tactics, Techniques, and Common Knowledge. (17, MITRE, n.d.)
  3. European Central Bank (2025). TIBER-EU: Threat Intelligence-Based Ethical Red Teaming framework. ECB. (20, ECB, 2025)
  4. Basiri, A. et al. (2016). Chaos Engineering. IEEE Software 33(3). (77, Basiri et al., 2016)
  5. Principles of Chaos Engineering community. Principles of Chaos Engineering. (78, Principles of Chaos Engineering community, n.d.)
  6. Nordqvist, G. (2025). Architecture as Code. The Transformed Architecture, Book 1. (87, Nordqvist, 2025)
  7. Nordqvist, G. (2026). AI Governance. The Transformed Architecture, Book 2. (88, Nordqvist, 2026)
  8. Nordqvist, G. (2026). AI Platform and Security. The Transformed Architecture, Book 3. (89, Nordqvist, 2026)
  9. Nordqvist, G. (2026). AI Red Team. The Transformed Architecture, Book 4. (90, Nordqvist, 2026)
  10. Nordqvist, G. (2026). AI Threat. The Transformed Architecture, Book 5. (91, Nordqvist, 2026)
  11. Parasuraman, R. & Manzey, D. H. (2010). Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors 52(3). (101, Parasuraman & Manzey, 2010)
  12. Nordqvist, G. (2026). AI Blue. The Transformed Architecture, Book 8. (94, Nordqvist, 2026)
  13. European Commission (2025). Commission Delegated Regulation (EU) 2025/1190 (regulatory technical standards on threat-led penetration testing under DORA). Official Journal of the European Union. (8, European Commission, 2025)