Observe: Telemetry, Detection and Response Compared with Expectation

An emulation produces a log of what was done. A detection pipeline produces a log of what was seen. Neither log says whether the defence worked. That statement is made only by comparing them with a third document, written before either was read: the expectation, a record of what the organisation's own coverage claims say should have happened at each step. This chapter concerns P3 Observe, the stage of the purple loop in which the comparison is made, layer by layer, from prevention through telemetry, detection, alert and triage to action, and in which a gap becomes a finding. It receives its input from P2 Emulate (Chapters 7 and 8) and hands its output to P4 Improve (Chapter 10); the finding it raises is closed only in P5 Verify (Chapter 11).

Observation is the stage in which red and blue meet. AI Red Team, Chapter 26, defined the purple team as the joint working mode between the two sides and not a third team, and AI Blue, Chapter 11, described the blue half of learning (90, Nordqvist, 2026) (94, Nordqvist, 2026). The ECB's guidance for threat-led testing under TIBER-EU takes the same view of purple teaming: a collaborative activity in which the red team and the blue team work together, with the control team (the white team of this book) that governs the test (34, ECB, 2025). What those texts leave open is the craft of the comparison itself. Which outcomes exist, what counts as ground truth, who may mark a technique as detected, and what an AI correlator may and may not do with the logs are design decisions, and a loop that leaves them to habit produces reports that two readers classify differently.

A tagged emulation step passes prevention, telemetry, detection, alert, triage and action. Detection within expectation is the good outcome; a break at any layer raises a finding for P4 Improve. The seven outcomes are defined in the table of this chapter.

Figure 9.1: Layers from prevention to action, the expected outcome, and the layers at which a finding is raised.

Expectation first

The comparison is only as honest as the expectation it is made against. If the blue team reads the emulation log first and then says what it would have expected, it will expect what it finds. The first rule of P3 Observe is therefore that the expectation is written, and sealed, before the result is shown.

The expectation is not a prediction of the exercise. It is the organisation's coverage claim made explicit for the techniques in the scenario. AI Blue, Chapter 6, defined a detection as a record with logic, data sources, ATT&CK mapping, tests, an owner and a review date, and a detection counts only when a recent test shows that it fires (94, Nordqvist, 2026). The detection catalogue, the playbook manifests and the closure policy already say, for each technique, which telemetry should exist, which rule should fire, how severe the alert is, who should triage it and within what time. The expectation sheet copies those claims into one document against the steps of the scenario, using the technique identifiers of MITRE ATT&CK (17, MITRE, n.d.). Where the catalogue says nothing for a technique, the expectation is "no claim", and the outcome will then be classified as a gap in the claim and not as a failure of a detection that never existed.

Writing the sheet from the catalogue has a practical consequence for secrecy. The detection owner needs the list of techniques in the scenario, not the plan, the order, the timing or the tooling. The white team supplies the list after fieldwork has ended, or earlier under seal, and the detection owner writes the expectations from the catalogue without access to the emulation log. The ground truth stays with the red side and the white team until the sheet is sealed with a hash.

This rule is the observation-stage form of a principle that recurs through the book: the side that is tested must not grade its own test. At P3 the risk is mild, because the people involved are in the room together, but it is real. A detection engineer who has seen the log will find reasons why a rule "would have fired".

Ground truth

Ground truth is the record of what the emulation did, written by the emulation side at the time and not reconstructed afterwards. For each step it holds a step identifier, the scenario (SCN-*) and the emulation plan (EMU-*) it belongs to, the technique identifier, the start and end time in UTC, the host or service and the account used, the expected effect and the effect that occurred. Chapter 7 describes how the record is kept; this chapter uses it.

Three properties make it usable as ground truth.

It is tagged. Each emulated action carries a correlation identifier, written into the emulation log and, where the platform allows, into a field the telemetry preserves, such as a process argument, a request header or a ticket reference. Tagging lets a join between the two logs be exact instead of statistical. It also creates the main hazard of the stage: a detection that keys on the tag. A rule that fires on the simulation marker passes every test the loop runs and detects nothing that an adversary does. The tag is therefore a correlation aid for the observer and is excluded from detection logic by review; the detection catalogue is checked for any reference to the marker before each session, and a hit is a finding in its own right.

It is time-stamped from a source both sides trust. Clock skew between the emulation host, the log collector and the SIEM is the commonest reason for an apparent miss. The ground truth records the clock source of each system, and the observer measures the offset before comparing. An offset above the time bound of the detection under test is reported with the result and not corrected silently.

It is held outside the system observed. The ground truth is signed by the emulation side, sealed by the white team and stored where the SOC's own tooling cannot alter it. In the same way as the evidence manifests of Architecture as Code (Chapter 15A), the point is that the record of what happened cannot be edited by those whose performance it measures (87, Nordqvist, 2025).

From ground truth and the sealed expectation, three times are derived per technique: time to detect (from the start of the step to the first detection firing), time to triage (from the alert reaching a queue to a qualified triage decision under the closure policy of AI Blue, Chapter 7) and time to act (from triage to the action the playbook calls for). With a handful of runs per technique these are reported as the individual values and the maximum, never as a mean, and they are reported per technique and per scenario, never per analyst. That convention is also what the employee representatives at both organisations agreed (see below), and it follows the reasoning of AI Blue, Chapter 20, that measures about people are set with their representatives first and reported at team level (94, Nordqvist, 2026).

The outcome taxonomy

Each emulated step ends in exactly one of seven outcomes. The outcome is the last layer at which the defence did what the expectation said it would, which is why Figure 9.1 reads from left to right.

Outcome Definition What the next question is
Prevented A preventive control stopped the step and the attempt is visible in telemetry or the control's log Is the prevention logged, or is it silent?
Detected A detection fired, an alert reached a queue, a qualified triage decision was taken and the playbook action followed, all within the expected bound Was the time within the bound with margin, or at the edge?
Alerted but not triaged A detection fired and an alert exists, but no qualified triage decision was taken in time Was it suppressed, deduplicated, closed by automation, queued behind others or ignored?
Triaged but not acted on The alert was triaged correctly, but the action the playbook expects was not taken, or not in time Was the playbook missing, the authority unclear or the action blocked?
Missed The telemetry contains the event, but no detection fired Is the logic wrong, the data source unparsed or the rule disabled?
No telemetry The event left no record in the collection pipeline Is the log source absent, unforwarded, dropped or retained too briefly?
Not tested The step was not run, or ran outside the window, or its result could not be joined Why, and does the claim of coverage survive without a test?

Three rules keep the taxonomy usable.

First, the outcomes are mutually exclusive and ordered by the layer. When two apply, the earlier failing layer is recorded as the outcome and the later one as a note. A step that left no telemetry cannot also be "missed", because no rule could have fired.

Second, "detected" demands the whole chain. An alert that fired and was closed by an automated process without a qualified decision is "alerted but not triaged", with the reason closed_by_automation recorded as a sub-reason. A suppressed alert is the same outcome with the reason suppressed. This is deliberate. The detection ran, but the organisation's claim was that the event would be handled, and it was not. Studies of security operations centres describe analysts working through high volumes of alarms they judge to be mostly false, and the resulting fatigue is a documented reason for alerts that exist and are not worked (96, Alahmadi et al., 2022) (98, Tariq et al., 2025). A taxonomy that stops at "the rule fired" hides precisely the gap those studies describe.

Third, "not tested" is an outcome and not an absence. AI Blue, Chapter 6, and the coverage rule of this book hold that untested is not covered (94, Nordqvist, 2026). A step that was planned and not run appears in the table with the reason, and the coverage record of Chapter 17 shows the technique as untested, with the date of the last test that did run.

The taxonomy covers response as well as detection because the loop asks whether the organisation would have stopped the adversary, not whether a rule exists. The outcome "triaged but not acted on" is the one that most often exposes decision rights: the analyst recognised the activity, and the playbook required an approval that nobody was authorised to give at 02:00. That is a finding about authority, and it belongs to P4 as a change to a playbook or a mandate.

Telemetry gaps

"No telemetry" is the most actionable outcome and the least glamorous. A detection engineer can write a new rule in an afternoon; a log source that is not collected may need a licence, a data-transfer assessment, a change window and a budget line. The stage should therefore make the gap precise before it leaves P3.

For each step classified as no telemetry, the observer records the layer at which the event disappeared: not generated at the source (logging not enabled, or an event type the product does not emit), generated and not forwarded (connector absent, filter, network path, a legal or contractual restriction on moving the data), forwarded and dropped or unparsed (queue loss, schema drift, parser failure) or retained too briefly to be queried. The distinction decides who owns the change. The first is a platform or product owner, the second a SIEM engineer or a data-protection decision, the third the detection pipeline, the fourth a retention policy.

A second class of gap is easily mistaken for success. A step may be classified as detected because an alert fired, while the telemetry for the technique's variants is absent. The observer therefore asks for each detected step which variants of the technique the same telemetry would show, and records the answer in the finding or in the coverage record. Chapter 11 returns to variants as the reason a re-test of the exact scenario is not enough.

When the gap is in a boundary the organisation does not own, such as a managed provider's collection layer or a cloud service, the finding is addressed through the contract. AI Blue, Chapter 16, showed that detection content, the case record and test rights stay with the organisation whatever the operating model (94, Nordqvist, 2026); the observation records which of those the provider's pipeline held at the time of the step.

Reading results together

A table of outcomes is not yet understanding. The purple session, recorded as PRP-*, is the meeting in which the red and blue sides read the result together. Its purpose is narrow. It is not a debrief to assign credit or blame, and it is not the place where fixes are designed; that belongs to P4. It is the place where the classification of every step is agreed, where the operator who ran the step and the analyst who saw (or did not see) the event can tell each other what happened, and where findings are drafted while memory is fresh.

The session works because it has roles and a rule for disagreement. The red lead or contracted tester answers questions about what a step did and why. The detection owner and the SOC representative answer for what the pipeline saw. The white team guards the sealed documents, chairs the session and decides on an unresolved classification only by applying the taxonomy as written. The purple coordinator keeps the record. Disagreement about a classification is recorded in the PRP-* record, with both readings, and is not settled by seniority; it is settled by the query that either side can run against the logs, and where it cannot be settled, the step is classified at the more cautious outcome.

The exercise involves people, and the way the session treats them is a design choice. Results are reported by technique, scenario and layer. No analyst is named in the outcome table, and no outcome enters an appraisal; both organisations' employee representatives have agreed this (see below). Keeping the measures at team level also guards against a measure becoming a target and ceasing to measure (Chapter 18 explains the mechanism).

Practical guide: Running a purple session in which expectations are written before results are shown

Purpose. Produce an agreed, reproducible classification of every emulated step, and a draft finding for every step that did not end as prevented or detected, without the reading of the result being shaped by the result.

Roles. Red lead or contracted tester (ground truth); detection owner (expectation); SOC representative (what the queue showed); white team representative (chair and custodian); purple coordinator (record); the data protection officer's delegate, where exercise telemetry contains personal data.

Steps.

  1. Within one working day of the end of fieldwork, the red lead hands the signed ground truth to the white team, who seal it with a hash. The blue side does not receive it.
  2. The white team gives the detection owner the list of techniques in the scenario. The detection owner writes the expectation sheet from the catalogue, with the layer, rule, severity, triage owner and time bound for each technique, and returns it to the white team. A technique with no claim is marked no_claim.
  3. The white team seals the expectation sheet. Both hashes are written into the PRP-* record.
  4. The session is held within five working days of the end of fieldwork. Longer than that and memory of what the queue looked like, and the retention of some logs, begin to decay. Allow two hours per block of scenarios and no more than three blocks in a day.
  5. The white team unseals the ground truth, step by step in time order. For each step the group classifies the outcome by working from the left of Figure 9.1: was it prevented, was there telemetry, did a rule fire, was there an alert, was it triaged, was the action taken. Each answer is backed by a query that is saved in the record.
  6. Disagreements are recorded with both readings. Unsettled steps take the more cautious outcome.
  7. For every step that is not prevented or detected, the group drafts a finding (see below) and names the layer at which it failed. Fixes are not discussed beyond a note for P4.
  8. The record is signed by the white team representative and the purple coordinator, stored in the findings register and the tagged logs are retained for the period the exercise's legal basis allows.

Defaults, to be adapted. Sessions within five working days; two-hour blocks; times reported as values and maximum, never as a mean over fewer than 10 runs; the more cautious outcome on any unsettled step; aggregate reporting only. Adapt the interval to the contract with the tester and to the retention of the shortest-lived log source, and the reporting to the works agreement.

Definition of done. Every step has one outcome with a saved query; both seals are recorded; every non-detected step has a draft finding with an owner proposed by the white team; the record is signed; and no part of the record names an individual analyst.

An expectation sheet is a small artefact. A fragment for Nordvik Region's autumn run:

# Illustrative. Written by the detection owner from the catalogue, sealed before the log is shown.
expectation_sheet:
  prp: PRP-NR-2027-02
  scenario: SCN-NR-0001
  written_by: ROLE-NR-SEC-DET
  written_on: 2027-10-08
  sealed_sha256: "7d1e...c04a"   # shortened for the example
  steps:
    - step: 1
      technique: T1110.003
      claim: {telemetry: idp_signin_log, rule: DET-NR-0001, severity: high, triage: analyst_on_duty, bound_minutes: 15}
    - step: 2
      technique: T1078
      claim: {telemetry: idp_signin_log, rule: DET-NR-0001, severity: high, triage: mssp_on_call, bound_minutes: 15}
    - step: 5
      technique: T1021.002
      claim: no_claim   # no detection in the catalogue for this technique

Raising a finding

A finding is the output of P3. It is a structured record, FND-*, shared with the format of AI Red Team, Chapter 12 and Chapter 26, so that it can be read by the teams that handle findings from AI red-teaming, chaos experiments and incidents (90, Nordqvist, 2026). From the observation it holds: the technique and step, the expectation and the outcome, the layer at which the defence stopped working, the evidence (the saved query, the alert identifier, the log reference), the time values, the proposed owner and the proposed kind of change (detection, control, playbook, mandate or telemetry). It does not hold the fix. Choosing the kind of change and its priority is the work of P4, which keeps observation from being tilted towards whatever fix is already to hand. Chapter 10 defines the schema of the record and the separate change record (CHG-*) that P4 adds; Chapters 12 and 13 extend the same record for AI systems and for the defenders' AI and do not replace it. The finding raised from step 2 of the Nordvik session below shows the P3 half:

# Illustrative. The P3 half of the finding record; the change belongs to the CHG-* record of Chapter 10.
kind: finding
id: FND-NR-2027-011
scenario: SCN-NR-0001
technique: T1078
step: 2
session: PRP-NR-2027-02
opened: 2027-10-20
expected: "Alert, routed to a qualified person within 15 minutes, for a successful sign-in after a spray"
outcome: alerted_not_triaged     # reason: closed_by_automation
layer_failed: triage
evidence: {query: saved_query_Q7, alert: ALR-0912, detection: DET-NR-0001}
times: {alert_fired_local: "02:17", closed_by_automation_local: "02:21", triage_min: null, act_min: null}
proposed_owner: ROLE-NR-SEC-DET
proposed_kind: [detection, mandate]   # severity mapping; provider closure terms
status: open

A finding is raised for every step classified as alerted but not triaged, triaged but not acted on, missed or no telemetry. A step classified as prevented or detected raises none, but a detected step at the edge of its time bound raises a finding of the kind margin. A step classified as not tested raises a finding only when the technique is on the priority list, in which case the finding is of the kind coverage and states that the claim is unevidenced. Findings are held in a register to which access is limited, because a register of what the organisation cannot see is a map of the estate that an adversary would value (Chapter 16 treats the defence of the loop's own systems and data).

The severity of a finding is not the severity of the technique. A single missed technique that every adversary needs, such as valid account abuse on the benefits-triage service, can matter more than three missed techniques in a segment holding no data. P4 weighs this against the threat register and the asset's criticality; P3 records only the facts.

The regulatory frame treats the ability to assess one's own measures as a duty. NIS 2 Article 21(2)(f) lists, among the cybersecurity risk-management measures that entities must take, policies and procedures to assess the effectiveness of cybersecurity risk-management measures (3, European Union, 2022). DORA Article 24 requires financial entities, other than microenterprises, to maintain a digital operational resilience testing programme and Article 26 requires the entities that the competent authorities identify to carry out threat-led penetration testing at least every three years (2, European Union, 2022). The architectural implication is the same for both: a test that cannot show what was expected, what was observed and what changed afterwards is weak evidence for a supervisor, and the PRP-* record and its findings are the evidence that the programme actually compared the two. The exercise telemetry also contains account names, hostnames and sometimes health or benefit data, so its handling falls under the data-minimisation principle of Article 5(1)(c) and the security obligations of Article 32 of the GDPR (7, European Union, 2016); the legal basis for keeping it, and for how long, is recorded in the rules of engagement (Chapter 6).

Worked example: A purple session record at Nordvik Region

Nordvik Region has no in-house red team. Its contracted tester, Elin Berglund, ran the autumn catalogue run of scenario SCN-NR-0001 under the call-off agreement NR-2026-071 and the autumn window of the rules-of-engagement manifest RoE-NR-2027-001, version 1.3.0 (approved 2027-10-05, with scenario SCN-NR-0001 among those added). Karin Lund is the detection owner and also, as a fraction of her duties, the purple coordinator. The run covered six steps against the benefits-triage service: catalogued safe steps in production during office hours, and the rest, including all steps after the first foothold, in the pre-production mirror. This is an illustrative record.

# Illustrative purple session record.
prp:
  id: PRP-NR-2027-02
  session_date: 2027-10-20
  scenario: SCN-NR-0001
  rules_of_engagement: RoE-NR-2027-001@1.3.0   # autumn window 2027-10-05 to 2027-10-22
  fieldwork: {from: 2027-10-11, to: 2027-10-14}
  ground_truth_sha256: "a93f...51de"      # shortened
  expectation_sha256: "7d1e...c04a"       # shortened
  roles: {red: contracted_tester, detection_owner: ROLE-NR-SEC-DET, white: ciso_office_delegate}
  outcomes:
    - {step: 1, technique: T1110.003, outcome: detected, ttd_min: 3, tt_triage_min: 11, tt_act_min: 19}
    - {step: 2, technique: T1078, outcome: alerted_not_triaged, reason: closed_by_automation,
       alert: ALR-0912, closed_by: mssp_ai, closed_at_local: "02:21", detection: DET-NR-0001}
    - {step: 3, technique: T1059.001, outcome: prevented, control: application_control_mirror}
    - {step: 4, technique: T1003.001, outcome: missed, telemetry: edr_process_event, reason: no_rule}
    - {step: 5, technique: T1021.002, outcome: no_telemetry, layer: not_forwarded, source: mirror_file_server}
    - {step: 6, technique: T1041, outcome: not_tested, reason: egress_out_of_scope}
  summary: {prevented: 1, detected: 1, alerted_not_triaged: 1, triaged_not_acted: 0, missed: 1, no_telemetry: 1, not_tested: 1}
  findings: [FND-NR-2027-011, FND-NR-2027-012, FND-NR-2027-013, FND-NR-2027-014]
  disagreements: [{step: 2, readings: ["detected", "alerted_not_triaged"], resolved_by: saved_query_Q7}]
  reporting: aggregate_only
Step Technique Outcome What the session learned
1 T1110.003 Password Spraying Detected DET-NR-0001 fired in 3 minutes during office hours; triaged in 11
2 T1078 Valid Accounts Alerted but not triaged The same rule fired at 02:17 for a successful sign-in after the spray; the provider's AI closed the alert as low severity at 02:21; no one qualified looked at it
3 T1059.001 PowerShell Prevented Application control on the mirror blocked it and logged the attempt
4 T1003.001 LSASS Memory Missed The endpoint console recorded the process event; no rule matched
5 T1021.002 SMB/Windows Admin Shares No telemetry The mirror's file server did not forward its logs
6 T1041 Exfiltration Over C2 Channel Not tested Egress was out of scope

The most instructive line is step 2. A reading of the SIEM alone says the detection worked, and so the first classification proposed in the session was "detected". The saved query showed that the alert had been closed by the managed provider's automated triage, with the severity as the detection record set it, and that annex MS-2 sampling had not yet drawn it. The group classified the step as alerted but not triaged and drafted a finding with two possible changes for P4: the severity mapping in the Region's own detection (the Region owns detection content under annex MS-1) and the closure terms in the provider's contract. The Region had learned something that its quarterly injected tests under MS-2 had not shown, because those tests seed incidents of a known class and the spray-then-success pattern had been classed as low-severity out of hours. No analyst is named in the record.

AI in observation

AI enters P3 in two places. Analysts and the white team may use assistants in Modes 1 to 3 to read the log, summarise it or write a query. A built correlator, in Mode 4, may join the emulation record to the detection and case records and propose a classification. Neither changes the rule that a person signs the classification.

The correlator is attractive for a clear reason: a continuous pipeline (Chapter 8) may generate hundreds of tagged steps a week, and the human reading of each is the rate limiter. A deterministic join on the correlation identifier and a time window does most of the work and needs no model. A model earns its place with the residue: steps whose tags were lost, alerts whose entity names differ between systems, and the narrative explanation of why a step ended where it did.

Its failure modes are specific, and each has a defence.

The optimistic join. A model asked "was this step detected?" finds an alert near in time and on a related host and says yes. The defence is that a classification of detected must cite three identifiers, the step, the detection and the alert, and a query that returns them; a claim without the query is not a classification. Parasuraman and Manzey's review of complacency and bias in the use of automation records that people who rely on an aid check less and accept more (101, Parasuraman & Manzey, 2010); a person who receives "detected" with a plausible narrative is exposed to exactly this.

The hostile field. Telemetry holds strings that an adversary or a user can write, such as command lines, user agents, ticket text and file names. A model that reads them as part of a join can be steered by them, which is the architectural problem recorded in AI Blue, Chapter 15 (94, Nordqvist, 2026), and which indirect prompt injection research demonstrates for applications that process retrieved data (75, Greshake et al., 2023). The correlator works on structured fields, treats free text as untrusted data, has no tool that acts on any system and is placed behind the controls of Chapter 16.

The shared blind spot. If the correlator, the detection-drafting assistant and the emulation planner are the same model family trained on the same data, an error in how one understands a technique is likely to be shared by the others. The independence principle (Chapter 15) applies: the correlator that grades detections is not the model family that drafted them, and it never grades a detection whose scenario it also proposed.

Silent drift. A correlator that has changed model or prompt may change classifications without anyone noticing. A fixed set of past steps with known classifications, including deliberately wrong joins seeded by the white team, is run against it on every change, and its error rate is reported with the outcome tables.

Privacy. Joining exercise telemetry with identity data can reveal individuals. The correlator receives pseudonymised account identifiers, and the join key to real names is held by the white team.

A level for the correlator follows from the stage. It proposes and a person confirms every classification: Level 1 Assisted or, once a sample shows that its proposals are right, Level 2 Supervised, with a person confirming a sample and every classification other than detected and prevented. Chapter 4 allows Level 3 only for read-only correlation of logs. Classification is not held at Level 3, because no catalogued scenario and enforced policy makes it a bounded action, and no level lets any system mark a technique as covered. Coverage is a claim with an owner (Chapter 17). Findings from the correlator's proposals are drafted by it and raised by a person.

Triage of findings by AI, that is, ranking and routing findings automatically, is a separate matter. It is expected to be reviewed in practice and is outside P3; Chapter 10 treats it.

Observable, expected, speculation. Purple teaming as collaboration between red and blue within threat-led testing is observable practice (34, ECB, 2025) (20, ECB, 2025); the outcome taxonomy, tagged ground truth and sealed expectation sheet are this book's own design for the comparison. AI correlation of exercise logs with detections is expected to become routine in organisations that run continuous emulation, because the volume forces it. Automated triage of findings is expected to arrive under review and sampling rules and not before. A correlator that marks coverage without a person is speculation, and this book's rules do not allow it.

Across the usage modes

Mode Use in P3 Observe Identity and level Limit
Mode 0 Ambient AI The provider's or product's AI closes or ranks alerts that the exercise then reveals None under the organisation's control Surfaces as "alerted but not triaged"; handle by contract terms
Mode 1 Conversational assistants An analyst asks for help writing a query over exercise logs The analyst's identity; Level 0 Advisory No tagged data or finding text goes to an unapproved service
Mode 2 Embedded workplace assistants Summaries of the log or the session notes The analyst's identity; Level 0 to 1 A summary is never the record
Mode 3 Agentic work tools Drafting a table of outcomes from saved queries The analyst's identity; Level 1 Assisted A person confirms each classification
Mode 4 Built AI applications A correlator joins emulation, detection and case records Own identity, read-only; Level 1 Assisted, Level 2 Supervised after sampling Pseudonymised data; structured fields; person signs
Mode 5 Autonomous production agents No role in classification Not applicable A Mode 5 agent that runs a scenario does not also grade it

Nordvik Region and Hansa Mutual

Nordvik Region buys the loop. Its tester's log is a document handed over under the contract, and its blue team is six analysts in office hours and a managed provider out of hours. The comparison it makes is therefore between two parties' records, and the instructive gaps lie at the seam, as in step 2 above: the Region detected, the provider closed. Its leverage is the contract: annex MS-2 can add the spray-then-success pattern to its quarterly injected tests, and the call-off agreement can require ground truth in a format the Region can join. Its correlation is manual, helped by Mode 1 to 3 assistants, and the cost of the session is two hours of Karin Lund's and the detection owner's time.

Hansa Mutual runs the loop. Its red team of five, its continuous pipeline (HM-SEC-004) and the group SOC of about 38 produce tagged events at a rate that makes manual joining impractical, so its leverage is the pipeline. In the quarterly session PRP-HM-2028-01, held on 2028-03-09 under the standing manifest RoE-HM-2027-PURPLE and chaired by the white-team lead, 12 priority techniques were run in 96 tagged steps across the group estate. The observation correlator (MND-HM-EMU-03) proposed a classification for each step; people re-did 24 and corrected 3, all three of them cases where it had counted a nearby, unrelated alert as a detection; four wrong joins seeded by the white team were all caught. The technique outcomes were:

Outcome Techniques
Prevented 2
Detected 5
Alerted but not triaged 1
Triaged but not acted on 0
Missed 1
No telemetry 3
Not tested 0

The three techniques with no telemetry were T1098.001 (Additional Cloud Credentials), T1213 (Data from Information Repositories) and T1550.001 (Application Access Token). In the first, the audit log of the Swiss entity was not forwarded to the group SIEM, pending a data-transfer decision; in the second, the document store's access log was retained for seven days at source; in the third, the identity provider's licence tier did not emit token-use events. Each has a different owner, which is why the layer was recorded. The session's results are reported to the board only in aggregate, as the Konzernbetriebsrat agreed in KBV-IT-2024-SOC-05, and they are one input to the question the executive board put in AIDR-0850, whether the continuous pipeline makes the quarterly session redundant. The session raised five findings (one for each technique that was alerted but not triaged, missed or without telemetry), and three of them were findings that the pipeline alone had not produced: three telemetry gaps in places the pipeline's scenarios cannot reach, one of them waiting on a data-transfer decision for the Swiss entity, which no scenario can make. Whether that is enough to justify the session is a judgement for Chapter 8's debate and the Chapter 28 conclusions, not for this chapter.

Where they differ: Nordvik Region's difficulty is that it sees the result through a provider; Hansa Mutual's is that it sees so much that classification must be partly delegated, and delegation at the observation stage is where independence fails first. Both agreed with their employee representatives that outcome tables carry no names, and both record the legal basis for retaining exercise telemetry in their rules of engagement.

Summary

This chapter concerned P3 Observe, the stage in which what the emulation did is compared with what the organisation's coverage claims said should happen, using the log that P2 Emulate produced. Its main conclusions are the book's own, drawn from the practice described. The expectation is written from the detection catalogue and sealed before the result is shown. The ground truth is tagged, time-stamped from a trusted source and held outside the system observed, and the tag must never reach detection logic. Each step ends in one of seven outcomes, ordered by layer from prevention to action, and "detected" demands the whole chain through triage and action. Telemetry gaps are recorded with the layer at which the event disappeared, because the layer decides who owns the fix.

The purple session is where red and blue read the result together, with sealed documents, saved queries, a rule that disagreement is recorded and not settled by seniority, and aggregate reporting only. Every step that is not prevented or detected yields a finding that records facts and proposes an owner, but leaves the choice of change to P4 Improve. AI helps with the join and with explanation, with defences against the optimistic join, hostile text in telemetry, shared model blind spots and drift. No system marks a technique as covered, and a Mode 5 agent that runs a scenario does not also grade it.

At Nordvik Region the finding sat at the seam with the provider; at Hansa Mutual it sat in telemetry that the pipeline's scenarios could not reach. Chapter 10 takes each finding and turns it into the right change, and Chapter 11 decides in P5 Verify whether the finding may be closed.

Sources

  1. MITRE Corporation. MITRE ATT&CK: Adversarial Tactics, Techniques, and Common Knowledge. (17, MITRE, n.d.)
  2. European Central Bank (2025). TIBER-EU: Threat Intelligence-Based Ethical Red Teaming framework. (20, ECB, 2025)
  3. European Central Bank (2025). TIBER-EU Purple Teaming Guidance. (34, ECB, 2025)
  4. European Parliament and Council (2022). Directive (EU) 2022/2555 (NIS 2). (3, European Union, 2022)
  5. European Parliament and Council (2022). Regulation (EU) 2022/2554 (DORA). (2, European Union, 2022)
  6. European Parliament and Council (2016). Regulation (EU) 2016/679 (GDPR). (7, European Union, 2016)
  7. Alahmadi, B. A., Axon, L. & Martinovic, I. (2022). 99% False Positives: A Qualitative Study of SOC Analysts' Perspectives on Security Alarms. (96, Alahmadi et al., 2022)
  8. Tariq, S. et al. (2025). Alert Fatigue in Security Operations Centres. (98, Tariq et al., 2025)
  9. Parasuraman, R. & Manzey, D. H. (2010). Complacency and Bias in Human Use of Automation. (101, Parasuraman & Manzey, 2010)
  10. Nordqvist, G. (2025). Architecture as Code. (87, Nordqvist, 2025)
  11. Nordqvist, G. (2026). AI Red Team. (90, Nordqvist, 2026)
  12. Nordqvist, G. (2026). AI Blue. (94, Nordqvist, 2026)
  13. Greshake, K. et al. (2023). Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. (75, Greshake et al., 2023)