LAKHOWAL

TEST IT

Don't just test the decision. Test the resulting effect.

  1. Decision
  2. Resulting effect
  3. Independent observation
  4. Evidence

THE QUESTION

The assurance question

The control says it denied the action.

Did the action actually fail to take effect?

The control says it permitted the action.

Was it the action that was authorized that took effect?

The record says what happened.

Does the record correspond to the observed effect?

BENCHMARK

AI Execution Assurance Benchmark

A public benchmark release defines the tests, vectors and metrics.

EI-M01

Unauthorized Execution Rate (UER)

EI-M02

False Permit Rate (FPR)

EI-M03

Fail-Closed Rate (FCR)

EI-M04

Boundary Bypass Rate (BBR)

EI-M05

Replay Determinism Rate (RDR)

EI-M06

Evidence-Ordering Conformance (EOC)

Source: Execution Integrity metric family v1.0 · benchmark release v1.2.0.

Benchmark release

TESTS

Twelve tests in five families

FamilyTestsWhat it establishes
Binding
  • T3Action and parameter binding (action, parameter)
  • T4Target binding (target)
  • T5State binding (state)
an authority authorizes exactly what it was issued for
Authority lifecycle
  • T6Temporal validity (not-yet-valid, expired)
  • T7Replay / single-use protection (consumed, sequential replay)
  • T8Revocation enforcement (revoked, revocation race)
  • T11Concurrent single-use redemption (concurrent double redemption)
an authority authorizes only within its validity, only once if single-use, and never after revocation
Boundary integrity
  • T2Missing or invalid authorization prevents execution (no or invalid authority; direct bypass)
  • T9Fail-closed runtime behaviour (enforcement-component failure, fail-open attempt)
  • T12Prohibited partial execution
nothing crosses the boundary that was not authorized, in whole or in part, by any path
Evidence integrity
  • T10Evidence and effect ordering (evidence before effect)
  • Replay determinism across all tests (EI-M05)
every effect is reconstructable from committed evidence and repeats identically
Utility / positive conformance
  • T1Valid authorization permits the intended action
  • The positive twin of every test; supporting False Denial Rate
a system cannot pass by denying everything

Source: Execution Integrity metric family v1.0 · benchmark release v1.2.0.

OBSERVATION

Independent observation

System claim

≠

Observed effect

REPRODUCIBILITY

Reproducibility

  • · Frozen test vectors.
  • · A public release with a persistent identifier.
  • · A public repository.
View reproducibility

CONFORMANCE

Conformance-oriented testing

  1. Define the property
  2. Run the tests
  3. Observe the effect
  4. Report the result

Conformance-oriented testing is not certification, approval or endorsement.

AVAILABLE NOW

Independent Execution Assurance