# Part 8: How to evaluate an LLM governance system

Canonical page: https://vikrammohan.com/thought-leadership/llm-harnesses-and-governance/evaluate-llm-governance/

An LLM governance system can be assessed by the work it helps complete, the limits it enforces and the evidence it leaves when something goes wrong. A large language model (LLM) generates responses and proposed actions; governance assigns responsibility, sets permitted uses and defines how those actions and their results are checked. The assessment needs to cover the surrounding software, the controls and the people responsible for accepting the work. NIST's AI Risk Management Framework treats governance as a continuing responsibility across the system's lifecycle. [NIST: AI Risk Management Framework 1.0, GOVERN](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf)

Useful evaluation begins with a specified task and evidence of its outcome. Parts 5 to 7 recommend controls for Claude Code, a procedure for diagnosing refusals and a guide for classifying incidents; this part explains how an organisation can assess whether controls of that kind work. The evaluation questions that follow are proposals for an assessment; they do not report measurements.

## Distinguish a completion claim from an accepted result

An agent is a system that uses model decisions and tool results to pursue a task through successive actions. Its operating harness is the software that coordinates the model, context and tools. An evaluation harness supplies assessment tasks, records execution and judges outcomes. Anthropic distinguishes these roles and separates an execution transcript from the resulting state of the environment. [Anthropic: Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents)

Consider a hypothetical request to make a name-formatting function handle empty input, update its tests and write a release note. The requester must specify the required behaviour. Several pieces of evidence then answer different questions:

| Evidence | What it can establish | What still needs checking |
|---|---|---|
| The assistant says the function is fixed | The assistant has made a completion claim | Whether the code satisfies the request |
| A test runner reports success | The invoked tests passed under that run's conditions | Whether those tests cover the required behaviour for empty input |
| The resulting code and behaviour are checked against the specification | The checked requirement is met or unmet | Any requirements or environments outside that check |
| The release note is read alongside the change | Whether the note accurately describes the implemented behaviour | Whether the other acceptance conditions are met |

These are possible observations, not results of an executed test. A successful test command can coexist with an incomplete change if the tests omit the behaviour the requester needs. Acceptance criteria are the conditions used to decide whether the requested work is complete; those conditions give the result a standard to be checked against.

![Task criteria feed both execution and assessment. The execution produces a completion claim, action records and resulting artefacts, which reach the assessment as three distinct forms of evidence.](https://vikrammohan.com/thought-leadership/llm-harnesses-and-governance/figures/fig-08-01-different-evidence-answers-different-outcome-questions.svg)

*Figure 8.1. A proposed assessment distinguishes the assistant's completion claim, execution records and the resulting state. Acceptance criteria provide the basis for examining how those forms of evidence relate to the requested task.*

## Read incident evidence at its stated scope

Claude Code is an application that combines an LLM with tools for software work. The controls recommended in [Part 5](https://vikrammohan.com/thought-leadership/llm-harnesses-and-governance/claude-code-guidance-and-enforcement/) include custom hooks, which are scripts invoked at defined runtime events, and a log of their refusals. [Part 6](https://vikrammohan.com/thought-leadership/llm-harnesses-and-governance/claude-code-refusals-and-hook-testing/) explains how refusals and failures are investigated, and [Part 7](https://vikrammohan.com/thought-leadership/llm-harnesses-and-governance/claude-code-governance-incidents-and-limitations/) sets out the limits of hook controls, a guide for classifying incidents and a matrix of control tests.

One scenario in that guide has a guard refuse the creation of an operating system service while allowing a service query. Such an observation provides evidence about those two operations. It does not supply a count of all permitted and prohibited operations from which to calculate a general accuracy rate. The guide's two false positive scenarios, in which the controls blocked permitted actions, are useful diagnostic cases; their frequency across a workload is not established.

The scenario in which a hook error appears and the command runs has a different evidential status. A traceback is the error record showing where execution failed. Without that record, the suggested exception at import time remains a probable explanation, and attributing the event to a confirmed cause would overstate the case.

## Choose evaluation questions and the evidence they require

The following questions could guide an assessment. They are not mandatory targets; select those that match the risks of the workload.

| Evaluation question | Possible measure or comparison | Evidence needed |
|---|---|---|
| Is useful work completed? | Accepted tasks divided by attempted tasks for a defined workload | Task specification, acceptance decision and resulting artefact |
| Do prohibited actions proceed? | Prohibited cases that proceed divided by prohibited cases tested | Cases classified in advance, requested operations and observed outcomes |
| Is permitted work interrupted? | Permitted cases blocked divided by permitted cases tested | The intended permission rule, the operation and the control decision |
| Can a refusal be explained? | Whether sampled refusals can be attributed to a particular layer | Transcript, configuration version and available control records |
| Does work resume correctly? | Correct resumption and rework across comparable interrupted tasks | State before interruption, retained records and the resumed result |
| Is the system economical to operate? | Total cost, elapsed time and human effort per accepted task | Recorded usage, run duration, review time and acceptance results |
| Can a behaviour be traced to its configuration? | Whether a sampled run can be associated with the controls that governed it | Version history, decision records and run configuration information |

A denominator is the total against which a count is compared. A count of blocked actions does not show how often permitted work is blocked unless the number of permitted cases is also known. Likewise, a count of completed tasks becomes a success rate only when the attempted tasks are defined and counted.

![Three calculation groups show accepted tasks divided by attempted tasks, prohibited cases proceeding divided by prohibited cases tested, and permitted cases blocked divided by permitted cases tested.](https://vikrammohan.com/thought-leadership/llm-harnesses-and-governance/figures/fig-08-02-define-the-denominator-before-reporting-a-rate.svg)

*Figure 8.2. Three possible rates use different populations: attempted tasks, prohibited cases tested and permitted cases tested. These definitions support a future evaluation; worked incident scenarios do not contain the totals needed to calculate them.*

Keep workload categories visible. A short code lookup, a resumed editing task and the production of a document place different demands on the system. Combining them into one average may conceal the category with the most failures or the greatest review burden.

A comparison between configurations also needs comparable tasks and starting conditions. A faster run may have received an easier problem or inherited more completed work. Describe those conditions alongside any outcome so the comparison can be interpreted.

## Include the effort needed to accept and recover work

A control can prevent an unwanted operation while interrupting legitimate work. More detailed records can improve diagnosis while taking time to produce and review. Additional model calls may add useful scrutiny while increasing cost. An evaluation should account for these effects in relation to the task being assessed.

For the hypothetical code change, relevant effort includes producing the change, checking it, investigating any refusal and correcting incomplete work. Model response time covers only part of that effort. A useful comparison would follow the work through to acceptance, including the person's review time.

The controls recommended in [Part 5](https://vikrammohan.com/thought-leadership/llm-harnesses-and-governance/claude-code-guidance-and-enforcement/) make explicit choices about these effects. Bounded jobs are delegated under specified briefs, the written instructions given to a delegated agent, and safety hooks and convenience hooks have different failure behaviour. Safety hooks are intended to fail closed, withholding an operation when the control cannot evaluate it; convenience hooks fail open, allowing continuation.

## Run the assessment in a defined order

One workable order for an assessment is:

| Step | Activity |
|---|---|
| 1 | Define the workload, its categories and the acceptance criteria for each task. |
| 2 | Classify test cases in advance as permitted or prohibited under the intended rules, and record the configuration version that will govern them. |
| 3 | Run the tasks and cases, recording the requested operations, the control decisions and the resulting artefacts. |
| 4 | Calculate each rate against its own denominator, and report workload categories separately. |
| 5 | Record the effort needed to reach acceptance, including review time and the investigation of refusals. |
| 6 | Keep any unexplained event open, with a note of the evidence that is missing. |

## Check which findings transfer to another environment

An environment with a different operating system, permission model, tool interface or workload may need different evidence before adopting the same conclusions. The numerical thresholds and identifiers in Parts 5 to 7 are examples to adapt; they do not establish universal values for every LLM application.

For a proposed transfer, identify which responsibilities belong to the model, which the runtime enforces, which depend on the operating environment and which remain with a person. Then specify what observation would show each responsibility being met. An unresolved incident should remain unresolved until new evidence supports a firmer conclusion.

Part 8 of 8 in the series [LLM harnesses and governance](https://vikrammohan.com/thought-leadership/llm-harnesses-and-governance/).

Previous part: [Part 7: How to classify Claude Code governance incidents and manage the limits of hook controls](https://vikrammohan.com/thought-leadership/llm-harnesses-and-governance/claude-code-governance-incidents-and-limitations/)

Vikram Mohan is not a solicitor or barrister. Nothing on this site is legal advice. For legal advice, speak to a regulated legal professional.

© Vikram Mohan - All rights reserved. No liability accepted for content on this website. [LinkedIn](https://www.linkedin.com/in/mohanvikram/) · [Privacy](https://vikrammohan.com/privacy)