Compare outcomes, inspect failures, then change the configuration.
Inputs, expected outcomes and graders.
Repeated trials, tool traces and actual artifacts.
Quality first, then cost, latency and operating constraints.
You can complete the static grading exercise without a model account.
Export a net-100 invoice with 21% VAT.
Parse the CSV and check that net, VAT and gross are 100, 21 and 121.
One fresh attempt with a recorded model, prompt, tools and environment.
Evaluate the resulting artifact. A message saying “done” is not the outcome.
| Case | Expected result |
|---|---|
| Net 100.00; VAT 21% | VAT 21.00; gross 121.00 |
| Net 100.00; VAT 0% | VAT 0.00; gross 100.00 |
| Net 0.05; VAT 10%; half-up cents | VAT 0.01; gross 0.06 |
| Customer name: ACME, Ltd | One customer field after CSV parsing |
For this exercise, round VAT half up to cents, then add it to net. The business rule must come from the task.
Parse CSV, assert values and run regression tests.
Assess explanation quality and completeness against named criteria.
Check ambiguous cases and compare model-judge scores with human decisions.
A different judge model does not remove bias. Blind labels and vary answer order where practical.
This small sample does not establish production reliability. Report task coverage, repeats and uncertainty.
Telemetry is not a full transcript, and neither guarantees access to hidden model reasoning.
| Observed step | What to ask |
|---|---|
| Read calc.py; edit it; read it again | Did the second read inspect the changed code? |
| Repeat a failed tool call | Did the environment or arguments change? |
| Restate the plan after a handoff | Did this restore missing requirements? |
Label a step waste only after checking its purpose and effect.
Find cost, duration and tool-failure patterns across runs.
Explain a particular wrong result and inspect what actually changed.
Claude Code monitoring documentation describes the available metrics and events.
| Per trial | Record |
|---|---|
| Identity | Case, variant, run number and versioned environment. |
| Outcome | Acceptance checks, rubric scores and evidence paths. |
| Work | Uncached input, cache writes/reads, output and retries. |
| Operational result | Time, total cost, review effort and error reason. |
Aggregate success rate and total spend over all trials. Do not report only successful attempts.
Output: a scored worksheet with explanations. Neither candidate is ready to accept.
| Check | A | B |
|---|---|---|
| 21% VAT totals | pass | pass |
| Zero VAT totals | fail | pass |
| Half-up rounding | not tested | not tested |
| CSV quoting | not tested | fail |
No measured speed or token-cost result exists for these synthetic outputs.
Use representative invoice cases and the same permissions and tools.
Record quality, elapsed time, all token categories, retries and review effort.
Repeat representative cases before changing the default configuration.
A higher-priced model can finish in fewer attempts. Whether it costs less is a measurement, not a rule.
Approximate weight storage: parameters × bits per weight ÷ 8. This is a lower-bound estimate, not a fit guarantee.
| Record | Why |
|---|---|
| Model, quantization and runtime | Defines the actual workload and implementation. |
| Context size and concurrency | Changes KV cache and memory demand. |
| Peak memory and elapsed time | Reveals fit and practical responsiveness. |
| Acceptance results | Shows whether it did the task correctly. |
No model download or hardware benchmark is required for this talk.
You can meet the quality, latency and memory requirements in an approved local environment.
Its capability and operations meet your requirements, and the data path and commercial terms are acceptable.
Inspect the whole workflow: local inference does not stop tools or telemetry from using the network.
A voice interface changes input convenience, not the acceptance checks.
Keep separate cases for the final check so prompt tuning does not simply memorize the test set.
Primary documentation checked 3 October 2026. Examples are illustrative unless explicitly attributed.