A half-day workshop: define, measure, improve and verify one workflow.
| Block | Time | Participant output |
|---|---|---|
| Define and evaluate | 50 min | Acceptance cases + scored worksheet |
| Cost and context | 45 min | Usage sheet + revised instructions |
| Break | 15 min | |
| Coordinate and package | 55 min | Ticket contract + verification skill |
| Choose tools and next experiment | 45 min | Comparison design + next action |
Total: 210 minutes. Includes discussion; live execution is optional.
Export a net-100 invoice with 21% VAT.
Parse the CSV and check that net, VAT and gross are 100, 21 and 121.
One fresh attempt with a recorded model, prompt, tools and environment.
Evaluate the resulting artifact. A message saying “done” is not the outcome.
| Case | Expected result |
|---|---|
| Net 100.00; VAT 21% | VAT 21.00; gross 121.00 |
| Net 100.00; VAT 0% | VAT 0.00; gross 100.00 |
| Net 0.05; VAT 10%; half-up cents | VAT 0.01; gross 0.06 |
| Customer name: ACME, Ltd | One customer field after CSV parsing |
For this exercise, round VAT half up to cents, then add it to net. The business rule must come from the task.
Parse CSV, assert values and run regression tests.
Assess explanation quality and completeness against named criteria.
Check ambiguous cases and compare model-judge scores with human decisions.
A different judge model does not remove bias. Blind labels and vary answer order where practical.
This small sample does not establish production reliability. Report task coverage, repeats and uncertainty.
Telemetry is not a full transcript, and neither guarantees access to hidden model reasoning.
| Observed step | What to ask |
|---|---|
| Read calc.py; edit it; read it again | Did the second read inspect the changed code? |
| Repeat a failed tool call | Did the environment or arguments change? |
| Restate the plan after a handoff | Did this restore missing requirements? |
Label a step waste only after checking its purpose and effect.
| Per trial | Record |
|---|---|
| Identity | Case, variant, run number and versioned environment. |
| Outcome | Acceptance checks, rubric scores and evidence paths. |
| Work | Uncached input, cache writes/reads, output and retries. |
| Operational result | Time, total cost, review effort and error reason. |
Aggregate success rate and total spend over all trials. Do not report only successful attempts.
Output: a scored worksheet with explanations. Neither candidate is ready to accept.
| Check | A | B |
|---|---|---|
| 21% VAT totals | pass | pass |
| Zero VAT totals | fail | pass |
| Half-up rounding | not tested | not tested |
| CSV quoting | not tested | fail |
No measured speed or token-cost result exists for these synthetic outputs.
| Category | What to record |
|---|---|
| Uncached input | New input processed at its applicable input rate. |
| Cache writes and reads | Separate token counts and the configured cache rates. |
| Generated output | Output usage, including billable reasoning where applicable. |
Bill = sum of each category’s tokens × its rate, plus any separate tool or service fees.
| Illustrative usage and rates | Cost |
|---|---|
| 19,000 cached input × $0.20 / million | $0.0038 |
| 3,700 new input × $4 / million | $0.0148 |
| 1,600 output × $20 / million | $0.0320 |
| 22,700 input + 1,600 output in total | $0.0506 |
Assumed rates, not a measured bill. Without the cache hit: $0.0908 input + $0.0320 output = $0.1228.
| Need | Check before choosing |
|---|---|
| Daily interactive work | Native client, plan limits and actual monthly usage. |
| Custom service, batch or CI | Supported API authentication, rate limits and a budget. |
| Organizational controls | Identity, data handling, audit, residency and procurement requirements. |
Compare current offers against these requirements. Detailed shopping questions are in reference material.
The invoice rule, failing case, relevant code, decisions and latest test result.
Duplicate file dumps, stale logs and abandoned approaches, once their useful findings are captured.
Watch for missed requirements and repeated work. There is no universal “dumb zone” or safe fill percentage.
Use the instruction-file convention your harness actually loads. Keep standing rules short and specific.
Long project history, duplicate style advice, VAT rules, test commands and an old incident log.
Keep VAT source, rounding rule, test command and schema boundary. Link occasional procedures from a skill.
The acceptance check is unchanged behavior on the same tasks, not the smallest file.
Output: a comparison worksheet and one justified decision. A smaller prompt that drops the VAT rule loses.
| Requirement | A | B |
|---|---|---|
| VAT from order + rounding rule | present | verify preserved |
| Exact invoice test command | present | verify preserved |
| Schema change approval | present | verify preserved |
| Measured task usage and results | not run | not run |
Discuss what B removed and which regression case would expose a lost rule. No savings are claimed.
Start fresh with its own scope and needed project context.
Summarize the goal, evidence, decisions, failed attempts and next check.
Check what survives compaction. Neither a fresh session nor a summary has zero context cost.
Example return: VAT calculation locations, the failing path, supporting lines and unresolved questions.
Who gets a task, what it depends on and who integrates the result.
What files, credentials, processes and network endpoints the tools can access.
A dashboard or Git worktree does not by itself enforce a security boundary.
Failed or blocked work returns with evidence. A green tool run does not automatically publish a change.
| Event | Expected state or action |
|---|---|
| Worker finds missing VAT rule | Blocked; request the rule before changing behavior. |
| Two workers claim the same ticket | One claim wins; the other does not start editing. |
| Tests pass but escaping is untested | Review records the gap and adds an acceptance check. |
Exercise: explain who owns each decision and what evidence is attached.
Review mounts, egress and permissions in the actual runtime.
| Failure | Concrete control |
|---|---|
| Repeats the same failed repair | Limit attempts; require a new hypothesis before retrying. |
| Weakens tests to make them pass | Review the test diff; run independent acceptance checks. |
| Changes unrelated behavior | Scope allowed paths; inspect the complete diff. |
Prompts help communicate these rules. Enforce the important boundaries in tools, permissions and review.
Assign the reviewer and agree on the return format.
Stop filling the queue when review capacity is exhausted.
Inspect the integrated result, including interactions between patches.
More workers can move the queue downstream. Measure completed, accepted work.
| Need | Where it belongs |
|---|---|
| VAT rate comes from the order | Short project instruction. |
| How to verify an invoice export | Skill with cases and a reporting procedure. |
| Read the invoice specification ticket | An authorized CLI, API or MCP tool. |
A procedure teaches how to use access. It does not grant that access.
Procedure: inspect the diff → run the cases → report evidence and gaps.
Output: a small skill file and evidence that it is used when appropriate.
Example path: .claude/skills/invoice-check/SKILL.md. Check discovery in a fresh session.
The repeatable invoice-check procedure and supporting files.
A versioned way to distribute the procedure and supported integrations.
Test discovery and behavior in a clean session before sharing. Installation does not replace review or permission boundaries.
Tools exposed through a client/server protocol, with structured inputs and supported authentication paths.
Commands executed by a shell tool, with help, arguments, output and error handling.
Both can reach local or remote services. Both need appropriate credentials, permissions and review.
Measure the actual client configuration. There is no universal cheaper interface.
This is an experiment design, not a completed case study.
Use representative invoice cases and the same permissions and tools.
Record quality, elapsed time, all token categories, retries and review effort.
Repeat representative cases before changing the default configuration.
A higher-priced model can finish in fewer attempts. Whether it costs less is a measurement, not a rule.
Approximate weight storage: parameters × bits per weight ÷ 8. This is a lower-bound estimate, not a fit guarantee.
| Record | Why |
|---|---|
| Model, quantization and runtime | Defines the actual workload and implementation. |
| Context size and concurrency | Changes KV cache and memory demand. |
| Peak memory and elapsed time | Reveals fit and practical responsiveness. |
| Acceptance results | Shows whether it did the task correctly. |
No model download or hardware benchmark is required for this talk.
Output: a qualified feasibility note. Include KV cache, buffers, quantization overhead and speed.
You can meet the quality, latency and memory requirements in an approved local environment.
Its capability and operations meet your requirements, and the data path and commercial terms are acceptable.
Inspect the whole workflow: local inference does not stop tools or telemetry from using the network.
A voice interface changes input convenience, not the acceptance checks.
“Export the invoices without changing stored records.”
“Export the invoices, changing stored records.”
3 minutes: restore the missing boundary and state the acceptance check.
Output: one experiment plan with a clear owner and review date.
Primary documentation checked 3 October 2026. Examples are illustrative unless explicitly attributed.