Deep dive · advanced topics · ~30 minInternal talk · July 2026
WIP — draft

Measuring
What Works.

Evals and traces turn vibes into numbers — then use them to route across models and harnesses, and the daily habits that compound.

David Budáč
measure → route → habits
2
Agenda

Three parts, ~30 minutes

PartTopicNote
01Measuring what works: evals & tracesmost important for every engineer · live demo
02Alternative harnesses, models & APIslive demo
03Day-to-day tips & closelive demo

All pricing and product claims are date-stamped July 2026 and sourced — see the closing Sources slide.

01

Measuring what works: evals & traces

The most important part for every engineer in this room — evals are the core skill, not a nice-to-have.

3
Evals · why vibes don't scale

"Feels better" doesn't scale

  • One person's intuition about a prompt or CLAUDE.md change doesn't survive a team of ten making changes independently.
  • An eval turns a feeling into a number two people can agree on without a meeting.
  • This is the core AI skill for an engineer in 2026 — on par with writing tests.
the same CLAUDE.md change, judged two ways "feels better" "feels worse" "can't tell" ✕ a meeting — no way to settle it fixed task set real past work run both before / after one number a number settles what opinions can't — no meeting required
4
Evals · eval types

Task-completion suites

  • A fixed set of real tasks, each with a pass/fail or scored check.
  • Build it from actual past work — five to ten representative tasks beat fifty synthetic ones.
5
Evals · eval types

LLM-as-judge

  • For tasks with no clean pass/fail — writing quality, code style, clarity — score output against a written rubric using a second model.
  • Known failure mode: a judge tends to prefer its own family's outputs. Use a different model as judge than the one being evaluated.
scoring when there is no clean pass/fail output under test prose · code style · clarity written rubric the criteria, in prose judge model a different model family score 0–1 ✕ grading its own outputs ⚑ judges prefer their own family — never let a model grade itself
6
Evals · eval types

Pass@k on a fixed repo task set

  • Run the same task k times; check whether at least one attempt succeeds.
  • Accounts for run-to-run variance in agentic work — a single run can mislead either direction.
  • The mirror metric: pass^k — all k attempts succeed. Use it when you need reliability, not just best-case ability.
the same task, run 5 times ✗ ✗ ✓ ✗ ✗ pass@5 ✓ passes — one success is enough answers: can it do the task at all? pass^5 ✗ fails — every run must succeed answers: can you rely on it unattended? same five runs — two different questions
7
Evals · traces

Traces: capture the full transcript

  • Every tool call, every turn, every reasoning block — the raw material for analysis.
  • Also what turns a failed run from "it didn't work" into something debuggable.
8
Evals · traces

Reading traces for wasted turns

  • Look for: a file read twice, a plan restated repeatedly, a failed tool call retried blind.
  • This is where deck 05, Cost & Context's cost anatomy and context hygiene become actionable on a real session.
t1 read auth.ts t2 write the plan t3 read auth.ts — again ⚑ re-read t4 restate the plan ⚑ repeat t5 retry failed call, same args ⚑ blind retry t6 edit + tests pass ✓ every ⚑ is tokens paid twice — half this trace is waste
9
Evals · tooling

Tools: OTel & provider dashboards

  • OTel-based observability: capture agent traces the same way you'd capture microservice traces.
  • Claude Code ships OTel support out of the box — token, cost, and tool-usage metrics via standard exporters. Zero new vendors to start.
  • Provider dashboards: most model providers now ship usage/trace views out of the box — check what your org already has before adding a new tool.
agent traces land in tooling you already run Claude Code tokens · cost · tool usage OTel exporter built in your existing stack Grafana Datadog Kibana / Elastic model provider hosted API usage dashboard ships out of the box zero new vendors to start — check what your org already has
10
Evals · traces · anatomy

Anatomy of an agent trace: gen_ai.* spans

  • OTel GenAI semantic conventions — the emerging vendor-neutral standard for agent traces.
  • Every turn = one trace: invoke_agent wraps chat spans (LLM calls, token counts) and execute_tool spans (args, duration).
  • Spans are just data → token-spend dashboards, latency waterfalls, budget alerts ("fire above 256k tokens per conversation").
  • The wasted-turns reading becomes queryable: repeated spans = a re-read or blind retry you can alert on.
one conversation turn — the trace waterfall invoke_agent the whole turn chat (LLM) gen_ai.usage.input_tokens: 8 214 execute_tool · search args + result · 412 ms chat (LLM) output_tokens: 1 102 execute_tool · edit 96 ms chat (LLM) final answer width = time · every span carries gen_ai.* attributes

Worked example: Elastic Agent Builder emits this per turn by default — "a receipt for a single conversation turn" — queryable, dashboardable, alertable in Kibana.

11
Evals · the minimal loop

A minimal eval loop, this week

fixed prompts 5–10 real tasks run each variant score against a written rubric compare in a spreadsheet change ONE thing prompt, CLAUDE.md, or model — not two next cycle one variable per cycle — otherwise the number can't tell you what mattered

Deliberately unglamorous — the bar to start is nearly zero, no new tooling required.

12
Evals · smevals

smevals: the spreadsheet, grown up

Open-source Python framework — same loop as the spreadsheet, but repeatable and shareable.

Evala collection of Tasks measuring one capability
Configwhich model, parameters, prompts, tools
Taskone exercise the model must complete
Runimmutable record of one Task × one Config
Gradera sequence of Checks combined into a score
Runnerany executable — a model call or a full agent harness

Design detail worth stealing: a Runner that exits non-zero is a harness failure, not model evidence — infrastructure errors never pollute the scores. github.com/prime-radiant-inc/smevals

13
Evals · smevals

Three YAML files, three commands

# tasks/pelicans.yaml name: pelicans prompt: Write a haiku about pelicans. # graders/default.yaml checks: - checker: ../checkers/three-lines required: true
# repeat each task 5× → pass@5 / pass^5 $ smevals run my-eval -m claude-sonnet-5 -n 5 $ smevals grade my-eval $ smevals report my-eval # or: serve → live dashboard
  • Maps 1:1 onto the minimal loop — fixed prompts → task YAMLs, written rubric → grader, spreadsheet → report.
  • Checkers are plain executables — write domain-specific validators in anything.
14
Evals · smevals · how a run works

How a run works: env vars in, files out

Task (YAML) prompt + custom keys Config (YAML) model, params, runner env vars only SMEVALS_MODEL SMEVALS_PROMPT… Runner any executable — curl, script, or a full agent files out runs/<task>/<config>/<model>/<timestamp>/ output.txt ← runner stdout artifacts ← any file it writes run.yaml ← written last = complete immutable — never edited by hand exit code 0 — a real model response, however bad → gets graded, counts as evidence non-zero — harness broke (network, crash) → never graded, excluded from every report

No SDK to integrate — the contract is env vars in, files out, exit code as the verdict. That's why the thing under test can be a one-line curl or an entire Claude Code session.

15
Evals · smevals · grading & reporting

Grading: cheap gates first, judgment last

# graders/default.yaml — checks run in sequence checks: - checker: contains # cheap gate value: "<svg" required: true # fail → halt, skip the rest - checker: ../checkers/render-svg creates: render.png # no file → check fails - checker: ../checkers/llm-judge # expensive — runs last; scoring: # its score becomes THE score pass_threshold: 0.5
  • A Checker reads $SMEVALS_RUN_DIR/output.txt, exits 0/non-zero, optionally prints JSON: score 0–1, metrics, tags, notes.
  • Runs are immutable → re-grade old runs with a new rubric, no re-running.
  • -n 5 tops each task × model up to 5 successful runs — pass@k built in. Leaderboard: mean ± stderr per config × model.
  • smevals serve — live dashboard, re-reads disk on every poll; watch runs land in real time.
16
Evals · live demo
Live

Run a small eval, live

  • A small fixed task set (3–5 tasks), two variants — e.g. the two CLAUDE.md variants from deck 05, or two models on the identical prompt.
  • Score each run against a written rubric, tally in a spreadsheet on screen.

Live Same method as the minimal loop — no special tooling, just discipline.

17
Evals · fallback

Fallback: recorded run

▢
Placeholder — screenshot of the scored spreadsheet comparing two variants goes here.

Use only if a live run stalls or scoring runs long. Narrate the rubric scores and the winning variant from rehearsal.

02

Alternative harnesses, models & APIs

Why not monoculture: route by cost, intelligence, and taste — the same discipline at every layer.

18
Alt. harnesses · why not monoculture

Route by cost, intelligence, taste

  • No single model or harness wins on all three axes at once.
  • Cheap and fast for bulk, mechanical work; smarter models for hard unsupervised problems; highest-taste models only where UI/copy quality is the actual bottleneck.
A cheap & fast B workhorse C frontier cheaper → C B A smarter → A B C finer taste → A B C

No column sweeps all three rows — route per task. This is the same discipline at every layer: model choice, harness choice, even open-vs-hosted choice.

19
Alt. harnesses · Cursor

Cursor as a harness

  • IDE-native, not terminal-native — strongest for engineers who live in an editor rather than a terminal.
  • Auto mode and Cursor Router (deck 05, Cost & Context) route per-request across models automatically.
  • Beats a terminal agent when routing quality and IDE integration matter more than terminal-native workflows.
20
Alt. harnesses · OpenRouter

OpenRouter: one API, many models

  • 315+ models from Anthropic, OpenAI, Google, DeepSeek, Meta, Mistral, xAI, and others — one key, one bill.
  • Pay-per-token; roughly a 5.5% fee on credit purchases rather than a heavy per-token markup on most models.
  • Caveat: failover is a feature, but it means a request can silently route to a different provider than expected.
your app one key · one bill OpenRouter 315+ models ≈5.5% fee on credit purchases, not a heavy per-token markup Anthropic OpenAI Google DeepSeek Meta Mistral · xAI · … ⚑ failover can silently land a request on a different provider

Some trackers cite a "30-60% markup in some cases" — that figure is inconsistent across sources; the 5.5% credit-fee framing is the more defensible one.

21
Alt. harnesses · Pi

Pi: a minimal harness

  • Terminal coding-agent harness by Mario Zechner (now Earendil Inc) — system prompt under ~1,000 tokens.
  • Supports 15+ providers/hundreds of models, plus local models via Ollama for zero-cost inference.
  • Tradeoff: using Claude models through Pi bills at raw API rates — a Claude Max subscriber effectively pays twice.
22
Alt. harnesses · open models, July 2026

The frontier gap is closing — for coding

  • Several open-weight releases in 2026 close in on frontier coding-benchmark scores at a fraction of the parameter count or cost.
  • Closing, not closed — the next slides give specific, verified examples rather than the trend in the abstract.
23
Alt. harnesses · Gemma 4

Gemma 4: laptop-viable

  • Google, arrived ~March 2026. Sizes: E2B, E4B, a 12B unified multimodal (June 2026), a 26B MoE (3.8B active), a 31B dense.
  • The 12B model runs on a 16GB laptop (e.g. 16GB MacBook Air) at Q4_0 quantization — nearly matching the 26B model at under half the memory.
the Gemma 4 family — bar height = total params fits a 16GB laptop @ Q4_0 ≈ same benchmarks, under half the memory E2B E4B 12B 26B 31B multimodal MoE · 3.8B active dense the coral bar is the laptop pick — local inference, not aspirational
24
Alt. harnesses · Qwen 3.6

Qwen 3.6: dense and MoE flagships

  • Alibaba, Apache 2.0. Two April 2026 flagships: 27B dense and 35B-A3B MoE (only 3B active params/token).
  • Native 256K context, extensible to ~1M via YaRN; reportedly outperforms a 397B MoE model on agentic coding benchmarks.
  • The 35B-A3B MoE is the realistic local-laptop pick given its low active-parameter count. [exact laptop RAM requirements not independently verified]
what one token pays for, per flagship 27B dense all 27B active every token pays full compute 35B-A3B MoE ← 3B active / token the other 32B sit parked RAM for 35B, compute for 3B context window 256K extensible to ~1M via YaRN low active params make the A3B the realistic local-laptop pick
25
Alt. harnesses · live demo
Live

Local models, small hardware

  • Gemma 4 (12B) or Qwen 3.6 (35B-A3B) running locally on the presenter's own laptop via Ollama.
  • A real coding task from this team's repo, run to completion on screen.

Live Message: absolutely viable even on modest hardware — not "as good as the frontier."

26
Alt. harnesses · fallback

Fallback: recorded run

▢
Placeholder — screenshot or recording of a local Gemma 4 / Qwen 3.6 run completing a real coding task goes here.

Use only if the live run is too slow to finish on stage. Keep the honest framing: slower than hosted, genuinely usable.

27
Alt. harnesses · Colibri + GLM-5.2

Colibri: GLM-5.2 without a GPU

  • GLM-5.2 (Z.ai): open-weight MoE, ~753B total params (~40B active), 1M context. Reportedly matches/beats GPT-5.5 on several long-horizon coding tasks. [total param count varies 744B–753B across sources]
  • Colibri: a ~2,400-line pure-C inference engine — keeps ~17B dense params in RAM (int4), streams routed experts from disk via an LRU cache.
  • Runs on ordinary hardware, no GPU, about 25GB RAM. Speed: ~0.05–1 tokens/sec — value is running it at all, not fast.
SSD all routed experts the bulk of ~753B params never all in memory per token: only the routed few RAM — ~25GB total dense core ~17B · int4 · always resident hot experts — LRU cache no GPU anywhere · ~0.05–1 tok/s — the value is that it runs at all
28
Alt. harnesses · when local makes sense

When local makes sense — and when it doesn't

Makes sense

Privacy-sensitive or air-gapped work, driving marginal cost toward zero, latency requirements loose enough to tolerate it.

Doesn't

Need frontier quality or fast turnaround; hardware/ops overhead outweighs the savings.

03

Day-to-day tips & closing

The habits that compound daily — then the recap, then the sources.

29
Day-to-day · voice mode
Live

Voice mode: dictate, don't type

  • Claude Code shipped native /voice dictation ~March 2026 — Hold (default) and Tap modes, tuned for coding vocabulary (regex, OAuth, JSON, localhost).
  • Hands-free review: dictate a prompt, keep typing while speaking, review output without touching the keyboard.

Live "Hlas" (Czech voice tooling per earlier notes) [TODO: verify tool name — could not confirm as a real product; may be a generic reference]. Demo Claude Code's own /voice command instead, which is solidly verified.

30
Day-to-day · fallback

Fallback: recorded run

▢
Placeholder — screenshot or recording of Claude Code's /voice command dictating and executing a real prompt goes here.

Use only if the venue's microphone makes live dictation unreliable.

31
Day-to-day · habits

Habits that compound daily

  • Keyboard-first — fewer context switches to the mouse.
  • Parallel sessions on unrelated tasks instead of waiting serially.
  • Background/async tasks — kick off, check back later.
  • Checkpoint with git — small commits so an agent's mid-task state is never the only copy.
small wins × every day keyboard-first parallel sessions background tasks git checkpoints each habit buys time the next one spends — the wheel spins faster
32
Day-to-day · recap

Measure → route → habits

01 measure
→
02 route
→
03 habits
  • Measure first. Five real tasks and a written rubric beat any amount of prompt debate — start this week, in a spreadsheet.
  • Route by the numbers. No model or harness wins on cost, intelligence, and taste at once — pick per task, not by loyalty.
  • Habits compound. Voice, parallel sessions, background tasks, git checkpoints — small daily wins that add up.

Act on measurement (part 01) first — without evals, the routing choices in part 02 are just a more expensive set of vibes.

33
Close · sources

Sources — all claims dated July 2026

Evals & tracing

smevals — eval frameworkgithub.com/prime-radiant-inc/smevals OTel tracing for AI agents (Elastic)elastic.co/search-labs

Harnesses & routing

Cursor pricing guide 2026nxcode.io OpenRouteropenrouter.ai Ollama — local inferenceollama.com

Models

Qwen 3.6github.com/QwenLM/Qwen3.6 NIST CAISI assessment of GLM-5.2nist.gov

Full source list with per-claim citations: deck05-facts.md (internal research doc). One item flagged unverified: "Hlas" as a named product.