Optional reference · Back to the talk
Optional referenceRevised October 2026

Measuring What Works
Reference.

Supporting procedures and worksheets. Return to the main talk

David Budáč
Optional reference · Back to the talk
2
In practice

Record what you are comparing

LayerRecord before running
ModelExact identifier, effort and generation settings.
HarnessVersion, prompts, tools, permissions and retry logic.
Routing serviceProvider selection, fallback behavior and data path.
Local runtimeQuantization, hardware, context and concurrency.

A changed harness can change the result even with the same model.

Optional reference · Back to the talk
3
In practice

The independence assumption matters

If each trial succeeds independently with probability p: P(at least one success in k trials) = 1 − (1 − p)^k P(all k trials succeed) = p^k For p = 0.8 and k = 3: 0.992 versus 0.512.

These are mathematical examples, not measured model results. Correlated failures can invalidate the simple model.

Optional reference · Back to the talk
4
In practice

Active parameters are not the RAM requirement

  • Stored weights depend on total parameters, representation and quantization.
  • Inference also needs a KV cache, activations, runtime buffers and overhead.
  • Offloading can move weights to CPU memory or disk, trading memory for transfer time.

Approximate weight storage: parameters × bits per weight ÷ 8. This is a lower-bound estimate, not a fit guarantee.

Optional reference · Back to the talk
5
In practice

Review dictated requirements before execution

Spoken: “Export invoices without changing stored records.” Check the transcription for: - negation: WITHOUT changing records - exact file names and amounts - scope and permission boundaries

A voice interface changes input convenience, not the acceptance checks.