The short version: what the pipeline does for you, what it needs from you, and the handful of jobs that are actually yours. Every job here links to the page that covers it properly.
You have millions of Oracle alert, listener and dataguard events in Elasticsearch and no time to read them. This pipeline reads them for you every two hours, decides deterministically whether anything happened, and only then asks an LLM to write it up — as a journal line, an incident, an error-class page, a fleet report — into a git-versioned markdown wiki you can actually read.
Deterministic Python decides what happened; a model is only woken to say what it means.
uv sync # python 3.13+, thin deps
export DBWIKI_ES_PASSWORD=... # never commit a real password
uv run dbwiki health # is everything it needs actually up?
uv run dbwiki dbs # which databases does ES know about?
uv run dbwiki run --explain # what would a tick do right now, and why?
uv run dbwiki ingest --db cdb1 --date 2026-08-29 --dry-run # see the prompt, spend nothing
Only compact (which writes digests) and the mutating stages touch disk; every command above is safe to run on a live deployment. Full setup, config and the command list: user-guide.md.
Never hand-edit wiki/digests/ or wiki/html/. Machine output; your edit is overwritten and an agent that touches them is rolled back.
Never leave the wiki tree dirty. Uncommitted changes outside those two directories abort the next agent run (dirty_tree) — because a failed run rolls the tree back and would take your edits with it.
wiki/sources/ pages enter by human commit only. Research may cite an approved source's domains; it may not approve one.
Silence is not recovery. Nothing auto-closes an incident; recovery needs a resumed flow or your note in .state/recovery_annotation.txt.
How to run this thing day to day: setup, the commands you will actually type, recipes for common jobs, and the rules a human sharing the wiki repo has to respect. For what happens under the hood see how-it-works.md; for keeping it alive see monitoring.md.
uv sync # Python 3.13+, thin deps
cp config/dbwiki.yaml.example config/dbwiki.yaml # or edit in place
export DBWIKI_ES_PASSWORD=... # never commit a real password
config/dbwiki.yaml is your deployment's file and is not meant to be committed; the example ships the docker-elk lab default password. Set DBWIKI_ES_PASSWORD in your shell and in the crontab's environment.
Optional extras:
pi install npm:pi-web-access # only if research runs agentic through pi
What must exist on PATH: uv, git, a reachable Elasticsearch, and the adapters you actually configured (codex and/or claude and/or pi). Cron has none of your shell's PATH — see "Installing the schedule" below.
The wiki lives in wiki/, a separate git repo. If it is missing, every mutating command fails with wiki_missing; clone it back beside config/ (or start an empty one from wiki-template/, see its README).
# compaction (no LLM, no lock, free)
uv run dbwiki compact --db cdb1 --date 2026-08-29 # one daily digest
uv run dbwiki compact --all --date 2026-08-29 # every database
uv run dbwiki backfill --from 2025-08-24 --to 2026-07-11
uv run dbwiki dbs # discover databases
# the loop
uv run dbwiki run # one adaptive tick
uv run dbwiki run --consolidate # force the daily report
uv run dbwiki run --explain [--json] # why would each db wake? (read-only)
# individual stages
uv run dbwiki ingest --db cdb1 --date 2026-08-29 [--dry-run] [--adapter pi]
uv run dbwiki report --date 2026-08-29
uv run dbwiki lint # deterministic lint + LLM health check
uv run dbwiki lint --deterministic-only [--json] # mechanical rules only, no LLM
uv run dbwiki research [--limit 3] [--dry-run]
uv run dbwiki research --sources-only --review-sources
uv run dbwiki research --caveats [--limit 2] [--dry-run] # practitioner notes from the approved sources (web)
uv run dbwiki review [--explain] [--force] # weekly attention review (--explain: read-only)
uv run dbwiki review --deliver-only [--review-id 2026-W38] # resend a week's review, skipping what already went
uv run dbwiki render-daily [--day 2026-08-29]
# the incident workbench in a browser
uv run dbwiki portal serve [--bind HOST:PORT] # http://127.0.0.1:8765
# incidents from the command line (preview by default; --commit publishes)
uv run dbwiki incident list [--all] # open incidents, or all of them
uv run dbwiki incident show <slug> # status, window, action log
uv run dbwiki incident record-action <slug> --intent ... --summary ...
uv run dbwiki incident monitor <slug> --signal error_absent:ORA-00600 \
--until 2026-09-01T00:00:00Z --intent ... --summary ...
uv run dbwiki incident extend <slug> --until ... --intent ...
uv run dbwiki incident resolve <slug> --summary ...
uv run dbwiki incident reopen <slug> --reason ...
# operations
uv run dbwiki health [--json] [--alert]
uv run dbwiki retry [--dry-run] [--db cdb1]
uv run dbwiki stats [--json] [--task ingest] [--since 12h|2026-09-23] [--until ...] [--by model]
uv run dbwiki awr --file awr.json [--emit]
uv run dbwiki redact errors/ORA-12154.md [--all|--candidates|--source <slug>]
# drill-down into the raw evidence
uv run dbwiki es search --source alert --db cdb1 \
--from 2026-08-29T00:00:00Z --to 2026-08-30T00:00:00Z --query 'ORA-*'
uv run dbwiki es get --index .ds-logs-oracle.alert-default-2026.08.20-000008 --id <docid>
uv run dbwiki es trace --path /u01/app/oracle/diag/rdbms/cdb1/cdb1/trace/cdb1_ora_7903.trc
# ADR-0001 / ADR-0002 (both off here)
uv run dbwiki analyst --once
uv run dbwiki-researcher --once|--watch
Anything that mutates the wiki takes the single-flight lock and fails fast if another command holds it. Add --lock-wait SECONDS (or set DBWIKI_LOCK_WAIT) when you would rather queue than fail.
Every mutating incident subcommand previews by default: it prints the diff, writes nothing, and publishes only when you repeat it with --commit, which lands the whole change as one git commit. The preview never takes that lock, so it is safe to run while a tick is working. Only --commit takes it. The five mutating subcommands share --actor EMAIL, --base SHA (a full sha, defaulting to the wiki's HEAD), --at ISO_Z (defaulting to now), --commit, --notes-file PATH, and --lock-wait SECONDS. A slug is the incident page's name, 2026-08-05-cdb1-something. incidents/<slug>.md and a full path to the page are accepted too.
The fields differ per subcommand:
record-action needs --intent and --summary, and also takes --ticket, --outcome, --rollback, --evidence, and --notes.
monitor needs --signal, --until, --intent, and --summary, and also takes --ticket, --evidence, and --notes.
extend needs --until and --intent, and also takes --notes.
resolve needs --summary, and also takes --residual-risk, --ticket, --evidence, --notes, and --no-error-pages.
reopen needs --reason, and also takes --evidence and --notes.
--evidence repeats, once per page cited. --signal is KIND:PAYLOAD split on the first colon, and the kinds are error_absent:CODE, event_present:REGEX, flow_resumed:SOURCE, and manual:TEXT.
Exit codes: 0 published or nothing to do, 1 the wiki lock is busy or the preview has blocking lint findings, 2 the command is not legal from the incident's current status (or the text or the actor is unusable), 3 the base moved, 4 lint blocked the commit, 5 the wiki tree is dirty. Every --commit also writes one run-health event under command incident, naming the incident, the subcommand, the actor, the base, the at, and the resulting sha.
dbwiki portal serve puts a browser in front of the same incident lifecycle the dbwiki incident subcommands drive. It is the normal way to record an action and resolve an incident. The CLI stays the fallback, for when the service is down and for anything you script.
Configure it with a portal: block in config/dbwiki.yaml. Every key has a default, but dbwiki health reports the workbench only when the block is there, so write it out even if you change nothing:
portal:
bind: 127.0.0.1:8765 # a non-loopback --bind is allowed but warned about
operator_email: "" # empty -> the wiki repo's git config user.email
lock_wait_s: 3.0 # a commit's wait for the wiki lock before it answers 423
push: true # follows report.push unless you set it here
allowed_hosts: [] # extra Host/Origin names beyond the bind and loopback
trusted_proxies: [] # proxy IPs whose X-Forwarded-Host is checked instead
Those are the defaults, except push, which has no default of its own and follows report.push.
The server answers only requests whose Host names the bind or a loopback spelling of its port (127.0.0.1, localhost, [::1]); anything else is 403 bad_host. That refusal is the protection against DNS rebinding, so keep it narrow. On loopback, leave allowed_hosts empty. When browsers reach the workbench by another name (a --bind 0.0.0.0:8765 on the LAN, a Tailscale name, a reverse proxy), list the names they use:
A bare name (dbhost) matches on any port; name:port on that port only. A TLS proxy's browser sends Origin: https://wiki.example.com, which is wiki.example.com:443.
Names compare without regard to case. *, URLs and other junk are refused when the config is read, so the server does not start.
Origin, when a browser sends one, must be http(s):// plus an allowed name.
trusted_proxies is for a proxy that rewrites Host to 127.0.0.1:8765: from a peer on that list (IP addresses), the first X-Forwarded-Host is the name checked in place of Host. From any other peer the header is ignored. The proxy must still authenticate users; the portal does not. serve warns on stderr when it binds a non-loopback address with no allowed_hosts.
The advisory tools are a separate top-level block, because they are not the portal's: a run is a file under .state/, and its ceiling is read from the same ledger dbwiki health writes. Every key has a default, and the whole block may be absent:
advisory:
enabled: true # false turns every tool off; the panel says so
max_concurrent: 2 # model calls this process holds at once
retain: 200 # run files kept, newest first
window_h: 24 # the spend window every row inherits
max_runs: 20 # runs per row per window
max_cost_usd: null # null is a count ceiling only
timeout_s: 120 # per call; never agents.timeout_seconds
tier: cheap # which agents.pi tier a row resolves its model from
tools: # the same keys again, per row, overriding the above
explain-incident: {tier: strong}
draft-note: {enabled: false}
An unknown tool id or an unknown tier is refused when the config is read, so the server does not start rather than failing on an operator's first click.
Setting max_cost_usd means what it says: a window holding a run no adapter priced refuses the next click with cost_unknown rather than treating that run as free. Leave it null until you have read what a click actually costs out of .state/advisory_runs.jsonl.
Where the weekly review goes is a third top-level block. It is separate because it is not the portal's either: dbwiki review runs from cron and writes its own audit. Every key has a default, and the whole block may be absent:
delivery:
enabled: true # false delivers nothing at all
external_enabled: false # the switch that lets anything leave the box
smtp_host: "" # the relay
smtp_port: 25
starttls: false # true: STARTTLS, verifying the relay's certificate
from_address: "" # the envelope sender
subject_prefix: "[dbwiki] "
timeout_s: 10.0 # per send
portal_base: "" # the base URL a body links back to
recipients: [] # one row per person told
Those are the defaults. external_enabled: false means nothing leaves the box: the review is published to .state/review/, the inbox screen reads it there, and no mail is sent. Leave portal_base empty unless the portal is reachable at an address other than loopback. A body with no portal_base carries no link rather than one into the reader's own 127.0.0.1.
A recipient row is {id, address, channels, content_class, role}. The id is yours to pick and is what the audit keys on, so changing it re-sends. channels is [inbox], [email], or both. The one content class is internal_full, which sends the whole review to someone already trusted with the wiki.
The config is refused when it is read, and the refusal names the key. An unknown content_class, an unknown channel, an unknown role, a missing or duplicate recipient id, the reserved id inbox (the inbox row's key), and an empty address on a recipient that names the email channel are all refusals. So is external_enabled: true with an empty smtp_host, with an empty from_address, or with no recipient naming the email channel, which would be delivery configured to reach nobody.
Sending is idempotent, keyed on the review and the recipient. A relay that is down costs the review nothing: the failure is recorded in .state/review/log.jsonl, the command exits 1 naming the retry, dbwiki health lists it under "review delivery" (so alerts fire) until it goes, and any later dbwiki review in the same week — or uv run dbwiki review --deliver-only --review-id YYYY-Www for an earlier one — resends exactly what is missing. A failure stops being reported once its review is pruned or its recipient is no longer configured.
Start the workbench by hand:
uv run dbwiki portal serve
On the cron host it runs as a systemd user service instead. deploy/README.md has the unit and the install steps.
Then open http://127.0.0.1:8765. Use that address rather than localhost. The server binds 127.0.0.1, and a browser that resolves localhost to ::1 fails to connect before any check runs.
The Links tab is your own board of everything the pipeline runs on: Kibana dashboards, Langfuse, the services on the box, the repositories. It is config/links.yaml, a title, an intro and a list of sections, each holding links of {name, what, url, tag} with tag one of tailscale, github or artifact. Backticks in intro, note and what are code spans. The file is optional; without it the tab says where to write one. A url that is not http or https, a link with no name or no url, and an unknown tag are all refused when the file is read, and the refusal names the link. The board is read once, when the server starts, so restart the portal after editing it.
The flow, start to finish:
The queue lists the open incidents, in the order the fleet report puts them in.
Open one. You get its status, its action log, the monitoring facts behind it and the page's git history, and only the verbs that are legal from its current status.
Pick a verb and fill in the form. The fields are the same ones the CLI flags set.
Read the diff. You get the unified diff the commit would make, any lint findings, any uncommitted wiki changes the commit does not own, and the dbwiki incident ... line that would publish the same thing from a terminal. Nothing has been written yet.
Publish. The whole change lands as one git commit, the same one --commit would make, and writes the same run-health event with surface: portal instead of cli.
The incident screen also carries a panel of advisory tools. Opening it asks the server what each tool would read and what its window has left to spend; that answer costs no model call, so you see the size of the material and the state of the ceiling before you spend anything. Running one is the only button on this workbench that costs money. An answer is a proposal: it is labelled as one, it cites only paths the material actually carried, it says what it cost, and nothing about it reaches a commit unless you edit it into a field and publish it yourself.
The Agents tab answers what the loops themselves did. Pick 24, 48 or 168 hours. The table at the top rolls the window up by model: how many stages each ran, how many passed validation and how many did not, the tokens and the wall time, each sum beside the count of lines that reported it. Below it is one lane per tick, drawn against the window's clock, with a bar per stage coloured by the model that ran it and outlined in red where the stage failed or was rolled back; hover a bar for the task, the database, the tokens and the duration. Beside each lane are the incident pages that tick committed, and each one is a link to the case. The join behind the lanes is the Run-ID: trailer every machine commit carries, which is the same id .state/agent_runs.jsonl keys every stage by; a commit without one is a hand edit and belongs to no tick. The incident screen reads the same join the other way, under Which ticks wrote this page: every tick that has written the page in the last 30 days, its stages, and a link to the run. Both halves say what bounds them — the ledger's caps above the table, the wiki revision in the strip — because an absence can come from either.
The four refusals, and what each means for what you do next.
409 means the wiki moved under you. Three causes. A tick committed between your preview and your publish, so nothing was written and you re-preview. Someone has an uncommitted edit in wiki/, which has to be committed or restored first. Or the incident page itself has never been committed, which you meet when you open it rather than when you publish. Commit the page with the CLI, or wait for the next tick to commit it.
422 means the workbench refused the content. The lint blocked the diff, or the verb is not legal from the incident's current status, or a field is missing or malformed, or a resolve arrived with an empty residual-risk paragraph.
423 means another command holds the wiki lock. Almost always the two-hourly tick. Nothing was written, nothing failed, and dbwiki health stays green. Wait a moment and click again.
429 means an advisory tool declined to spend. Either every slot is busy, in which case the run holding one is on the panel with its own status, or the row has reached a ceiling for its window. Nothing was spent and no evidence was read either way.
Read-only, no side effects, and it uses exactly the code path the real tick does. Typical answers: content_unchanged (nothing new since the last ingest), routine_only (nothing worth waking a model for), or a wake line listing the deltas and notable groups that would trigger it.
uv run dbwiki retry --dry-run # what would be replayed, and why not
uv run dbwiki retry # replay the retryable ones
uv run dbwiki retry --db cdb1
Non-destructive: it only replays digests that still exist, whose content hash still matches, and whose failure a rerun can fix. Fix the cause first (a down model server, a missing adapter) — see monitoring.md.
"Bootstrap history / re-derive digests."
uv run dbwiki backfill --from 2025-08-24 --to 2026-07-11
Compaction only, and idempotent — it never calls a model. Ingesting that history is a separate, chronological pass.
"Re-do one day for one database."
uv run dbwiki compact --db cdb1 --date 2026-08-29
uv run dbwiki ingest --db cdb1 --date 2026-08-29
Re-ingesting a day rewrites that day's journal entry, incident update and occurrence row rather than duplicating them.
"Show me the prompt before spending a model on it."
uv run dbwiki ingest --db cdb1 --date 2026-08-29 --dry-run
uv run dbwiki research --dry-run
"What is this ORA code, and have we seen it before?"
Look at wiki/errors/<CODE>.md — occurrences across databases, contexts, what fixed it last time, and a ## Reference section if research has run. If the page has no researched: date, it is already a candidate for the next weekly research run; force one page sooner with uv run dbwiki research --limit 1 after checking --dry-run.
"Record a remediation and resolve an incident."
uv run dbwiki incident show 2026-08-05-cdb1_stby-tns-12564-ora-16603
uv run dbwiki incident record-action 2026-08-05-cdb1_stby-tns-12564-ora-16603 \
--intent 'restart the standby listener' \
--summary 'restarted LISTENER on cdb1_stby; TNS-12564 stopped' \
--evidence digests/cdb1_stby/2026-08-05.md
# read the diff, then publish the exact line it prints:
uv run dbwiki incident record-action 2026-08-05-cdb1_stby-tns-12564-ora-16603 \
... --base <sha> --at <timestamp> --commit
uv run dbwiki incident resolve 2026-08-05-cdb1_stby-tns-12564-ora-16603 \
--summary 'listener restarted; no TNS-12564 for 48h' --commit
The preview prints the unified diff, any lint findings, any uncommitted wiki changes it does not own, and last a preview only. publish with: line that repeats the whole invocation with --base and --at filled in. Copy that line to publish. Pinning --at is what makes a retry converge instead of writing a second action record.
A base moved refusal (exit 3) means the wiki's HEAD moved between the preview and the commit, usually because a cron tick committed. Nothing reached the wiki. Re-preview against the new base and publish again.
resolve does two more things. It moves the incident's bullet from ## Open incidents to ## Resolved incidents in index.md, and it writes a ## Resolution history row into every errors/<CODE>.md the incident links. --no-error-pages suppresses the rows.
The incident workbench above is the normal way to do all of this. The CLI is the fallback, for when the service is down and for anything you script.
"Add a new external source."
Source pages are human-committed only — an agent that creates one gets rolled back. Write wiki/sources/<slug>.md with status: approved, tier, domains, fetchable, last_reviewed, review_after_days, commit it in the wiki repo, and research may cite those domains from then on.
"Check what would leave the box before enabling research offload."
uv run dbwiki redact errors/ORA-12154.md # one page
uv run dbwiki redact --candidates # everything the next run would send
Prints the exact anonymized request JSON (pseudonym mapping on stderr). Nothing is sent.
"Feed an AWR summary in."
uv run dbwiki awr --file awr.json # validate + print the digest
uv run dbwiki awr --file awr.json --emit # write it beside the compactor's digests
Contract v1 is files-only (no live Oracle) and carries sql_ids, never SQL text — see awr-input.md.
Every step down is closer to the evidence; the last one is the digest every claim above cites.
wiki/html/index.html — the daily DBA page. Needs attention first (notable databases and every still-open incident), then Worth a look (first-ever codes, new services/programs, rate anomalies, silence — always telemetry, never asserted as database state), then a collapsed Routine table. Links resolve on the wiki's git remote, where markdown renders.
wiki/reports/<day>.md — the fleet report: what happened, with historical context ("last seen 2026-03-14, fixed then by …").
wiki/databases/<db>.md and databases/<db>/journal/<YYYY-MM>.md — the profile and the day-by-day record.
Never hand-edit digests/ or html/. They are machine output; agents are rolled back for touching them and your edit will be overwritten.
Never leave the wiki tree dirty. Any uncommitted change outside digests//html/ aborts the next agent run (dirty_tree) — commit or stash before the next tick. Editing wiki pages by hand is fine; leaving them uncommitted is not.
sources/ pages enter by human commit only.
Do not close an incident because the events stopped. Recovery needs explicit evidence — a recovery event, resumed expected flow, or an operator note in .state/recovery_annotation.txt.
Both repos push independently: the machinery repo is yours, the wiki repo is pushed by the pipeline when report.push: true.
Edit the tier under agents.<adapter> (or agents.pi for structured stages), or flip agents.mode / agents.escalated_report / research.mode.
uv run dbwiki health — the dependencies block verifies the adapter binary is on PATH and, when any route uses pi, that the local model server is up with the configured model at a usable context.
Try one stage by hand before trusting cron: dbwiki ingest --db <db> --date <day> or dbwiki report --date <day>.
uv run dbwiki stats after a few days: acceptance rate, rollback rate, median duration and cost per (task, adapter, tier). No verdict is offered until both tiers have ≥5 runs.
Record the measurement in CHANGELOG.md — that is where the reasoning behind the current settings lives.
The local structured stages need the model server running (unsloth start — it does not come back by itself after a reboot) and serving a context large enough for an escalated prompt; health.min_context floors that at 32k.
The crontab is the authority; docs/scheduling.md is its documentation and config/schedule.json its machine-readable mirror (used by the ELK emitter — update it whenever the crontab changes, nothing enforces the sync).
PATH=/usr/local/bin:/usr/bin:/bin
15 */2 * * * cd <repo> && uv run dbwiki run >> .state/cron.log 2>&1
30 23 * * * cd <repo> && uv run dbwiki run --consolidate >> .state/cron.log 2>&1
0 8 * * 1 cd <repo> && uv run dbwiki lint >> .state/cron.log 2>&1
0 9 * * 1 cd <repo> && uv run dbwiki research >> .state/cron.log 2>&1
0 9 1 * * cd <repo> && uv run dbwiki research --sources-only --review-sources >> .state/cron.log 2>&1
30 9 * * 1 cd <repo> && uv run dbwiki research --caveats >> .state/cron.log 2>&1
0 10 * * 1 cd <repo> && uv run dbwiki review >> .state/cron.log 2>&1
*/30 * * * * cd <repo> && python3 elk/scripts/emit_derived.py >> .state/elk/emitter.log 2>&1
The explicit PATH is load-bearing: cron's default has neither uv nor codex (linuxbrew), and agentic stages then fail only when unattended. Add --lock-wait 3600 to the mutating entries if you would rather a tick waited out a slow agent than skipped itself. Successful runs are silent; anything that appears in .state/cron.log is worth reading.
Recovery
Monitoring and recovery
What to look at, what each signal means, and the exact command for every way this pipeline is known to break. Companion to user-guide.md; the schedule itself is in scheduling.md.
uv run dbwiki health # exit 0 healthy · 1 failure/blocker · 2 state unusable
That one command reads the run-health log, the ingest ledger, the watermarks, the wiki tree, the analyst/exchange queues and one read-only ES probe per source, and prints every failure with the exact recovery command underneath it (-> …). If it exits 0, nothing else needs looking at.
The health output names the cause and the consequence separately: fix the cause, then replay the consequence.
If it exits non-zero, work top-down through the blocks below.
pipeline health 2026-08-29T20:49:23Z [UNHEALTHY] stale_hours=26
collection ← is data still arriving at all?
alert ok last event 2026-08-29T17:13:26Z [3.6h ago]
listener ok last event 2026-08-29T17:13:46Z [3.6h ago]
dataguard ok last event 2026-08-29T17:13:26Z [3.6h ago]
recovery evidence: unknown — absence of events is not recovery
last success ← is every stage still completing?
compaction 2026-08-29T20:16:15Z (from run) awrwh, cdb1, cdb1_stby, …
ingestion 2026-08-29T16:38:13Z (from run) cdb1, cdb1_stby, dgnonc, dgnonc_s
report 2026-08-29T16:38:13Z (from run) reports/2026-08-29-1615.md
lint 2026-08-24T06:05:49Z (from lint) 2 finding(s)
research 2026-08-29T13:56:35Z (from research) 3
dependencies ← can the next run even work?
model svr UNREACHABLE unsloth http://127.0.0.1:8888 [served ctx unknown]
adapters codex=ok git=ok pi=ok
wiki remote push=on upstream=origin/main unpushed=0 user.email=…
disk 60.38 GiB free under …/.state
PROBLEM model_server: the local model server is not reachable …
-> start the local model server (unsloth studio: `unsloth start`); …
watermarks ← per-db progress
cdb1 2026-08-29T20:15:00Z [0.6h]
…
failed ingests ← what a rerun could fix
digests/cdb1/2026-08-29.json harness_error [retryable] digest=present …
-> dbwiki retry --db cdb1
digest backlog ← compacted but never ingested
(none)
analyst queue ← ADR-0001 backlog (0 when unused)
pending: 0
blockers ← state that stops work entirely
(none)
The example above is the single most common real failure: the local model server was down, so every structured ingest failed as harness_error. Note how it reads — the dependency block names the cause, the failed ingests block names the consequence. Fix the cause, then replay the consequence.
Three distinctions the tool makes deliberately, and you should too:
source silence (this source is quiet, another one is live) vs collection failure (nothing recent anywhere — the shipper died) vs unknown (the probe itself failed). They are different problems.
failure vs blocker: a failure is something that went wrong and can be retried; a blocker is state that stops work until a human intervenes.
recovery is evidence, never absence. It is only ever reported as resumed_flow (events after a gap), operator_annotation (a note you put in .state/recovery_annotation.txt) or unknown. Nothing auto-closes an incident.
Stage staleness is judged against health.stage_stale_hours — run, report and research_history 26h, lint, research, research_caveats and review 192h by default. A stage that has never succeeded is never flagged. Every dbwiki research variant records as research; health keys them by mode, so the nightly research --history succeeding no longer hides a weekly agentic research that keeps failing (dry runs count for nothing). A run that finds nothing due (no stale error page, no source due review, no page due a caveat pass) still records an ok run of its mode, so a fully researched wiki does not go stale. The workbench's runs screen shows the same three research rows.
A weekly-review email whose last attempt failed is a review delivery failed problem (alert category delivery_failed, one per review) until dbwiki review --deliver-only --review-id YYYY-Www delivers it; see docs/user-guide.md for delivery.
A source's newest event is read up to now + 15h only (covers a shipper that labels local time as UTC). Events dated further ahead are ignored for collection state and reported as a future-dated events problem — one mis-parsed line dated 2099 used to keep a dead source ok forever.
1. Local model server down (structured stages fail in a block). Symptom: a run of harness_error ingests, model svr UNREACHABLE in the dependencies block. It does not restart after a reboot and there is no @reboot entry for it on purpose.
unsloth status
unsloth start
uv run dbwiki health # model svr should now report a served context
uv run dbwiki retry # replay the digests that failed meanwhile
2. Served context too small. Same block, [served ctx …] below health.min_context (32k). An escalated prompt is ~8.6k input tokens and a reasoning model spends thousands more thinking; at 8k it stops on length before writing a character. Historically both failures surfaced as a confusing schema_version: expected 1, got None; the adapter now says provider unreachable: … or hit its context limit … instead.
Since 2026-09-08 this repairs itself. The watchdog runs --fix from cron every 10 minutes, and a configured model loaded under health.min_context (or not loaded at all) is reloaded through the studio's load API at 98176, then re-read from the server's own model listing before the exit code is decided. A healthy server is never reloaded.
tail -n 40 .state/watchdog.log # the last round's report; `fix` says what it did
grep -c '"reloaded": true' .state/watchdog.log # how often the studio came back small
python3 n8n/scripts/watchdog.py --fix # force a repair now
uv run dbwiki retry # replay the digests that failed meanwhile
fix.reloaded false with verdict not_served or unreachable means the watchdog cannot help and item 1 applies. The report is indented JSON, one object per round, so read it with tail -n 40 rather than a per-line tool.
3. codex missing from cron's PATH. Agentic stages (notable reports, lint) die with FileNotFoundError: 'codex'only when unattended, because your interactive shell has linuxbrew on PATH and cron does not. Fix the PATH= line in the crontab; dbwiki health's adapters line verifies it.
4. Collection failure — the feed stopped.collection shows collection_failure (nothing recent anywhere) rather than source_silent (one quiet source). The fix is upstream: restart the filebeat shipper on the database host. The pipeline detects this; it cannot repair it. Watermarks stop advancing, and no incident is closed on the strength of the silence.
5. Dirty wiki tree.dirty_tree blocks every agent stage, because a failed run rolls the tree back and would take your uncommitted edits with it. Commit or stash anything outside digests/ and html/.
6. Lock contention.another dbwiki command holds the lock (pid …, run, since …). Expected when you type a command during a tick. Wait, or use --lock-wait. A stale-looking .state/orchestrator.lock file is not a stuck lock — the flock dies with the process.
7. Unpushed wiki commits piling up.wiki remote … unpushed=N above the threshold (10). cd wiki && git push, or set report.push: false if this deployment has no remote.
8. A stage silently stopped running.stage_stale plus a last success timestamp far in the past. Read .state/cron.log, check crontab -l against docs/scheduling.md, run the stage by hand once.
9. Portal down. Symptom: the browser does not connect at http://127.0.0.1:8765, or the workbench stops answering part-way through a form. dbwiki health confirms it in the dependencies block. That block carries a portal line only when the config has a portal: block, so an install without one never reports the workbench at all.
portal UNREACHABLE http://127.0.0.1:8765/api/health
PROBLEM portal: the incident workbench is not reachable at http://127.0.0.1:8765/api/health (nothing else is affected; incident work needs the CLI)
-> systemctl --user restart dbwiki-portal (or `uv run dbwiki portal serve`); until then use `dbwiki incident ...`
Until it answers, use the dbwiki incident CLI (user-guide.md), or commit a page by hand in wiki/ for something the CLI has no verb for. Two consequences are worth knowing before you do. A commit made out of band, by the CLI or by hand, appears in the incident's page history in the workbench once it is back, because that history is git log over the page and not portal state. And a portal commit that was never pushed rides the next tick's push, so an unreachable remote is not an emergency.
A 423 in the browser is not this failure. Another command holds the wiki lock, almost always the 2h tick. Nothing was written, nothing failed, and dbwiki health stays green. Wait a moment and click again.
Off by default; on here (alerts.enabled: true, sink: file → .state/alerts.jsonl). Sinks: stderr (one JSON line, lands in cron mail), file, webhook (one JSON POST, 5s timeout, no retries — ntfy/Slack/Teams shape), or several via sinks: [...].
Alerts consume the health assessment; they never decide what a failure is. dbwiki run and dbwiki health --alert evaluate it against .state/alerts.json, keyed by fingerprint (category + db + source + detail):
a new fingerprint alerts once;
the same failure next tick updates count/last_seen silently;
a material escalation — the category for a (db, source) changing, e.g. source_silent → collection_failure — is a new fingerprint, so it alerts again with reason: escalation;
a fingerprint that disappears is dropped and counted as recovered in the run-health event. There is no success alert, and a healthy run does not touch the state file.
Each alert carries: run_id, fingerprint, category, db, source, reason, escalated_from, the bounded window fact behind it, first_seen, at, last_success per stage, and recovery_hint — the exact command. Never a log message, a prompt, a page body or a credential. An unknown sink name is a configuration error: the run warns, counts a sink_error, and leaves the state untouched so the finding alerts again once the config is fixed.
cost and quality per (task, adapter, tier or model, mode), optionally over a time window
Everything except Langfuse is counts and identifiers only — never a prompt, a log message, a page body or a credential. Langfuse deliberately carries content (prompts embed real alert-log excerpts), so point it at a host you trust with that; the cron environment must carry LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY for the export to run. All telemetry is best-effort: a capture failure warns on stderr and lands in the run-health event, but never blocks or rolls back an otherwise-good run.
dbwiki stats output to read for a model decision:
task adapter tier mode n ok fail rollback med_s cost lint_defect att
ingest pi cheap structured 271 227 44 0.162 14.4 0 0 1.01
ingest pi strong structured 190 131 59 0.311 18.2 0 0 1.01
report codex strong agentic 32 32 0 0 92.7 unk 0 1.03
Missing usage is reported as unknown beside a cost_n sample size, never estimated. No cheap-vs-strong verdict is offered until both sides have ≥5 runs.
python3 n8n/scripts/watchdog.py # exit 1 + JSON when something is off
python3 n8n/scripts/watchdog.py --fix # and repair a small-context model
Covers the checks dbwiki health does not make: that the tick is not older than its schedule says, and that config/schedule.json matches the installed crontab. n8n/ itself is an inactive side-by-side experiment; the crontab stays authoritative, and cron runs the --fix form every 10 minutes (runbook item 2). Without --fix the script only reports; with it, the one failure it can repair is a configured model served under health.min_context or not loaded at all.
Daily — glance at wiki/html/index.html; it is the product.
When cron mail or .state/cron.log moves — dbwiki health, then the runbook above.
Weekly — the Monday lint run's findings, and dbwiki stats if you are tuning models.
Monthly — the source-catalog review run; deprecating a source makes every page citing it stale, so it gets re-researched automatically.
After a reboot — unsloth start, then dbwiki health. Nothing restarts the model server for you.
Orientation
Overview
The entry point for the whole project: what it is, why it is shaped this way, what it produces, and where to read next. Written 2026-08-29 against the running configuration.
Elasticsearch~1M raw log docs
Compactorpython, no model
Digest~30 lines a day
Triggerwake or skip
Agentonly if notablejudgment, not bookkeeping
Wikigit, markdown, cited
The whole architecture in one line: deterministic compaction decides
what happened, and only then does a model get asked what it means.
Logbook turns a firehose of Oracle log events into a small, curated, git-versioned markdown wiki that a DBA can actually read.
Every two hours it reads the day's alert / listener / dataguard events for every database out of Elasticsearch and boils them down deterministically. Only when the result contains something worth a human's attention does it ask an LLM to write that up: a journal line, an error-class page, an incident, a fleet report. The wiki is the product; Elasticsearch stays the source of truth and every wiki claim points back into it.
There are ~1M log docs in the local cluster and 99% of them are the same three lines repeating (Archived Log entry NNNN added, listener heartbeats). Letting a model paginate raw hits would be slow and expensive, and would bury the signal in the repetition.
So a deterministic compaction layer sits between Elasticsearch and the LLM. Plain Python, no model: a (database × window) slice becomes a small structured digest — counters for the routine, verbatim text plus ES doc ids for the notable, and a list of deltas against everything seen before. A boring day is ~30 lines; an interesting day maybe 200. That is all the model ever reads.
The consequence that shapes everything else: the machine decides what happened, the model decides what it means. "Notable" is a deterministic verdict from the compactor, never a model's opinion — the LLM is never woken just to conclude that a window was boring.
Each layer only ever talks to the one below it. The model lives at layer 2 and never reaches past layer 1 — it reads digests, not Elasticsearch.
DESIGN.md is the long-form version of this diagram, including the parts that were designed for but not built (per-sql_id pages, more sources, local search).
One deployment (the author's), as of 2026-08-29 (config/dbwiki.yaml, crontab -l):
Stage
Path
Runs on
compaction
deterministic Python
free, local, every tick
ingest
structured
local Nemotron-3.5-Lightning-30B-A3B via pi → unsloth studio
report, routine window
structured
same local model
report, notable window
agentic
gpt-5.6-luna via codex (cloud)
lint
agentic — always
gpt-5.6-luna via codex
research, error pages
structured (deterministic fetch + summarize)
local model, no browsing
research, source review
agentic — always
pi + pi-web-access
Schedule: a 2h adaptive tick, a 23:30 consolidation, weekly lint and research, a monthly source review, and a 30-minute telemetry emitter. docs/scheduling.md is the reference for the crontab and the model routing.
The two optional distributed shapes are off here: analyst.enabled is unset (ADR-0001, agentic reports delegated to a workstation) and research.mode is structured, not offload (ADR-0002, anonymized research on a cloud box).
Words used precisely throughout the docs and the code:
digest — compactor output, machine-facing (layer 1). report — reporter output, human-facing (layer 3). Never the same artifact.
notable — the compactor's deterministic verdict that a window contains something outside the routine classes, or a delta. It is what wakes the LLM.
delta — something never seen before or out of line with the baseline: first_ever_code, silence, rate_anomaly, new_service, new_client_program, after_change.
tier — cheap or strong, chosen per database per window: strong when the digest carries incident-grade evidence.
agentic vs structured — whether the model edits wiki files itself or returns one JSON proposal that deterministic Python writes.
rails — the non-negotiable checks around every agent run: single-flight lock, clean-tree requirement, result validation, provenance lint, rollback, ledger entry.
ledger — .state/ingest_ledger.json: per digest, whether it was ingested or failed, with the content hash and failure category dbwiki retry needs.
watermark — per-db window end already processed; makes ticks incremental.
run id — one id shared by the run-health event, the agent-run telemetry, the wiki commit's Run-ID: trailer and the Langfuse trace.
know what is scheduled and which model each stage spends
scheduling.md
know what an agent is allowed to write
wiki/AGENTS.md
know how claims are evidenced and linted
provenance.md
see the architecture rationale and the deferred extensions
../DESIGN.md
see what changed when, with measurements
../CHANGELOG.md
Mechanism
How it works
The pipeline end to end: what happens between an Oracle log line landing in Elasticsearch and a sentence appearing in the wiki. Read overview.md first for the vocabulary.
Three text sources plus optional metrics, declared in config/dbwiki.yaml under sources: — each is an index-pattern list, a timestamp field, the fields that may carry the database name, and a pattern library:
Both index generations are listed on purpose, so a window that straddles the legacy indices and the newer data streams still works. db_fields is tried in order (oracle.database.name, then db_name.keyword). ECS listener documents carry no database field at all, so listener events are attributed via db_service_field — the TNS service-name variants of a known database (cdb1, cdb1.world, CDB1_DGMGRL.world, …).
A source window that comes back empty is not trusted blindly: _check_schema (compactor.py) distinguishes "quiet" from "the field layout drifted" and raises UnsupportedSchemaError rather than silently reporting zero events.
Log shipping into Elasticsearch (filebeat on the database hosts) is a separate system this repo neither installs nor supervises — it only notices when the feed stops.
One run per (database, window). The tick uses day-so-far (00:00 → now); dbwiki compact and dbwiki backfill can use any range.
Pull the window from ES over plain HTTP (es.py, requests, paged by compactor.page_size).
Classify every event against the source's pattern library into lifecycle, dataguard, error, warning, routine, or unmatched.
Compact: routine classes collapse into counters (per service, per client program, per return code); notable classes are kept verbatim with their timestamp and ES index/_id, grouped by rule and message template. Group and sample counts are capped (max_groups_per_rule, max_samples_per_group, max_unmatched_groups) so one runaway error cannot blow up a digest. Each group also carries a context block: the context_lines (3) log lines either side of its first occurrence, routine neighbours included, so a reader can see what led up to the event and what followed. Only the first context_max_groups (6) groups in digest order (error class first) keep the block, so a storm of unmatched groups cannot crowd the prompt. context_lines: 0 switches it off and the key is then absent from the digest entirely; context never reaches the content hash, so a growing day-so-far window does not re-ingest just because a group picked up more neighbours.
Look up trace files (trace.py, optional, config trace_lookup). An error-class group whose event names a .trc file — "Errors in file …", "Incident … dump file: …" — triggers one search against .ds-logs-oracle.trace-* for that path, and a bounded excerpt of what comes back lands in the source section as trace_evidence. Excerpting here rather than in a prompt keeps the digest the single deterministic context artifact: structured ingest, escalated reports and agentic mode all read the same evidence, and a replay reproduces it. Only the path and the document count reach the content hash, so retuning the excerpt re-ingests nothing, while a trace arriving after its alert lines costs one re-ingest per new chunk, bounded by max_docs_per_path. The key is absent when there is nothing to say.
Detect deltas against the per-db registry (.state/registry/<db>.json, which remembers every code, service and program ever seen plus daily counters):
delta
fires when
first_ever_code
an ORA-/TNS- code never seen for this db before
new_service / new_client_program
first appearance in the registry
rate_anomaly
window per-hour rate > rate_anomaly_factor (10×) the median of the last rate_baseline_days (7), for counters above rate_anomaly_min_count
silence
no events although the source logged on each of the last silence_windows (3) days — only for windows ≥ silence_min_hours (20), so a quiet night is never an outage
after_change
an error or dataguard group's first occurrence lands within after_change_hours (2) of a lifecycle change on the same database — the gap is stated as a fact, so the model never has to infer cause itself
Verdict: notable = true if any notable group or any delta exists.
Emit the digest twins — digests/<db>/<date>.json and .md — through an atomic write, and advance the watermark for the database.
The digest is also hashed (Compactor.content_hash) over its semantic content only, so two ticks whose windows differ but whose events do not produce the same hash. That hash drives the dedupe in the next step and the replay safety of dbwiki retry.
Anatomy of the readable twin (real excerpt, trimmed):
# Digest: cdb1 — 2026-08-29
- window: `2026-08-29T00:00:00Z` → `2026-08-29T20:15:00Z`
- events: 1,459 total, 176 notable in 77 groups
- notable: **true**
## Deltas (never seen before / anomalies)
- **first-ever error code** `ORA-16649` (alert, first seen 2026-08-29T08:04:00.816Z)
## alert — 531 events
classes: routine: 365, error: 63, unmatched: 57, warning: 39, lifecycle: 6
### Notable
- **[error] tns_error** ×32 (`TNS-12543`, `TNS-513`) — 08:03:49 → 08:05:26
> Fatal NI connect error 12543, connecting to: (DESCRIPTION=…)
- es: .ds-logs-oracle.alert-default-2026.08.20-000008/AaBNRRdNRyfGx_MnTLot, …
Everything an agent later writes is anchored to lines like these — the digest path, and optionally the ES doc ids from the JSON twin.
Pure function: digest + prior ledger entry + tick flags → decision. No timestamps, no randomness, so --explain and the real run always agree.
Check order, first match wins:
First match wins, so the order is the semantics: an unchanged window can never reach the notability check, and a routine one never wakes a model by accident.
Tier comes from the same module: digest_needs_escalation() returns strong when the digest carries a first_ever_code or silence delta, or any error-class notable group; cheap otherwise. Tier is per database per window, not per tick, and it also decides which path an escalated report takes.
dbwiki run --explain [--json] replays steps 1–3 only — no agent call, no digest emit, no watermark or ledger write, no run-health event.
The harness is invoked with (task, prompt, wiki checkout) and must exit having (a) made its wiki edits and (b) written .agent-result.json: {task, db, notable, summary, pages_touched, incidents_opened, incidents_updated, flags}. The agent never touches git — the orchestrator validates, commits and rolls back.
Adapters (agents.adapter): codex (codex exec, reads AGENTS.md natively), claude (claude -p --output-format json), pi (any provider pi knows, including a local OpenAI-compatible server). Any other name fails at config load, naming the supported adapters. Each adapter also has a usage parser, so tokens and cost land in telemetry where the CLI reports them.
Structured mode covers ingest and routine reports; a notable window's report stays agentic unless agents.escalated_report says otherwise. Lint is always agentic.
Two modes, because small local models have fine judgment and terrible file mechanics:
agentic (agents.mode: agentic)
structured (agents.mode: structured)
model sees
the wiki checkout, AGENTS.md, tools
one text prompt, no tools, no file access (pi --no-tools --no-context-files)
writers are idempotent and lint-clean by construction; the model never names a path, never closes an incident, and codes it did not see in the digest are dropped into flags
Structured mode covers ingest and routine reports. A notable window's report stays agentic by default — attaching the window to existing history is the report's whole value and needs a model that can read the wiki — unless agents.escalated_report: structured, which instead hands the strong tier a deterministically assembled Material pack (notable digest excerpts, the latest same-day prior report, up to 3 open incidents, up to 5 error-class pages, each capped and marked [truncated]) and renders its extra notable_analysis field as a ## Notable items section. Lint always runs agentic.
Both ingest prompts also carry the database's own history, so a day is judged next to the days before it. The compactor writes a changes list at the top of every digest and a ## Changes section in the readable twin, both derived from the lifecycle-class groups the pattern library already classifies: ALTER SYSTEM SET, mount and open, tablespace and datafile DDL, redo config, instance startup and shutdown. db_history.gather reads those changes back off the digests on disk across the last agents.history_days (90) days, together with the journal headlines for days that had something to say, the incidents opened or updated in that window that have since been resolved and the day each was, the operator action records against them (record-action, resolve, reopen), and the past_fixes rows for the codes in today's digest. The open incidents are not in the block; both prompts already print them for themselves, directly above it. db_history.render puts the result between the error-page list and the digest, in structured.build_prompt and in the agentic ingest prompt alike. The sections run past fixes, operator actions, resolved incidents, changes, journal, and the caps follow in that order: 5000 characters for the whole block, 6 fixes, 8 actions, 5 incidents, 12 changes, 7 journal days, every line at MAX_LINE. The order is the cap's order, because the cap truncates the tail: the lists that answer "has this been fixed before" lead, and the journal, the most restateable and least actionable of the five, goes last. Changes are folded one line per day and rule before the cap applies, so a single restart's eight to eleven lifecycle groups read as four lines naming the close, the mount and the open, and the change line is the line the rule's regex actually matched, not whichever line the ES document happened to open with. A day whose digest records no events at all, or exists, is not notable and has no changes, contributes no line, so quiet days do not spend the journal budget. The block is labelled context, not evidence: INGEST_TEMPLATE carries a rule telling the model to use it to judge whether today's events are new, recurring or follow a recorded change, and never to cite it as evidence for today or restate it as a fact of this digest. The agentic prompt carries the same sentence. agents.history_days: 0 drops the block entirely.
When an escalated window runs agentic and the harness itself dies — a nonzero exit, a timeout, a quota refusal — the structured escalated path runs once for the same window, so the wiki still gets a report. That result carries a fallback: structured … flag naming the adapter's failure. Both attempts land in telemetry: the agentic one as a rolled-back agentic row, the structured one as the row that committed. A validation failure or an unparsable result is not a harness death and takes the feedback-retry path below instead. agents.escalated_report_fallback: structured|none, default structured.
A validation failure, or a clean exit with no/unparsable result JSON, gets agents.feedback_retries (1 here) second chance: same prompt with the problem list appended and the failed attempt's edits still in the tree to fix in place. A timeout, a nonzero exit or an unknown adapter is never retried. agents.timeout_seconds (1800) bounds every call.
Which local model goes in agents.pi is a measured choice, not a guess. dbwiki eval run --model M replays the seven real digest fixtures through the same build_prompt → propose → apply_proposal path this section describes, each into a throwaway wiki, and scores the result: did it parse, did it apply, were its error codes in the digest, did it invent an incident update, how many flags, how long, how many tokens. Nothing it touches is the real wiki, so it is safe to run against production config while a tick is due. The table it prints is the comparison; with langfuse.enabled the same run also lands as a Langfuse experiment on the dbwiki-ingest dataset (docs/langfuse.md, "Experiments").
dbwiki eval history --db cdb1 [--days 14] asks a different question about the same chain: whether the database-history block above the digest changes what the model writes. It replays the last --days digests of one database out of a real wiki checkout twice — arm none with history_days: 0 and arm history with the configured window — each arm on its own throwaway copy of the wiki with the incidents opened on or after the replayed day deleted, so an arm is scored on whether it would open the incident that day deserves rather than on having read it. The markdown report says, per day and per arm, whether the day came out notable, what the model did about an incident, which dates, slugs and codes out of the block its prose repeated, and the flags, tokens and duration; the days where the two arms disagree about notable or the incident action are marked and listed again at the end. That last list is the answer the command exists for: an empty one means the block is spending tokens and buying nothing. It is a local loop with no Langfuse, and it writes nothing but the report.
Single-flight lock.flock on .state/orchestrator.lock for the whole command (run, ingest, report, lint, research, retry, analyst, backfill, health --alert). Default wait is 0 — fail fast, with the holder's pid and command on stderr and a lock_busy run-health event; --lock-wait SECONDS or DBWIKI_LOCK_WAIT changes that. Read-only invocations (--explain, --dry-run, --deterministic-only, compact, health, stats, dbs, es, awr, render-daily) never take it.
Clean-tree check. Uncommitted changes outside digests/ and html/ abort the run (dirty_tree) — a failed run ends in a rollback that would otherwise wipe your edits.
Digests committed first, so the agent runs over a tree where its own changes are the only non-machine ones.
Result validation. Task matches, required keys present, every pages_touched entry exists on disk, nothing claimed under digests/ or html/, digests/ untouched, the tree actually changed, log.md updated, and every changed page other than index.md and log.md declared in pages_touched — an undeclared change refuses the publication. Ingest, report and lint additionally compare every changed incidents/ page against the base revision: an agent may append a ## Update <day>, an evidence line or an occurrence row and bump updated:, but status:, the monitoring: window, the ## Action records and the ## Resolution history rows belong to dbwiki incident (ADR-0003), and any change to one of them refuses the publication. Structural lint cannot see this: it judges the resulting page, not whether the actor was allowed to make the transition.
Stage-specific rails. Research may only change errors/, sources/, index.md, log.md; it may not create a source page (new sources enter by a human commit); every URL in a changed error page must sit on an approved source's domain.
Deterministic provenance lint over the changed paths only. Error-severity findings (frontmatter-malformed, frontmatter-contradictory, wikilink-broken, digest-missing, citation-malformed) block the commit; warnings do not. Rules: provenance.md.
Commit or rollback, as a unit. Success → one commit carrying a Run-ID: <run id> trailer, plus a push when report.push is on. Failure → git checkout -- . and git clean -fd excluding the machine directories, so a rolled-back run never deletes a digest or a rendered HTML page. Every commit is scoped by pathspec to the paths it owns — the proposal's paths for a stage, digests for the compactor's output, html for the rendered summary — so a file a human staged mid-run is neither committed nor pushed. The lock keeps two pipeline runs apart; it does not stop a human using git.
Ledger + telemetry. The digest's ledger entry records ingested or failed with the content hash, the error category and the cost/quality block; one run-health event and one agent_runs.jsonl line are appended.
With analyst.enabled and report.push on, pull the wiki remote (rebase with autostash; the analyst's pushed report wins a same-path conflict; any failure is undone and noted, and the tick carries on). Then fold any analyst telemetry results into .state/agent_runs.jsonl (ADR-0001; no-op when the analyst queue is unused).
Discover databases in ES and compact each one's day-so-far window.
Decide per database (section 3).
Ingest each woken database, through the rails.
Report if anything notable was ingested, or on the consolidation tick.
Render wiki/html/<day>.html + index.html — every tick, even one where every database skipped, so today's operator page is never stale.
Evaluate alerts against the state the tick leaves behind.
dbwiki run exits 1 when any database or the report failed, so cron sees partial failures; the HTML render and the alert evaluation still happen first.
The daily HTML page is deterministic machine output, like digests: a Changes strip first when anything changed (every database's changes, one fleet timeline, capped at 25 rows), then tiered Needs attention (notable databases plus every still-open incident), Worth a look (first-ever codes, new services/programs, rate anomalies, silence — always labeled as telemetry, never as database state) and a collapsed Routine table. Headlines come from the ledger's model-written summaries with a deterministic fallback; links resolve against report.link_base, which must point where markdown renders (the wiki's remote), not at a local path. The workbench draws the same strip over a day's report (the fleet's changes that day) and over a digest (that database's), from the same daily_html.change_rows, capped at the same 25.
7. Research — research.py, research_structured.py#
Its own weekly schedule, deliberately not part of the tick: its cost is bounded and independent of log flow, and new error codes become candidates on their own (a fresh errors/<CODE>.md has no researched: date yet).
Deterministic Python picks the workload — unresearched pages, pages whose research went stale because a cited source was re-reviewed or deprecated, pages linked from open incidents first. Then, depending on research.mode:
structured (current): no agent browses. Code fetches the Oracle error help page itself from the one approved + fetchable: true source, and a cheap local text call distills the fetched text into {"cause", "action"}. apply_research writes researched: and the ## Reference section, cited against exactly the source that was fetched. A page whose fetch fails is flagged and left untouched; the run still succeeds for the others.
agentic: a web-capable adapter (research.adapter / research.model) browses and writes cited ## Reference sections itself.
offload (ADR-0002): no agent and no web egress here at all — see §8.
Citations are gated by the source catalog: wiki/sources/<slug>.md pages with status, tier, domains, fetchable, last_reviewed, review_after_days. A URL may be cited only if its host is (a subdomain of) a domain on an approved page. External knowledge never enters Occurrences or Resolution history. Deprecating a source makes every page citing it stale, so it gets re-researched. Source-page review (--review-sources, monthly) always runs agentic — judging whether a live site went stale needs a browse.
Analyst node — ADR-0001. analyst.enabled: true splits the pipeline across two machines connected only by the wiki git remote: an on-prem node with ES credentials and no cloud-LLM credentials, and an operator workstation with cloud-LLM credentials and no ES access. An escalated window then writes the structured placeholder report immediately and enqueues the fully assembled agentic prompt as queue/pending/<...>.json in the wiki repo. dbwiki analyst --once on the workstation claims the oldest pending day (superseding older requests for the same day), runs the same rails, and overwrites the same report path — the live page silently upgrades from structured to agentic. Telemetry comes back through queue/results/, folded in by the next tick.
Research offload — ADR-0002. research.mode: offload sends anonymized requests through a separate exchange repo to a lean researcher (src/dbwiki_researcher/) that never clones the wiki. redact.py builds a per-run vocabulary from wiki pages, registries and ES config, sweeps patterns (IPs, FQDNs, (HOST=…), (SERVICE_NAME=…), USER=, ES ids, oracle paths) and replaces every hit with a consistent pseudonym (DB_A, HOST_B, …); a fail-closed leak check refuses any request still carrying an identifier. dbwiki redact errors/<CODE>.md prints exactly what would leave the box. Results are validated, de-mapped, and written through the same deterministic writer and rails.
Everything under .state/ is machinery state, never wiki content:
file
what
watermarks.json
per-db window end already processed
registry/<db>.json
every code/service/program ever seen + daily counters (delta baselines)
ingest_ledger.json
per digest: ingested/failed, content hash, error category, cost/quality block
run_health.jsonl
one event per command: per-db decisions, reasons, tier, validation outcome, commit
agent_runs.jsonl
one line per attempted LLM stage: duration, tokens, cost, attempts
alerts.json / alerts.jsonl
alert dedupe state / the file sink
cron.log
stdout+stderr of scheduled runs; silent when healthy
orchestrator.lock
{pid, command, since} of the current holder
elk/*.jsonl
derived streams for the filebeat shipper
Replay semantics, by design:
Compaction is idempotent — re-running a window overwrites its digest deterministically.
Ingest is deduped by window end and content hash, so a tick that finds nothing new does nothing.
Same-day supersede — the daily digest is cumulative, so re-ingesting a day rewrites that day's journal entry, incident update section and occurrence row instead of appending a reworded duplicate (enforced by construction in structured mode, by contract in agentic mode). A second open for a database that already has an incident open on that day collapses onto the existing page, however the model spells the slug.
Failures are replayable when the category says so.dbwiki retry re-ingests failed digests whose file still exists, whose content hash still matches, and whose category a rerun can fix (validation_failed, harness_error, agent_timeout, no_result — never dirty_tree or wiki_missing). It goes through the normal ingest path, rails included.
Nothing auto-closes. An incident is never closed because events stopped; recovery needs explicit evidence.
Failure categories and their recovery commands: monitoring.md.
