Operational memory: retrieving past incidents to rank today's hypotheses

Experienced engineers shortcut triage with memory: “This looks like the redis saturation we hit after auth deploys.”

Operational memory is the attempt to give software that shortcut, without pretending history is always right.

The retrieval problem

Each resolved incident can be indexed as a document with:

• Structured fields (services, error classes, deploy ids, fix type)
• Free text (RCA summary, timeline notes)
• Outcome labels (true root cause, effective remediation)

When a new incident arrives, we build a query from current signals and retrieve the top-k most similar past incidents.

Modern stacks use dense embeddings (transformer encoders) plus metadata filters:

retrieve(q) = top-k incidents where metadata filters match and embedding similarity is highest

Filters matter: same service, same region, same alert type. Unfiltered retrieval returns scary stories from unrelated fleets.

What similarity should mean

Good features for ops retrieval include:

• Alert fingerprint: normalized alert name + key labels
• Topology context: multiset of services on the signal map
• Symptom vector: which metric families moved (latency vs errors vs saturation)
• Change signature: deploy vs cert vs config vs capacity

Pure text similarity on Slack threads alone is brittle; structure stabilizes retrieval when wording differs.

Using retrieval as a prior, not a verdict

Let mem(h) be a boost to hypothesis h when past incidents with that cause appear in the top-k retrieval results.

Rules that keep memory safe:

1. Never retrieve-only: if live metrics contradict memory, live wins.
2. Show neighbors: “Similar to INC-1924 (resolved: pool exhaustion)” with link
3. Decay stale playbooks: architecture changes invalidate old fixes
4. Separate symptom match from cause match: many incidents look alike until topology differs

This is the same discipline as clinical decision support: suggest, don’t diagnose.

Cold start and privacy

New customers lack historical incidents. Memory falls back to:

• Public runbook patterns (weak prior)
• Topology-only ranking
• Rapid learning as humans confirm or reject hypotheses (feedback stored as new indexed incidents)

Teams also care where memory lives: tenant-isolated indices, no cross-customer training without contract, audit on what was retrieved.

Feedback loops without silent drift

When an engineer marks “leading hypothesis wrong,” that label should update:

• Down-rank similar false retrievals
• Up-rank the corrected cause for future (q)

Without labels, use weak supervision: accepted RCA text and whether repair playbook succeeded.

Product implication

Operational memory is why “carry what worked forward” is more than a tagline, it is a retrieval-augmented prior in the ranking function, visible in the workspace when similar incidents are cited.

The science is established; the craft is keeping humans in control of when history applies.