Multimodal evidence fusion: why incident triage is not a single-model problem

Production incidents arrive as heterogeneous signals: a PagerDuty payload is not a Grafana time series, and a ServiceNow ticket is not a distributed trace. Each modality has its own schema, noise profile, and time resolution.

Treating triage as “send everything to one LLM” ignores a basic lesson from applied ML: fusion architecture matters as much as model size.

The formulation

We can think of triage as estimating which root-cause hypothesis h best explains observed evidence E, roughly, how well each theory explains every signal we have seen (metrics, logs, deploys, tickets).

In practice we never have true probabilities. We have noisy detectors per modality, anomaly scores on metrics, rule hits on logs, temporal overlap with deploy events, similarity to past incidents, and we need a pipeline that:

1. Normalizes each signal into comparable “support” for a hypothesis.
2. Resolves conflicts when metrics say “database” and the ticket says “VPN.”
3. Surfaces provenance so humans can audit each term in the product.

That is evidence fusion, not chat completion.

Modality-specific models (specialists)

A useful pattern is specialist extractors plus a general synthesizer:

Modality	Typical ML / rules	Output to fusion layer
Metrics	Anomaly detection, changepoint heuristics	“Pool wait elevated 12× baseline”
Logs	Pattern clustering, error rate spikes	“TLS handshake failures concentrated on ring-2”
Deploys / config	Event alignment, version diff	“auth-svc v2.14.1 deployed T−11m”
Tickets	NLP intent + entity extraction	“User-reported slowness; endpoint id laptop-8812”
Topology	Graph reachability	“6 services share redis-cluster”

Each specialist is smaller, cheaper to evaluate, and easier to regression-test than a monolith that must “understand” raw JSON from six vendors.

Late fusion vs early fusion

Early fusion (dump all raw data into one prompt) fails in ops because:

• Context windows fill with irrelevant logs.
• Timestamps and units get mangled in prose.
• The model invents connections that never existed in telemetry.

Late fusion (structure first, language second) is what investigation workspaces are for:

1. Build a signal map and timeline from structured extractions.
2. Rank hypotheses with explicit scores per evidence row.
3. Use an LLM only where language adds value: summarizing for handoff, drafting RCA prose, phrasing tier-1 notes, grounded in cited artifacts.

Confidence as calibrated agreement

A practical confidence score is not “the LLM felt sure.” It behaves more like agreement across weak learners:

• Multiple independent modalities support the same target (e.g. DB pool + deploy on the auth path).
• No high-weight contradicting signal is present.
• Optional boost from nearest past incidents in operational memory.

When modalities disagree, confidence should drop and the UI should show why, not collapse to a single paragraph.

What this means for Fixed.sh

Fixed is designed around fusion you can inspect: evidence rows, linked metrics, graph edges, and a ranked leading hypothesis. The science is in the pipeline and the data model, not in asking one model to hallucinate a complete story.

Open problems (honest limits)

• Cold start: new services lack incident history; priors must come from topology and generic runbooks.
• Correlated noise: one upstream failure creates correlated alerts; fusion needs deduplication.
• Label scarcity: true root cause labels are sparse; ranking must stay humble and human-overridable.

Those limits are why human-in-the-loop is not marketing, it is a requirement for any fusion system that touches production narratives.