Designing Evidence-First Incident Investigations

Designing Evidence-First Incident Investigations

This is a set of engineering design notes drawn from building DupTrace, an incident-investigation assistant. It is not a controlled study, a benchmark, or a claim that the approach has been validated across production incidents. DupTrace's public demo uses deterministic fixture data, so its outputs demonstrate workflow and interface behavior rather than live operational performance.

The design question is practical: how can an assistant help an engineer investigate an incident without turning a plausible explanation into a claimed fact?

The central distinction: observation versus hypothesis

An incident investigation usually combines several kinds of information:

• Observations: measurements, logs, events, alerts, and deployment records returned by a system.
• Relationships: timestamps or shared identifiers that connect observations.
• Hypotheses: explanations that could account for the observations.
• Conclusions: explanations supported by enough independent evidence to justify a stronger claim.

These categories should not be collapsed. If a deployment occurs shortly before an alert, the timeline establishes temporal proximity. It does not, by itself, establish that the deployment caused the incident.

A useful investigation records both what is known and what remains unresolved.

Build a timeline before writing a story

A timeline helps an investigator check ordering and identify gaps. Each entry should retain its source, timestamp, relevant service or incident identifier, and enough context to revisit the underlying record.

Before comparing events, check basics such as:

• Are timestamps in the same timezone and format?
• Are event times and ingestion times being confused?
• Is the data complete for the period being examined?
• Are two records describing the same event or separate events?
• Is the apparent sequence supported by source records?

A clean timeline is a navigation aid, not proof of causality.

Make tool use inspectable

DupTrace exposes read-oriented tools for retrieving service health, incidents, deployments, events, and evidence. An assistant can use these tools to gather context, but its explanation should remain traceable to the records returned.

A robust interaction pattern is:

1. Identify the incident and affected service.
2. Retrieve the incident record and relevant health signals.
3. Inspect deployments and events in the incident window.
4. Gather evidence linked to the incident.
5. Compare the evidence with candidate explanations.
6. State what supports each explanation and what is still missing.

The assistant should not claim to have checked a source it did not query. It should also distinguish an empty result from a failed request or unavailable data. “No evidence returned” is not equivalent to “evidence that nothing happened.”

Keep uncertainty visible

A single confidence score can hide why two cases differ. Prefer explicit statements tied to evidence:

• Observed: a recorded event or measurement.
• Consistent with: evidence that fits a hypothesis but does not establish it.
• Contradicts: evidence that weakens a hypothesis.
• Unknown: unavailable, incomplete, or not yet checked.

These labels are a proposed communication discipline, not a validated scoring model. They make it easier for a human investigator to challenge the reasoning and decide what to inspect next.

Bound the assistant's authority

For an investigation assistant, read-only access is a useful default. Collecting evidence and explaining it should be separate from restarting services, rolling back deployments, changing configuration, or modifying infrastructure.

Any future action-taking workflow should require explicit authorization, narrow permissions, and an auditable record of what was requested and executed. A generated explanation should never silently become an operational action.

What the demo proves, and what it does not

DupTrace demonstrates a structured investigation flow over deterministic fixture data, with separate tool calls for retrieving evidence and correlating records. It can be used to inspect the interaction design and how the assistant communicates uncertainty.

It does not prove that the workflow improves mean time to resolution, identifies root causes more accurately, or generalizes to production incidents. Those claims would require a documented evaluation using representative incident cases, defined outcome criteria, baselines, and reproducible results.

A sensible next evaluation

A future evaluation could use a labelled set of incident timelines and ask independent reviewers to assess:

• Whether every factual claim is supported by a retrieved record.
• Whether the assistant distinguishes correlation from causation.
• Whether it identifies missing or contradictory evidence.
• Whether another engineer can reproduce the investigation from the cited records.
• How often it invents an event, source, or conclusion.

The dataset, labelling process, failures, and results should be published before reporting performance numbers. Until then, these remain proposed evaluation criteria, not results.

Closing note

An incident assistant earns trust less by sounding certain than by making its reasoning inspectable. Preserve the source of each observation, show the difference between evidence and explanation, and leave unresolved questions unresolved.

That is the design principle behind DupTrace. It is a starting point for evaluation, not a claim that the problem has been solved.