Incident Runbook

Production incident response procedures for FeatureSignals. Severity classification, escalation paths, rollback procedures, communication templates, and the blameless post-mortem process — everything you need when things go sideways at 3 AM.

Severity Levels

Incidents are classified by scope, impact, and urgency. Use these definitions to triage quickly and consistently:

First Response Checklist

When an alert fires or an incident is reported, the first responder runs this checklist:

1. Acknowledge the alert — Silence the pager. Acknowledge in the incident Slack channel. This buys you time to think.
2. Assess blast radius — How many customers are affected? Which services? Is this a partial or complete outage?
3. Declare severity — Use the severity definitions above. Err on the side of over-classifying — you can downgrade later.
4. Start the incident timer — Note the start time (UTC). This feeds SLA reporting.
5. Open an incident channel — Create a dedicated Slack channel (#incident-{number}) or Zoom war room for P0/P1.
6. Send initial communication — Use the template below to notify affected customers and internal stakeholders.
7. Begin investigation — Check dashboards (SigNoz), recent deploys, database metrics, and error logs.

Rollback Procedures

If a recent deployment caused the incident, rollback is your fastest path to recovery. Don't debug in production — roll back first, investigate later.

1. Step 1: Identify the last known-good deployment from the deployment history in your CI/CD pipeline.
2. Step 2: Redeploy the previous version. One-click rollback is available in the Ops Portal.
3. Step 3: Verify health endpoints return 200 and evaluation latency returns to baseline.
4. Step 4: Confirm with affected customers that service is restored.
5. Step 5: Preserve all logs, metrics, and traces from the incident window for post-mortem analysis.

Communication Templates

Consistent communication reduces panic. Use this template for all customer-facing incident updates:

Subject: [INCIDENT] FeatureSignals {SEVERITY} — {BRIEF_TITLE}

Status: {INVESTIGATING | MONITORING | RESOLVED}
Incident ID: {INCIDENT_ID}
Start Time: {START_TIME_UTC}
Impact: {DESCRIPTION_OF_IMPACT}

Current Status:
{WHAT_WE_KNOW_AND_WHAT_WE'RE_DOING}

Next Update: {EXPECTED_UPDATE_TIME}

FeatureSignals Incident Response Team

Status Update Cadence

Post-Mortem Process

Every P0 and P1 incident requires a blameless post-mortem within 48 hours of resolution. The goal is understanding, not blame:

1. Timeline — Construct a minute-by-minute timeline from alerts, logs, chat messages, and deployment records.
2. Root cause — What specifically caused the incident? Use the 5 Whys technique to trace back to process or systemic gaps.
3. Impact assessment — Duration, affected customers, evaluation failures, SLA impact.
4. What went well — Acknowledge effective detection, fast response, good communication. Celebrate smart decisions under pressure.
5. What could be better — Detection gaps, unclear runbooks, tooling deficiencies, training needs.
6. Action items — Specific, assigned, time-boxed remediation tasks. Each action item links to a GitHub issue.
7. Review — Post-mortem presented at the next engineering all-hands. Action items tracked to completion.

Learn More

• Disaster Recovery Plan
• Architecture Overview
• Deployment Guide
• Contact Support