How to Handle Software Incidents

This guide is for anyone who has just discovered that production is broken and needs to know what to do next, or who is setting up incident response for a system before the first incident happens. By the end you will have a structured way to triage, communicate, fix, and learn from incidents, along with a sense of which patterns make the next one cheaper than the last.

Who This Guide Is For

Operations leads, technical owners, and project managers responsible for a piece of software that the business depends on. The incident might be an outage, a data issue, a security event, a third-party failure, or a slow-burning problem that has been building for weeks. The mechanics of responding to it are similar; the skill is doing them under pressure, with a system half-broken and stakeholders asking questions.

Before You Start

You should have basic incident infrastructure in place before the first incident: monitoring that catches problems, an alerting path to a real human, a place where the team can coordinate (a Slack channel, a Teams call, a war room), and a list of who has the authority to make which decisions. If you have not got those, How to Keep Your Software Secure After Launch covers the operational setup that incident response sits on top of.

If you are dealing with an active incident right now, skip to step 1.

Step 1: Confirm the Incident Is Real

The first action when an alert fires is to confirm there is actually a problem. Alerts have false positives. Monitoring tools have blind spots. A “the site is down” report from one user can mean a real outage, or it can mean their wifi is broken.

Confirm from at least two angles. Open the application in a private browser window. Check the monitoring dashboard. Look at the error rate over the last hour. If multiple sources agree the system is degraded, this is a real incident. If they disagree, investigate before declaring an incident. False positives consume team time and erode the team’s willingness to respond quickly to real ones.

Once confirmed, declare the incident explicitly. A Slack message: “Incident declared: client portal is unreachable, started approximately 14:30, investigating.” This message creates a clear starting point, signals to the team to focus on the response, and creates the timeline that the post-incident review will use.

Step 2: Triage the Severity

Not every incident is equal. Severity drives the response intensity. A useful three-level scale:

• Severity 1: Critical. The system is down or significantly impaired for many users, data is at risk, or the impact is regulatory or financial. Full team focus, executive notification, communication to affected parties within an hour.
• Severity 2: Significant. Part of the system is broken, a subset of users are affected, or important workflows are degraded. The on-call response covers it, communication to affected parties as appropriate, executive update at end of day.
• Severity 3: Minor. A small subset of users are affected, the issue has workarounds, or the impact is cosmetic. Fixed during normal hours, no urgent communication needed.

The triage should happen in the first ten minutes. Severity 1 changes who needs to be on the call. Severity 3 means the team carries on with their planned work and addresses it in the normal flow. Mis-triaging severity is expensive in both directions. Calling Severity 1 on a Severity 3 wastes the team’s day; calling Severity 3 on a Severity 1 leads to slow response on a real crisis.

Step 3: Establish a Single Coordination Channel

In an active incident, communication fragments quickly if you do not actively concentrate it. People DM the operations lead asking what is going on, executives email asking for updates, customers contact support, and the technical team is debugging while answering five threads. The fragmentation slows the fix and produces inconsistent messaging.

The fix is to pick one coordination channel (usually a Slack or Teams channel created for the incident) and direct everything there. “All incident communication in #incident-2026-05-27. Updates posted every thirty minutes.” Anyone DM’ing the operations lead gets redirected. This sounds harsh; it is the only way to keep the technical team’s attention on the fix.

Within the coordination channel, designate an incident commander: one person responsible for keeping the response coordinated. The incident commander is not necessarily the senior technical person; their job is to make sure the response is structured, decisions are made, and communication keeps flowing. A junior person with clear authority over the incident process is often a better commander than a senior engineer trying to debug and coordinate simultaneously.

Step 4: Communicate at Predictable Intervals

Stakeholders not on the technical response need updates. Without updates, they generate them: phone calls, escalations, executive interventions. The way to prevent this is to provide updates predictably, even when there is nothing new to say.

The cadence depends on severity. For Severity 1, every 30 minutes externally and every 15 minutes internally. For Severity 2, every hour or two. For Severity 3, on resolution.

The template that works: what we know, what we are doing, what we do not yet know, when the next update will be. “We know the client portal has been unreachable since 14:30. We are investigating the database connection layer; initial signs point to the connection pool exhaustion. We do not yet know the root cause. Next update at 15:30.”

That paragraph is much more useful than “we are still working on it”. It tells stakeholders that the response is real, structured, and time-bounded. It also reduces the volume of “any update?” messages by 90%.

Step 5: Fix It

The technical response is its own work, and the right approach depends entirely on the incident. Some patterns that apply broadly:

Roll back if you can. If the incident started with a recent deployment, reverting is almost always the fastest path to restoring service. Investigating in production while users are affected is rarely the right call when you can return to a known-good state and investigate at leisure.

Stabilise before you fix root cause. The job in the first hour is to restore service, not to understand the underlying problem. Sometimes those are the same and the obvious fix is also the root cause fix. Often they are not. A bandaid that gets the system running again is acceptable for the next 24 hours while the proper fix is designed.

Two people on the fix is usually better than one. One person investigates, the other verifies hypotheses and watches the system. Three or more on the technical fix is usually counterproductive. Too many cooks, too much coordination overhead.

If the incident is genuinely beyond the team’s expertise, escalate. Open a support ticket with the hosting provider, the database vendor, the third-party service. Vendor support exists for exactly these moments. Asking for help is faster than struggling alone, and the cost of an hour of vendor escalation is dwarfed by an extra hour of downtime.

Step 6: Resolve and Verify

When the system appears restored, verify. Restoration after an incident is more fragile than it looks. A database that came back online may not be processing queries correctly. An integration that recovered may have lost messages during the outage. A deployment rollback may have left configuration in an inconsistent state.

The verification: run the critical user flows end-to-end. Confirm the error rate has returned to baseline. Spot-check recent data to confirm nothing was lost or corrupted during the incident. If the incident was a security event, additional verification is needed. Confirm no data exfiltration occurred, audit any access during the incident window, and check for newly created accounts or escalated permissions.

Declare resolution explicitly: “Incident resolved at 16:45. All systems verified. Full post-incident review to follow.” This signals to stakeholders that they can return to normal work and creates the closing bookend for the timeline.

Step 7: Hold a Post-Incident Review

Within a week of resolution, hold a structured review. The review is blameless. The goal is to understand what happened and improve, not to identify whose fault it was.

The structure that works:

• Timeline: what happened, in order, with timestamps
• Root cause: what was the underlying reason for the failure
• Detection: how did we find out, and how could we find out faster next time
• Response: what worked well in the response, what did not
• Action items: specific changes to prevent recurrence or improve future response

The action items are the most important output. “We should improve monitoring” is not an action item. “Add an alert when database connection pool utilisation exceeds 80%” is. Every action item has an owner and a date.

A real example. A client’s incident review of a two-hour outage produced five action items: improve the alerting on database resource usage, document the rollback procedure for the affected service, add a check to the deployment pipeline that catches the specific class of issue that caused the incident, schedule a tabletop drill of the same scenario in the next quarter, and update the runbook with the diagnostic steps the team used. All five were done in the following two weeks. The next similar incident took 20 minutes to resolve instead of two hours.

Common Mistakes

• No incident declared explicitly. People start working on the problem without aligning. The response is uncoordinated and updates do not flow.
• Communication scattered across DMs. The technical team is interrupted by stakeholder questions. The response slows.
• No incident commander. Decisions are deferred or made by whoever is loudest. The response loses coherence.
• Trying to investigate in production with users affected. Roll back first, investigate after. Restoration is the priority.
• No updates for two hours. Stakeholders escalate, executives intervene, the response is derailed by management attention. Predictable updates prevent this.
• No post-incident review. The same incident recurs three months later. The lessons did not stick because they were not written down.
• Blameful post-incident reviews. The team stops being honest about what went wrong; the reviews stop producing real action items. Blameless is non-negotiable.

What Good Looks Like

Effective software incident response is structured but not bureaucratic. Incidents are declared explicitly, severity is triaged in the first ten minutes, an incident commander coordinates the response, communication concentrates in a single channel with updates at predictable intervals, the technical team has space to work without being interrupted by stakeholder questions, resolution is verified before being declared, and a blameless post-incident review produces specific action items that prevent the same incident next time. The team gets better at it over time because every incident produces learnings that are written down and acted on, not just remembered until they fade.

Next Steps

If you have not yet established the broader operational discipline, How to Keep Your Software Secure After Launch covers the ongoing security and operational rhythms. If incident response is part of a longer-term relationship with a development partner, How to Manage Software Vendor Relationships covers that side. For the broader quarterly cadence that keeps systems healthy between incidents, see How to Run a Software Review Quarterly. For structured ongoing support, see Software Support Retainers or browse the full Operational Guides index.