If a change isn't backward-compatible, rolling back to the previous version breaks any client still running the old code, which turns a rollback into a second outage.

There's no way to know what the previous known-good state actually was during a rollback if config and scripts aren't versioned the same way code is.

These attack classes are the ones most likely to happen, since new endpoints and data paths often skip validation the old system had in place.

Migrations tend to carry over broader access than the new environment needs, since copying old permissions is easier than redefining them.

Average latency can look fine right up until peak load, so validating at the 99.9th percentile and under peak RPS is what catches the failure mode.

Individual services can each pass their own tests while the full user journey still breaks at the handoff between them.

A manual step is a step someone can skip or get wrong under pressure, and cutover is exactly when that pressure is the highest.

Limiting exposure to a small percentage first turns a bad deploy into a contained incident.

If users can tell a deployment is happening, through latency spikes or dropped requests, the deployment process itself has become a source of risk.

A rollback that requires building and shipping a new deployment is slower exactly when speed matters most.

An alert that just says something's wrong sends someone straight into the logs to work out what and how bad, and that's minutes lost during an active incident.

Without a threshold tied to the availability target, a rising 5xx rate reads as noise right up until it's already breached the SLA.

An empty catch block swallows the exact information needed to diagnose a failure, so the error disappears from the logs even though the failure still happened.

A correlation ID that survives every hop is what lets one failed request get traced across a dozen services.

Logs scattered across each service's own host mean an engineer has to already know which service failed before they can start looking.

Your staff should be ready for when a live incident happens.

Manual failover needs someone to notice, decide how to act, and execute the switch, all before the failure reaches customers.

A system that fails all-or-nothing turns every small failure into a full outage.

A process can be running and still unable to serve requests, so a check that only confirms the process is alive will keep routing traffic to a host that's actually broken.

During a staged rollout, old and new versions run side by side for a while. You need a plan for that overlap.

CPU, memory, storage, and I/O all get profiled under stress, and the results feed a documented growth model with more than 20% headroom. A leak, a GC pause, or a storage bottleneck invisible at normal traffic can each take a service down once real production load hits.

A migration is a natural point for a misconfigured cert or a loosened auth policy to slip through unnoticed.

A build can pass code review and still ship a regression in performance or security that review alone wouldn't catch.

Every first-level dependency gets a documented profile (latency, peak RPS, failure behavior) plus a quantifiable target to avoid gaps during cutover.

A patch that takes longer than the time-to-mitigate target arrives after the damage is already done, which defeats the point of having a target at all.

Automating the revert closes the gap between a metric breaching limits and a human reacting to it.

A problem caught on one canary host affects just that host. The same problem caught after a full rollout affects the whole fleet.

Alerts set to high severity by default get ignored the first time they're wrong, and once ignored, they stay ignored even when they're right.

4xx errors get their own monitoring, separate from 5xx, since a spike there is usually a client-side or API-contract issue. Volume anomalies, sudden drops or spikes in request rate are the earliest warning available.

An outage in one smaller region gets averaged out by every other region's traffic in a global metric.

Waiting for a real user to hit a broken flow means the first one to discover a failure is a customer, not a monitoring system.

The median can look perfectly healthy while the slowest 1% of requests are already timing out.

A fleet-wide average hides the one host that's actually struggling, since every other healthy host pulls the average back down.

Every service logging in its own format means learning a new parsing pattern for each one mid-incident, instead of searching across all of them the same way.

You should prepare a manual for scenarios when you are not around during a sudden incident.

Make sure the person in charge is the one who still works for the company, not someone who left the team for a while.

A post-mortem that assigns blame teaches people to hide problems instead of surfacing them.

Failover only works if the region absorbing the traffic can actually handle it.

An unbounded retry storm is how a small, recoverable failure turns into a full outage.

A service with no rate limit at its boundary can be taken down by a traffic spike that is unexpectedly large.

The only way to know whether a fallback actually works is to trigger the failure it's supposed to handle before a real incident happens.

Moving all traffic at once means any problem shows up at full scale immediately.

A layout tested only with placeholder text in one language can look fine and still break once real content replaces it.

An explicit user preference is essential for anyone traveling or on a shared device.

A missing translation shows up to the user as broken text or a blank space instead of a readable default.

An external dependency can change its behavior on its own schedule with no warning to your team.

Waiting until late in the project to stand up a full integration environment means integration bugs surface right before cutover.

A data migration that's already run can leave the database in a state the old code was never designed to read.

A staging config pointing at a test endpoint is easy to miss in review and expensive to discover in production.

A flag that's never been tested on is a flag whose actual behavior is unknown.

Monitoring blended across every active flag hides which one actually caused a regression.

A slow degradation that never crosses an alert threshold can still add up over weeks, and a daily summary is what surfaces that kind of drift.