Revolut outage on 20 October 2026: how a degraded quorum and a Google Cloud power failure caused 11 hours of downtime

Most accounts of Revolut’s 20 October incident blame a Google Cloud power cut. Revolut’s own postmortem points somewhere less comfortable: coordination clusters that had been running degraded for weeks, an alert that never fired, and a failover system that refused to gamble with a bank’s ledger.

What happened to Revolut on 20 October 2026

On the morning of Tuesday 20 October 2026, Revolut customers in several markets found their cards declined, their ATM withdrawals blocked, and their app unable to show a balance. The disruption began around 06:00 UTC, when a power failure took out part of the Google Cloud infrastructure Revolut uses in London. Revolut says customer funds and data stayed safe throughout.

On 13 November the company published a technical overview that is more candid than most incident write-ups. Its most consequential sentence is easy to skim past: some routine server replacements had failed to restore all five members of certain coordination groups before the outage, and no alert told anyone. The power cut triggered the outage, but the conditions for it had been put in place weeks earlier.

I read Revolut’s account alongside Google’s final incident report and the public documentation of the software patterns involved. The result is an unusually clean case study. Two independent systems, one physical and one logical, failed in the same way. In each, redundancy that existed on paper had been quietly used up by a maintenance state, and a sensing layer reported health while measuring the wrong thing. Below I reconstruct both failures and the recovery mechanics that turned a six-hour infrastructure event into an eleven-hour banking outage.

Revolut outage timeline (all times UTC)

• 05:53: an underground utility cable fault kills feed A at the London facility hosting europe-west2-b.
• 06:00: UPS batteries run out and the data halls go dark.
• 07:20 to 07:45: power returns to the halls, then to network devices.
• 10:13 to 12:08: Google’s storage recovers, traffic is re-admitted from 10:45, APIs are fully available at 11:11, and 99.5% of Compute Engine VMs are healthy by 12:08.
• 12:40: Revolut card authorisations and in-store payments work again.
• 14:03: Google closes its incident, putting the main impact at six hours and eight minutes.
• 15:45: Revolut app access, balances, and transfers begin returning.
• 17:15: the last Revolut services are restored, a little over eleven hours after the lights went out.

Root cause part one: the Google Cloud power failure in London

Google’s final report, published on 4 November, describes the physical chain in detail. The data centre facility hosting europe-west2-b normally runs a 2N utility power design. It has two independent feeders and a central high-voltage generator bank managed by dual-redundant programmable logic controllers (PLCs), backed by UPS batteries sized for roughly five minutes of autonomous runtime.

Since 14 October, when a quarterly generator load test had surfaced problems, the facility had been in a temporary single-feed maintenance posture. Feeder B’s breaker was physically open and locked out for repairs, so feed A carried the entire critical load. At 05:53 UTC on 20 October, an external underground cable fault on the utility network killed feed A.

The generators should have started, and they did not. The PLC undervoltage sensing inputs were wired upstream of the isolated feeder breakers, on the line side. Feeder B was still energised upstream of its open breaker, so both master PLCs, running identical logic, sensed live voltage on a path that could not deliver power. They concluded a healthy utility source was available and never issued a start command. A remote load transfer attempted by the onsite team was inhibited for the same reason. The UPS batteries carried the load until they were exhausted, and at 06:00 UTC the data halls went dark, taking hosts, local storage, and network aggregation hardware with them.

Why Google’s recovery took hours

Restoration was deliberately slow. Staff manually transferred the facility to a standby utility source. The storage fleet had suffered unclean shutdowns at scale, and responders chose not to restore services in ways that would make storage recovery worse. Google’s published sequence runs from power at 07:20 and network at 07:45, through storage recovery at 10:13 and controlled traffic re-admission from 10:45, to fully available APIs at 11:11 and healthy VMs by 12:08.

Revolut’s remaining services were not back until 17:15, and the gap between those two clocks is where the interesting engineering sits. The platform was substantially recovered by early afternoon. The extra hours went into the stateful layer, on decisions about which database replica could safely accept writes.

Two more details from the reports belong side by side. Google says it found no impact outside the europe-west2 region, yet Revolut’s early information suggested several regions might be affected, including its disaster-recovery region. With parts of its own monitoring offline, Revolut needed time to establish the true blast radius, and only once the failure was confirmed as a single-zone event did it begin its single-zone recovery plan in the surviving zones. The facility’s failure also mirrors everything that follows: a degraded configuration plus a sensor reading the wrong side of a breaker has the same shape as a degraded quorum plus a monitor that never noticed the missing member.

Root cause part two: how a missing quorum member froze the databases

Revolut’s affected databases run as replicas across three London zones, labelled A, B, and C in its post, with an additional replica in another region for disaster recovery. A failover manager installed on every replica consults a distributed configuration store (DCS) to decide which replica may act as primary. The coordination servers operate in groups of five spread across the three zones, and at least three must agree, a quorum, before the group can record a decision such as which replica holds the primary role.

The DCS exists to make promotion safe, and speed is secondary. In this class of architecture the primary holds a leader record with a short time-to-live that it must keep renewing. A replica can take that record only through an atomic operation the store guarantees will succeed for at most one contender. That guarantee stops two replicas from both believing they are the writer, which in a payments ledger would mean double-spent balances and divergent histories.

Revolut’s overview does not name the failover manager or the DCS software. The design it describes is the standard shape of Patroni, the open-source PostgreSQL high-availability system, so the mechanics below come from Patroni’s and etcd’s public documentation and make no claim about Revolut’s internals.

Quorum arithmetic: why four of five is dangerous

A majority-quorum group of n members needs floor(n/2)+1 votes. Five members tolerate two failures, while four members still need three votes and tolerate only one. The etcd documentation tables this explicitly.

In the affected Revolut groups, one server was already offline before the power cut, left behind by server replacements that had not completed cleanly. Four of five members remained, which is enough to keep serving, so customers saw nothing. Then Zone B, which held two of the surviving members, lost power. Two votes remained out of the three required. The group could no longer elect, record, or authorise anything, and the failover manager could not promote a new primary.

Zone placement: 2-2-1 versus 3-1-1

Placement is the other half of the arithmetic, and five nodes across three zones gets subtle. The only possible distributions are 2-2-1 and 3-1-1. A healthy 2-2-1 group survives the loss of any single zone, because the worst case leaves three members. A healthy 3-1-1 group does not survive the loss of its heavy zone, because two members remain. Degrade either layout by one missing member and a zone holding two of the survivors becomes fatal.

Revolut admits its deployment process favoured one zone when placing the first server of each group. Its remediation changes that placement rule and verifies, as part of zonal failover exercises, that every group can still operate if any one zone goes offline.

Google’s report shows the same lesson at smaller scale in its own product. Apigee X is designed to survive a single-zone loss because its datastore keeps three replicas across the region’s three zones. A scheduling misconfiguration had put two of the three replicas in the zone that lost power, so the datastore lost quorum and the runtime failed requests instead of failing over, returning elevated 5xx errors for more than seven hours. Google’s fix mirrors Revolut’s: audit and rebalance the fleet, alert when replica placement drifts across zones, and strictly enforce that the three replicas run in three different zones. When the cloud provider and the bank independently find the same failure, replica placement deserves to be treated as a correctness constraint and not a scheduling preference.

Why quorum groups degrade in the first place

Replacing a member of a quorum group is itself a consensus operation, and the etcd documentation is blunt about the danger. Remove the dead member before adding its replacement, because adding a member to a degraded cluster raises the quorum threshold and can tip the group into permanent quorum loss if the new node fails to join. By default etcd rejects reconfigurations that would obviously destroy quorum. Routine replacements are exactly the operations most likely to leave a group at four of five, and a cluster at four of five is fully available and one failure away from paralysis. That is the state Revolut’s monitoring failed to surface.

Why the PostgreSQL clusters went read-only without a DCS

It is tempting to read “the failover manager couldn’t promote a new primary” as a story about idle replicas. The consequence runs both ways. In a Patroni-style cluster, a primary that cannot renew its leader record before the TTL expires demotes itself to read-only. From a single node, a dead DCS looks identical to a network partition in which another replica might legitimately acquire the lock and promote, and demotion is the price of never having two writers.

So when the DCS group loses quorum, surviving replicas cannot acquire a lock the store cannot grant, and any primary depending on that group cannot renew the lock it holds. Writes freeze even though the PostgreSQL processes themselves are healthy.

Patroni failsafe mode and its limits

Patroni has an escape hatch for DCS outages. With failsafe_mode enabled, a leader that fails to update its lock in the DCS may stay primary if every known member of the cluster acknowledges it over the Patroni REST API. If even one member fails to respond, the primary demotes. The documentation requires all members and not a quorum because the DCS’s view of majority and the database cluster’s view of reachability can differ, and only a unanimous check rules out a new leader being elected elsewhere.

A zonal power loss defeats this mechanism, because replicas in the dead zone can never acknowledge. Failsafe mode covers the coordination layer failing alone. It cannot cover the coordination layer and part of the data layer failing together.

etcd disk latency and unclean storage recovery

The consensus layer is also more fragile under power events than its diagrams suggest. etcd persists every proposal to its write-ahead log before it can commit, and its documentation says leader loss from disk latency spikes is intentional. A leader that takes a second to fsync while the election timeout is one second is effectively unavailable, because it cannot commit anything. The defaults, a 100 ms heartbeat interval and a 1000 ms election timeout, assume healthy disks and low-latency networks. Operational guidance asks for p99 wal-fsync latency under 10 ms and backend-commit latency under 25 ms.

A zone recovering from unclean storage shutdowns, with Google deliberately throttling restoration to protect data, is where those assumptions collapse and replication lag balloons. Revolut says it is working with Google to understand a suspected infrastructure “brownout” and how it affected database recovery. Google’s published sequence describes a hard power loss after UPS depletion followed by a long, gated storage recovery, and the two accounts are best read as different phases of the same event. Either way, the condition of the storage layer, along with the membership count, shaped how slowly the databases could safely return.

Replication lag: why restoring quorum did not end the outage

Revolut’s report says that after DCS quorum was restored, some PostgreSQL clusters stayed read-only because replication lag prevented automatic failover. Engineers had to verify replica state and choose a safe recovery path. This is the failover manager behaving the only way a bank can tolerate.

PostgreSQL replicas stay current by streaming the write-ahead log (WAL) from the primary and replaying it. Every position in the log has a log sequence number (LSN), and a replica’s replayed LSN is the ground truth of what it holds. With asynchronous replication, the default, a commit is acknowledged to the client before any replica has necessarily received it, so promoting a lagging replica silently discards acknowledged transactions.

Patroni limits this exposure with maximum_lag_on_failover, the maximum WAL bytes a follower may lag and still enter a leader election. It defaults to 1 MiB, and the documentation notes that the true worst case also includes whatever was written during the last TTL window before the failure. When every candidate exceeds the threshold, the correct behaviour is to refuse promotion and wait for a human, which is what Revolut’s engineers were doing.

Synchronous replication and quorum commit in Patroni

Stronger configurations show what safety costs. In Patroni’s synchronous_mode, the system keeps a /sync key in the DCS naming the leader and the synchronous standbys. It enforces the rule that a node that is neither the leader nor a current synchronous standby may not promote itself automatically. With synchronous_mode_strict, the primary blocks writes entirely when no synchronous standby is available, instead of silently reverting to asynchronous commits.

The quorum-commit variant keeps in the DCS the set of voters and a quorum count chosen so that any subset of voters able to reach quorum includes at least one node holding the latest acknowledged commit. A candidate’s LSN is compared against at least that many other voters before promotion is allowed. These mechanisms turn failover from a gamble into a proof, and they share one corollary: when no replica can be proven current, the cluster stays read-only.

The Datadog gameday: the same failure mode in practice

Revolut is not the only company to hit this. Datadog’s engineering team described a gameday in June 2026 in which a simulated zonal network-latency event left its Patroni-managed PostgreSQL clusters in exactly this state. The primary kept accepting writes and the replicas fell behind. When failover was needed, no standby was within maximum_lag_on_failover, so Patroni correctly refused and the clusters were stuck.

Their remedy was to make failover candidates synchronous, with strict mode and remote_apply commit durability. They accepted measured costs of roughly 53% higher write latency and 34% lower throughput in pgbench testing, and added an alert for when no synchronous standby exists.

Manual recovery with LSNs and pg_rewind

The manual path is well documented too. Compare LSNs and timelines across replicas, then choose a candidate that can be proven safe or consciously accept a bounded loss. Use pg_rewind to reconcile the diverged former primary onto the new timeline before it rejoins. That is the work inside Revolut’s phrase “verify replica state”, and it explains why database recovery trailed platform recovery by hours.

Why Revolut did not fail over to its disaster-recovery region

Revolut’s own account gives a partial answer. Early reports suggested the backup region might also be affected, and once the failure was confirmed as single-zone, the company followed its single-zone recovery plan in Zones A and C. The general mechanics add context without speculation about Revolut’s configuration. Cross-region replicas in this class of architecture are typically asynchronous, so promoting one is a data-loss decision and not a routine zonal move. It is the last resort for a regional outage and the wrong first response to a zonal one.

Why Kubernetes could not fix the database problem

Revolut notes that Kubernetes separately reschedules affected applications onto healthy servers, and for stateless services that is largely the end of the story. For databases, the orchestrator’s guarantees run out quickly.

Zonal Persistent Disk and Hyperdisk volumes carry zone affinity and cannot be mounted from another zone. Regional Persistent Disk replicates synchronously across two zones, but Google’s report says its failure detection struggled with this event’s “atypical sequencing”. That delayed Cloud SQL HA failovers and left a subset of AlloyDB instances waiting for manual recovery. The storage a rescheduled pod needs does not necessarily follow the pod.

Identity guarantees are stricter still. A StatefulSet promises at most one pod with a given identity, and to the control plane a node that has lost power looks the same as a node that is partitioned but still running and writing. Kubernetes therefore will not simply recreate stateful pods elsewhere after a hard node death. The documented remedy for a non-graceful shutdown is for an operator to confirm the node is truly gone and apply the node.kubernetes.io/out-of-service taint, which authorises force-deleting the pods and detaching their volumes so replacements can start. The manual step exists because a wrong guess produces two writers on one dataset.

PodDisruptionBudgets, often cited as protection, constrain only voluntary disruptions such as drains, and do nothing when a zone loses power. Even where rescheduling is authorised, pods need somewhere to land. Google’s report says GKE scale-ups in the surviving zones were constrained by available capacity and slowed by failing underlying APIs. In VPC-native clusters every pod draws an address from the subnet’s secondary range, so address exhaustion blocks node creation outright. Revolut’s planned warnings about cloud quotas and network-address headroom for zonal failover respond directly to this class of constraint.

Kubernetes can restart a process on a healthy node. It cannot manufacture a quorum the consensus layer has lost, and it cannot certify that a replica’s WAL contains every acknowledged commit. Those decisions belong to the DCS and the failover manager, and on 20 October both were unavailable or unconvinced. Orchestration health and database correctness are separate problems, and rescheduling solved only the first.

How card stand-in processing (STIP) kept some Revolut payments working

While the databases were frozen, some card payments still worked, because the card networks run an explicit degraded mode that the database layer lacked. Under stand-in processing, STIP on Visa and Stand-In on Mastercard, the network approves or declines transactions on the issuer’s behalf when the issuer cannot be reached. It uses parameters the issuer configured in advance, including transaction and velocity limits, risk rules, and blocked-card lists.

Visa’s own research describes the shift from purely rule-based stand-in, which it documents as producing materially lower approval rates during issuer outages, toward an AI-based decisioning model called Smarter STIP. Revolut confirms that backup processing let some card payments through during the outage.

The bill for that continuity arrives at reconciliation. Stand-in decisions reach the issuer afterwards as advice or store-and-forward messages, which issuers are expected to retrieve and process within days and not weeks. Some transactions the network approved would have been declined by the issuer had it been reachable, most commonly for insufficient funds, and those approvals must be reversed once the issuer’s ledger is back. Until that backlog is processed, app balances and transaction histories lag reality.

That is why some outage-period payments appeared late in the Revolut app, and why one of the company’s completed remediations is a system enhancement to process card reversals immediately after the STIP backlog is resolved. The card networks’ degraded mode is deliberate, parameterised, and rehearsed, with a defined reconciliation path. The database layer’s degraded mode that day was accidental, and its reconciliation path was improvised under time pressure.

Monitoring failures: the alert that never fired

The most human failure in Revolut’s account is also the most familiar. The primary instance for visualising database metrics sat in Zone B. When Zone B died, engineers lost the instrument panel they would normally use to see which replicas existed, where they were, and how far behind they ran. They were left with logs, command-line utilities, and ad hoc standalone monitoring instances built in the surviving zones before remediation could fully begin. Every minute spent rebuilding observability was a minute not spent on recovery, and it came on top of the earlier failure of the alerting layer to notice, while there was still time to act, that some coordination groups were running at four of five members.

Google’s report shows the same weakness in its own stack. AlloyDB regional primaries saw delays of up to 95 minutes in starting automated failovers because the failover notifications themselves were delayed. The remediation is to make zonal-outage detection reliable enough to send those notifications. Detection is part of the failover path, and sits inside it.

Revolut’s corresponding commitments cover both halves of the problem: end-to-end monitoring of the alerting pipelines to catch failures in alert generation and delivery, and database monitoring infrastructure that survives a single-zone outage. A pattern worth borrowing here is the watchdog, a synthetic alert that fires continuously, where its absence at an external receiver is itself the alarm. A dead pipeline then pages someone instead of silently swallowing every real alert. An alerting system that cannot report its own death is a hope and gives no protection.

Revolut’s remediation plan

As of 13 November, Revolut’s completed items are the root-cause fix and resolution of the underlying coordination-server degradation, a full root-cause analysis, the STIP reversal enhancement, and zone-resilient database monitoring.

The in-flight programme is more interesting because it changes what the system treats as true. Comprehensive monitoring of DCS cluster membership, voter health, and cluster failure tolerance is the core of it, and it moves the alerting target from availability to remaining margin. A group at four of five is available, and the old monitoring was satisfied. The new monitoring has to fire because the group’s tolerance has dropped to one failure while a single zone can take two.

Placement rebalancing turns the 2-2-1 versus 3-1-1 arithmetic into an enforced constraint checked by the zonal failover exercise, replacing an emergent property of a deployment script that favoured one zone. More frequent failover tests now cover the conditions actually observed, and the exercises will check whether services can be restored under load. That separates a failover that works in a quiet staging window from one that works at banking traffic on a Tuesday morning.

The supporting changes fit the same direction. Proactive warnings on cloud resource quotas and network-address headroom acknowledge that a zonal failover concentrates a whole zone’s demand into two survivors, as Google’s capacity-constrained GKE scale-ups showed. More resilient tooling with less manual intervention goes after the diagnostic bottleneck directly. The test of the whole programme is whether the drills reproduce the compound failure of degraded membership, a zone loss, and blinded tooling, and go beyond the clean single-zone kill the architecture was already built to survive. Revolut says the extended tests cover the observed conditions. The published programme has the right shape, and its value will depend on how adversarially it is run.

Lessons for teams running stateful systems across availability zones

Reduce the event to its parts and the economics are stark. A server replacement that did not complete cost nothing at the time, yet it consumed half of a coordination group’s failure tolerance. An alert that did not fire also cost nothing, and it ensured nobody knew. Then a cable fault outside Google’s facility, an everyday utility event, collected the debt.

The facility’s generators failed for the same structural reason. A maintenance posture had removed redundancy, and a sensor wired to the wrong side of a breaker reported the missing redundancy as present. In both systems the diagrams were correct and the runtime was not.

For anyone running stateful systems across availability zones, the takeaways are concrete:

• Quorum is a property of the running system at this instant, and an architecture drawing does not guarantee it. It decays through ordinary operations such as replacements, drains, and failed joins, so something has to measure it continuously and raise an alarm.
• Replicas in three zones and a disaster-recovery copy in another region do not make a database resilient if the layer that authorises promotion has lost its majority.
• Treat zone placement of quorum members as a correctness constraint, and alert on remaining failure tolerance and not only on availability.
• Run a watchdog on your alerting pipeline, and keep monitoring outside any single zone.
• A failover manager that refuses to promote a lagging replica is working correctly. It is the last component still telling the truth.

Revolut’s outage ended at 17:15 UTC because engineers restored the conditions under which that truth could be acted on. The remaining work is to make the system notice its own decay long before a power cut asks the question.