Zone-Failure Resilience Without Shard Ownership

Uber recently described how it made OpenSearch resilient to the loss of an entire availability zone. Its design combines three isolation groups, at least three copies of every shard, forced shard-allocation awareness, and five cluster-manager nodes.

It is a thoughtful solution to a difficult problem: how do you keep a stateful, shard-owning search cluster available after losing a zone—and then another node?

At Mach5, we started with the same failure requirement but asked a different question:

What if query nodes did not own the durable data at all?

That changes how the system recovers, how it operates, and how its cost grows with retained data and query demand.

The guarantee

We use the same failure sequence described by Uber:

1. One complete availability zone becomes unavailable.
2. One additional node fails in a surviving zone.
3. Querying and ingestion continue.
4. Acknowledged data is not lost.
5. Recovery requires no emergency operator intervention.

The surviving deployment also retains enough compute to satisfy its declared latency and ingestion SLOs.

Uber's OpenSearch approach

Uber's OpenSearch design uses three isolation groups mapped onto physical failure domains. Every index has at least two replicas, creating three total copies of every shard. Shard-allocation awareness places those copies across the three isolation groups.

When a zone fails, forced allocation awareness prevents OpenSearch from recreating the missing replicas in the two surviving zones. That avoids a sudden wave of disk, network, and CPU activity that could destabilize the remaining cluster.

Uber also uses five cluster-manager nodes. After losing two managers with a zone, the voting configuration contracts to the three survivors. The cluster can then tolerate one additional manager failure.

This is strong engineering for a system in which data nodes own shards. That ownership also determines the system's scaling and recovery behavior:

• Durable data is replicated at the application layer.
• Retaining more data requires more block storage and data nodes.
• Changing node types can require shard relocation.
• Recovery is coupled to replica placement.
• Workload isolation means sharing a heterogeneous cluster or creating additional clusters.

Read Uber's original article.

Mach5 moves the state boundary

Mach5 separates durable data from query compute.

Authoritative segments live in S3 Standard, which stores objects across at least three availability zones and is designed to sustain the loss of one zone. Durable metadata is also replicated across three failure domains.

Query nodes hold replaceable execution state and local caches. Their NVMe contents improve performance, but losing them does not lose authoritative data.

For an important query path, Mach5 provisions six replicas—two per availability zone:

The deployment is sized so those three survivors can continue satisfying the failure-state workload.

Mach5 fails over by rerouting requests and warming replaceable caches, not by repairing or relocating durable shards.

No recovery storm

In a conventional OpenSearch cluster, the disappearance of shard replicas creates missing durable state. Uber uses forced awareness specifically to prevent OpenSearch from immediately recreating that state in the surviving zones.

In Mach5, losing a query node does not create missing durable state. S3 remains authoritative. The failure sequence is therefore:

1. Kubernetes and the load balancer remove failed pods.
2. In-flight idempotent queries retry against surviving replicas.
3. Surviving nodes continue reading the same segment objects.
4. The degraded-mode scheduler launches replacement nodes across the two live availability zones.
5. New query replicas enter service immediately by reading from S3, initially with empty local caches.
6. Warm-cache performance returns progressively as the caches repopulate under explicit I/O and memory limits.

There is no shard-relocation decision and no cluster-wide replica-repair traffic.

This produces predictable failure behavior. The durable data remains available, replacement capacity can be added without copying it, and recovery does not compete with production queries for the resources required to rebuild shards.

A different data-cost curve

Let D be the compressed size of one indexed copy.

S3 performs its own internal replication, but the customer pays for one logical object copy rather than three separately provisioned application replicas.

Using an AWS configuration with gp3 storage, 70% maximum OpenSearch disk utilization, and storage-driven data-node compute, the marginal retention costs are approximately:

As the data under management increases, the difference compounds:

Query compute is additional in both cases. The distinction is that Mach5 does not add query nodes merely because the retained dataset grew.

Warehouses make compute specialization practical

Separating data from compute also allows Mach5 to create multiple warehouses over the same durable dataset. Each warehouse has its own instance families, processor architecture, replica count, failure-state capacity, memory limits, concurrency, cache policy, scaling behavior, query admission, priority, and availability SLO.

All these warehouses read the same S3 segments. Creating another warehouse does not create another durable copy of the dataset.

This removes two expensive compromises:

1. Provisioning every node for the most memory-intensive query.
2. Allowing scans and investigations to evict the cache or consume the concurrency budget of latency-sensitive dashboards.

A batch warehouse can use a different node family and run for only part of the day. The interactive warehouse remains warm and continuously available. Neither needs to be provisioned for the other's peak.

Compute arbitrage without data movement

Because query nodes do not own durable shards, Mach5 can continuously evaluate different EC2 options:

• x86 versus Arm.
• Compute-, general-, or memory-optimized families.
• Local NVMe versus EBS cache.
• New versus older instance generations.
• On-Demand guaranteed capacity versus diversified Spot capacity.
• Different instance families for different warehouses.

The selection metric is not dollars per vCPU. Different processors deliver different throughput for the same nominal vCPU count.

Mach5 selects on dollars per successful query at the target p99.

That calculation includes:

• Measured throughput per node.
• Local-cache hit rate.
• S3 requests and bytes read.
• Memory pressure and spilling.
• Cache-warming cost.
• Interruption and replacement behavior.

Changing the winning instance family does not require migrating the authoritative dataset. New nodes start, warm their caches, and enter service.

Workload isolation without data duplication

OpenSearch can isolate workloads by creating separate clusters, but each cluster brings its own managers, data nodes, shard replicas, and operational lifecycle. Mach5 warehouses isolate compute while sharing durable storage and metadata.

This gives each workload an independent performance envelope:

• Batch scans cannot consume interactive concurrency.
• Large aggregations cannot exhaust dashboard memory.
• One tenant cannot evict another tenant's hot cache.
• Low-priority work can queue without increasing critical-path capacity.
• Warehouses can scale and upgrade independently.
• Cost attribution is available per workload or tenant.

Engineering the complete failure boundary

The Mach5 configuration combines separated data and compute with disciplined multi-zone deployment:

• Hard three-zone spreading in steady state, with an automated two-zone degraded mode.
• Host-level anti-affinity.
• Pod disruption budgets.
• Readiness, liveness, and startup probes.
• Durable metadata replicated across three failure domains.
• HA metadata and control services.
• Per-warehouse post-failure compute headroom.
• Diversified replacement capacity in each live availability zone.
• Protected S3 data with versioning and recovery controls.
• Controlled cache warming and query admission.

Critical warehouses use at least six replicas, distributed two per zone. Compatible warehouse pods share family-specific node pools, allowing Kubernetes to bin-pack them efficiently without weakening failure-domain placement.

Warehouses select capacity profiles rather than owning dedicated Auto Scaling Groups. This preserves workload-level guarantees without requiring six dedicated EC2 instances for every warehouse.

The larger lesson

Uber's article demonstrates how to make a shard-owning system resilient by carefully controlling replica placement and recovery. Mach5's answer is to remove durable shard ownership from the query fleet.

Once durability is separated from compute:

• Zone failure does not initiate shard recovery.
• Retention grows object storage instead of a fleet of data nodes.
• Query compute scales independently from retained data.
• Warehouses isolate workloads without duplicating the dataset.
• Each workload can use the most economical instance family.
• Compute fleets can change without migrating authoritative data.

The result is a search and analytics architecture designed to remain available through zone failure while becoming progressively more cost-efficient as data, workloads, and infrastructure options grow.