Circuit Breaker.
Give a failing dependency room to recover. Keep the rest of the system moving.
Change one thing at a time.
The interesting part is usually why it moved.
Checkout failure experiment
The payment provider still accepts requests, but each call holds a worker for three more seconds. Four in five calls eventually fail. Choosing a situation starts a fresh, paused run.
Introduce the failure and watch the next few seconds.
The checkout path has capacity to spare.
Start with a healthy provider and 24 incoming requests per second. The readouts show the baseline.
Why this happens
Healthy calls finish in the same simulated second they arrive, leaving workers available for the next requests.
Try this next
Introduce failure, then follow the payment call and the order service’s in-flight requests.
The trade-off
A healthy baseline tells you little about how the caller behaves when a dependency slows down or fails.
Tune the circuit breaker
A lower threshold reacts sooner. A longer cooldown sends fewer probes, but delays the discovery of recovery. Success resets the consecutive-failure count.
Introduce a breaker to observe the state machine.
- System ready. Requests are flowing normally.
One step represents one simulated second. Response throughput includes fallbacks; it is not a count of completed payments. The setup link saves your settings, and opens a fresh run.
SYSTEM OPERATIONAL. Circuit not installed. Payment dependency is healthy.
Slow remote calls occupy resources upstream. One unhealthy dependency can exhaust an otherwise healthy service.
A synchronous request chain with bounded timeouts and an optional, explicitly degraded fallback.
What this buys, and what it charges you for.
ResilienceHigher is preferable↑ 55
A failed dependency is isolated before it consumes all upstream capacity.
LatencyLower is preferable↓ 66
Fast rejection or fallback lowers waiting time during this outage; it does not make the dependency faster.
AvailabilityHigher is preferable↑ 39
Fallback keeps a limited response available. The original operation still fails.
Implementation complexityLower is preferable↑ 27
A state machine, timeout policy and fallback contract must be maintained.
Illustrative scores for this scenario, on a 0–100 scale. They express a direction of change, not a benchmark. Your constraints can change the result. Select a quality to explore the reasoning.
+ What improves
- Reject predictably failing calls quickly.
- Protect upstream capacity during an outage.
± What gets harder
- Some calls are rejected even after the dependency recovers.
- Thresholds and fallback semantics need deliberate design.
- Deciding what a degraded answer means to a customer is a product decision, not only a technical one. Someone has to own it before the breaker is useful.
Where the edges are.
A pattern is only useful once you know where it stops working. These are the boundaries worth knowing before you commit to it.
When it earns its place
- A slow remote dependency threatens shared request capacity.
When to leave it out
- For local function calls, or as a replacement for timeouts.
- When every failed call is already fast and cheap. A breaker mainly protects capacity that slow calls are holding.
New ways to fail
- Aggressive probes can overwhelm recovery.
- An overly broad breaker can isolate healthy operations.
Operating the pattern
- Track state changes, rejected calls, probe outcomes and fallback use.
The people behind the system
- Service owners must agree on acceptable degraded behavior.
Other directions to consider
- Bounded retries with jitter
- Bulkheads and concurrency limits
- Asynchronous work queues