Controlled outage and recovery
An open-loop load generator exercises Kevlar against an in-process asynchronous dependency. Each composition gets fresh strategy state and the same healthy, slowdown, outage, and recovery schedule. These measurements describe the configured simulation, not production capacity or a comparison with another library.
Measured 2026-09-28T07:29:35.7634385+00:00 at commit 813cbbc0208420514e23de38e369cfe1174cd3d1.
Request outcomes and latency
Latency starts at the scheduled arrival, including scheduler delay and queue time. p95/p99 include every submitted outcome, including fast pipeline rejections; success p99 is separate. Harness rejections remain in offered/failed counts but have no submitted latency. Requests belong to their scheduled-arrival phase, even when they finish later.
| Composition | Phase | Offered | Success / failed | p95 / p99 ms | Success p99 ms | Scheduler p99 ms |
|---|---|---|---|---|---|---|
| retry | healthy | 3000 | 3000 / 0 | 6.28 / 6.83 | 6.83 | 1.14 |
| retry | slowdown | 3000 | 3000 / 0 | 301.75 / 302.16 | 302.16 | 1.14 |
| retry | outage | 3000 | 3 / 2997 | 62.63 / 63.61 | 45.85 | 1.87 |
| retry | recovery | 3000 | 3000 / 0 | 6.28 / 6.83 | 6.83 | 1.15 |
| retry-breaker | healthy | 3000 | 3000 / 0 | 6.26 / 7.18 | 7.18 | 1.87 |
| retry-breaker | slowdown | 3000 | 3000 / 0 | 301.79 / 302.15 | 302.15 | 1.20 |
| retry-breaker | outage | 3000 | 0 / 3000 | 1.21 / 21.18 | not observed | 2.00 |
| retry-breaker | recovery | 3000 | 2995 / 5 | 6.26 / 7.03 | 7.06 | 1.54 |
| limit-retry-breaker | healthy | 3000 | 3000 / 0 | 6.26 / 6.89 | 6.89 | 1.15 |
| limit-retry-breaker | slowdown | 3000 | 1600 / 1400 | 598.16 / 599.95 | 600.12 | 1.14 |
| limit-retry-breaker | outage | 3000 | 0 / 3000 | 1.12 / 20.72 | not observed | 1.12 |
| limit-retry-breaker | recovery | 3000 | 2996 / 4 | 6.26 / 6.88 | 6.88 | 1.14 |
| limit-hedge-breaker | healthy | 3000 | 3000 / 0 | 6.25 / 6.67 | 6.67 | 1.33 |
| limit-hedge-breaker | slowdown | 3000 | 3000 / 0 | 51.92 / 52.59 | 52.59 | 1.19 |
| limit-hedge-breaker | outage | 3000 | 0 / 3000 | 2.07 / 21.36 | not observed | 1.36 |
| limit-hedge-breaker | recovery | 3000 | 2984 / 16 | 6.25 / 7.04 | 7.04 | 1.22 |
Offered load and downstream amplification
Submitted means accepted by the harness. Admitted means at least one dependency attempt began. A request can be admitted and later rejected by the circuit breaker on a retry. Attempt counts below belong to logical requests in the row; JSON also records attempts by actual start phase.
| Composition | Phase | Submitted / admitted | Rejected: harness / limiter / circuit | Attempts | Attempts / offered | Attempts / admitted |
|---|---|---|---|---|---|---|
| retry | healthy | 3000 / 3000 | 0 / 0 / 0 | 3000 | 1.00 | 1.00 |
| retry | slowdown | 3000 / 3000 | 0 / 0 / 0 | 3000 | 1.00 | 1.00 |
| retry | outage | 3000 / 3000 | 0 / 0 / 0 | 8999 | 3.00 | 3.00 |
| retry | recovery | 3000 / 3000 | 0 / 0 / 0 | 3000 | 1.00 | 1.00 |
| retry-breaker | healthy | 3000 / 3000 | 0 / 0 / 0 | 3000 | 1.00 | 1.00 |
| retry-breaker | slowdown | 3000 / 3000 | 0 / 0 / 0 | 3000 | 1.00 | 1.00 |
| retry-breaker | outage | 3000 / 68 | 0 / 0 / 2993 | 84 | 0.03 | 1.24 |
| retry-breaker | recovery | 3000 / 2995 | 0 / 0 / 5 | 2995 | 1.00 | 1.00 |
| limit-retry-breaker | healthy | 3000 / 3000 | 0 / 0 / 0 | 3000 | 1.00 | 1.00 |
| limit-retry-breaker | slowdown | 3000 / 1608 | 0 / 1384 / 13 | 1616 | 0.54 | 1.00 |
| limit-retry-breaker | outage | 3000 / 56 | 0 / 3 / 2997 | 56 | 0.02 | 1.00 |
| limit-retry-breaker | recovery | 3000 / 2996 | 0 / 0 / 4 | 2996 | 1.00 | 1.00 |
| limit-hedge-breaker | healthy | 3000 / 3000 | 0 / 0 / 0 | 3000 | 1.00 | 1.00 |
| limit-hedge-breaker | slowdown | 3000 / 3000 | 0 / 0 / 0 | 6000 | 2.00 | 2.00 |
| limit-hedge-breaker | outage | 3000 / 142 | 0 / 0 / 2996 | 146 | 0.05 | 1.03 |
| limit-hedge-breaker | recovery | 3000 / 2984 | 0 / 0 / 16 | 2984 | 0.99 | 1.00 |
Recovery and hedge cleanup
Recovery time ends at the later of the end of the first fully successful 250 ms scheduled-arrival window and completion of every request in that window, measured from the recovery phase boundary. The window must include at least floor(rate × 0.25), minimum one, successful arrivals. Any harness shedding or pipeline rejection invalidates that window. Not observed means no qualifying complete window.
Cancelled attempts include cancelled hedge losers. Cleanup delay is measured after caller completion, clamped to zero. Every attempt must finish cleanup before a scenario publishes results or the next scenario starts.
| Composition | Recovery ms | Cancelled | Losers pending at caller completion | Cleanup p99 / max ms | Peak logical / downstream | Active after drain |
|---|---|---|---|---|---|---|
| retry | 250.00 | 0 | 0 | 0.00 / 0.00 | 31 / 31 | 0 |
| retry-breaker | 500.00 | 0 | 0 | 0.00 / 0.00 | 31 / 31 | 0 |
| limit-retry-breaker | 500.00 | 0 | 0 | 0.00 / 0.00 | 33 / 16 | 0 |
| limit-hedge-breaker | 500.00 | 3088 | 3088 | 21.02 / 21.89 | 7 / 12 | 0 |
Configuration and limitations
- 100 offered requests/s, 30 seconds per phase, four phases per composition, and a harness cap of 512 active logical requests.
- Arrivals keep their scheduled times when the runner falls behind; delayed arrivals catch up without waiting for prior responses. Scheduler lag exposes bursts caused by a saturated generator. High lag or harness shedding limits interpretation.
- Healthy/recovery dependency latency: 5 ms. Slowdown: first attempt 300 ms, additional attempts 30 ms. This intentionally models a slow original replica with a faster alternate; hedging will not help uniformly slow replicas.
- Outage: every attempt fails after 20 ms. An attempt uses the phase at its start, so outstanding outage attempts can fail after recovery begins.
- Cancelled attempts perform 20 ms of asynchronous cleanup before completing.
- Retry: two extra attempts with no backoff. Breaker: five consecutive handled failures, 500 ms break. Limiter: 16 permits, 16 queued. Hedge: one extra attempt after 20 ms; handled failures can launch it earlier.
- Compositions run sequentially in fixed order, with separate warmup and fresh shield state. This is a scenario demonstration, not an A–B performance acceptance experiment. Shared runners, timer resolution, GC, and process scheduling affect latency. No latency threshold gates CI.
- Percentiles use nearest rank. Short smoke runs have small samples and exist to validate accounting and cleanup. The simulator omits sockets, serialization, server queues, database locks, and real cancellation delays.
Environment and raw results
- Ubuntu 24.04.5 LTS; .NET 10.0.12; X64; 4 visible processors.
- Current published JSON accompanies this page. The outage workflow also uploads JSON and Markdown results for each run. JSON records all configuration, scheduled phase boundaries, actual phase attempt counts, and wall duration for each composition.
Reproduce
dotnet run -c Release --project benchmarks/Kevlar.OutageTests -- --phase-seconds 30 --rate 100 --max-inflight 512
python .github/scripts/outage_docs.py --input artifacts/outage/outage-results.json --output artifacts/outage/outage-results.md
Use --phase-seconds 1 for smoke validation. Scheduled/manual runs use 30 seconds per phase. For sustained throughput and allocation measurements, see stress tests.