Methodology

How benchmarks are run and judged. Distilled from the framework's BENCHMARKS.md; fine detail lives in the pilot report and in that file.

Control hierarchy (always in this order)

  1. Same WhiteThroated path before vs after a change.
  2. Equivalent raw SwiftNIO control (transport lower bound, no framework routing).
  3. Equivalent WhiteThroated responder modes (prepared / sync / future / async).
  4. External frameworks (Actix Web reference, Express/Hono controls) only after Swift-side overhead is understood. External controls live outside the Swift repository, which stays Swift-only.

External contract routes

Every cross-framework comparison uses the same three routes with identical semantics:GET / returns an empty 2xx response; GET /user/:id returns the route parameter as the body; POST /user returns an empty 2xx response. Never compare WhiteThroated's prepared diagnostic route against another framework's dynamic route.

Concurrency gates

Run the matrix at 64, 256, and 512 concurrent connections. All three levels are independent acceptance gates: a change is not a win if peak throughput rises while another level's tail latency regresses materially.

Throughput vs latency runs

Collect separate throughput-only samples (primary RPS result) and latency-enabled samples (paired distribution: min, average, p50, p75, p90, p99, p99.999, max). The Swift benchmark client opens all connections before the timer starts, keeps one request in flight per connection, and excludes connection setup from measurement. Timestamp collection is part of the client workload, so compared servers must use the same latency setting.

What every published result records

Requests/second; min/avg/p50/p75/p90/p99/p99.999/max latency; CPU and resident memory; Swift and SwiftNIO versions; external versions when applicable; OS and CPU model; exact commands; commit SHA; warm-up, rounds, and ordering. Raw samples are archived with the record.

Evidence gates (non-negotiable)

Reproducing the pilot matrix

The pilot report reproduces its matrix with the server and client commands inBENCHMARKS.md — that file is part of the framework repository, not of this site's data pack, so the exact commands are not quoted here. The report fixes everything else (report, Network experiment):

Read the full pilot report or download the raw measurement data.