Your micro-benchmark reports an absurdly fast time. What did the optimiser probably do, and how do you write a benchmark that cannot be fooled?
Predict first
What happens to a loop whose result is never used, in a -O build?
import Foundation
@inline(never) func sink(_ x: UInt64) { if x == 42 { print("never") } }
func ms(_ f: () -> Void) -> Double {
let t = DispatchTime.now(); f()
return Double(DispatchTime.now().uptimeNanoseconds - t.uptimeNanoseconds) / 1e6
}
func lcg(_ n: Int) -> UInt64 { var x: UInt64 = 1; for _ in 0..<n { x = x &* 6364136223846793005 &+ 1442695040888963407 }; return x }
let n = 50_000_000
let unused = (0..<3).map { _ in ms { _ = lcg(n) } }.min()!
let used = (0..<3).map { _ in ms { sink(lcg(n)) } }.min()!
print("loop whose result is discarded is more than 50x faster:", unused * 50 < used)- loop whose result is discarded is more than 50x faster: true
- The two loops take about the same time
- The discarded loop is slightly slower
Mental model. Timing a runner who was teleported to the finish line. The stopwatch is accurate; the race never happened.
In an optimised build the compiler deletes work whose result is never used. A loop that computes a value and throws it away has no side effects, so it can be removed entirely. In the check, a 50-million-step loop took 0 ms when its result was discarded and tens of milliseconds when the result was passed to a function the optimiser cannot see through (@inline(never)). The same program built with -Onone takes about three seconds either way, which is the other half of the trap: Debug builds measure something your users never run. A test that is too fast to be physical, or that does not change when you make the code slower, is usually measuring nothing.
A trustworthy harness does four things. It builds in Release. It consumes every result through a sink the optimiser must respect (an @inline(never) function, a global accumulator, or withExtendedLifetime), and checks the answer is correct. It feeds inputs the compiler cannot constant-fold (read from the process, a file or a seeded generator, not literals). And it repeats: take the minimum or median of several runs, discard the first (caches, lazy initialisation), and report spread (Q428). Put hot-path work in a measure block with a baseline (Q430), and confirm any win in Instruments on a device, because a micro-benchmark that is fast for one input shape may not matter in the app.
The optimiser removes side-effect-free work whose result nobody reads.
Common traps
- Benchmarking in Debug. The numbers can be tens or hundreds of times off and the ranking of two options can flip.
- Ignoring the result of the code under test. If nothing reads it, the optimiser may delete it.
They might follow up with
- Can the optimiser also make a benchmark look too slow? Yes: constant inputs can be folded away or, on the other side, an unrealistic cache-hot loop hides real memory costs.
- How do I check what ran? Look at the profile or the disassembly, or change the input size and see whether the time moves.
Related questions
- 21.8.2 How much slower is a Debug (-Onone) build than Release (-O), why, and what does that mean for profiling and…
- 21.8.3 How do you find out how an operation really scales, and which Swift operations that look cheap are quadratic?
- 21.1.2 How do you measure performance reliably: what makes numbers lie, and why do percentiles beat averages?
- 21.2.2 How do signposts and Points of Interest work, and how do you turn a measurement into an XCTest performance…
How much slower is a Debug (-Onone) build than Release (-O), why, and… →