How do you measure performance reliably: what makes numbers lie, and why do percentiles beat averages?
Predict first
Ten latency samples, nine fast and one slow. What do the mean, median, 90th percentile and maximum report?
import Foundation
let samples: [Double] = [12, 11, 13, 12, 14, 11, 12, 13, 12, 480]
let sorted = samples.sorted()
func percentile(_ p: Double) -> Double { sorted[Int((p * Double(sorted.count)).rounded(.up)) - 1] }
let mean = samples.reduce(0, +) / Double(samples.count)
let median = (sorted[4] + sorted[5]) / 2
print(mean, median, percentile(0.9), percentile(1.0))- 59.0 12.0 14.0 480.0
- 12.0 12.0 14.0 480.0
- 59.0 59.0 480.0 480.0
Mental model. Averaging commute times with the one day a train broke down gives a number that matches no real morning. The median tells you the usual one and the p95 tells you how bad a bad day is.
Measurements lie for boring reasons. Debug builds are unoptimised, so Swift code can be many times slower than the Release build users run. The simulator uses the Mac’s CPU, memory and disk, so it hides real device limits. A cold start (first run, empty caches) differs from a warm one, so decide which one you are measuring and warm up the other. Thermal throttling and Low Power Mode change speed mid-test, and background load on a CI machine adds noise. Run each scenario several times on a real device, in a Release configuration, in the same state, and report the spread, not one number. XCTest’s measure block does this for you: it runs the block once extra to warm up and then records each iteration.
Summarise with percentiles. Latency distributions are skewed: most runs are fast and a few are very slow, so the mean is dragged up by outliers and describes nobody, while the median hides the slow tail that users notice. Report p50 (the typical experience), p90 or p95 (the unlucky one in ten or twenty) and the maximum. Field data from MetricKit and Xcode Organizer gives you the same percentiles from real devices, which is what a launch or hang goal should be written against (for example “p90 launch under one second”).
The one outlier pulls the mean to 59, while the median stays at the typical 12 and the nearest-rank p90 is 14. Always look at more than one statistic.
Common traps
- Judging from a single run. Noise of ten percent is normal; take several and compare spreads.
- Reporting the mean of a skewed distribution. A single 480 ms outlier moved it from 12 ms to 59 ms below.
They might follow up with
- Cold versus warm? Cold is what a new user sees after a reboot or an eviction, warm is a relaunch; measure the one your goal is about.
- Why does the first XCTest iteration not count? It pays one-off costs (code loading, lazy initialisation) that would skew the result.
Related questions
- 21.1.1 How do you approach a performance problem, and which kinds of problem are there?
- 21.2.2 How do signposts and Points of Interest work, and how do you turn a measurement into an XCTest performance…
- 21.3.1 What does MetricKit give you in production, how do you subscribe, and what is the difference between metric…
- 6.4.3 Your app launches slowly. How do you find out why and what do you cut?
← How do you approach a performance problem, and which kinds of problem…