22.7 · Flakes, hermetic CI, and a quality system for a large team

A test suite is only as valuable as people’s trust in a red result. Flakes destroy that trust quickly, and at team scale the problem becomes organisational as much as technical.

Mental model. Every red build should mean something. Make tests hermetic so they fail for one reason, and organise the suite so feedback is fast where it matters.

Reliability

Where flakes come from, and what a hermetic CI run looks like.

1. Why do tests flake, how do you find and fix them, and what makes a CI test run hermetic? 22.7.1 · L5
   A flaky test passes and fails on the same code. The causes are usually one of a few: time (real sleeps, timeouts too tight, dates and time zones), ordering and…

At scale

Tiers, ownership, coverage as a signal and release gates.

1. How do you design a quality system for a large team: test tiers, ownership, coverage, flake budgets and release gates? 22.7.2 · L8
   At scale testing is a system with goals, owners and feedback loops, not a folder of tests. Start from what the team needs: fast feedback to developers, trust…

More in Testing

• 22.1 · The pyramid, and what not to test
• 22.2 · XCTest and Swift Testing
• 22.3 · Async tests, expectations and controlling time
• 22.4 · Fakes, mocks and the inputs that make tests random
• 22.5 · Snapshot tests and UI tests
• 22.6 · Testing SwiftUI, tvOS focus, and watchOS logic
• 22.8 · Swift Testing in depth: arguments, traits, errors, confirmations, isolation and scoping
• 22.9 · XCTest lifecycle and expectations, Sendable doubles, and the main actor in tests