How do you reduce disk and database cost: batching writes, transactions, atomic writes, memory mapping and the system’s disk-write limit?

21.5.2 · L5 · Interview

Predict first

Insert 500 rows into SQLite one statement at a time, then in a single transaction. Is the single transaction at least five times faster?

import Foundation
import SQLite3

func ms(_ f: () -> Void) -> Double {
    let t = DispatchTime.now()
    f()
    return Double(DispatchTime.now().uptimeNanoseconds - t.uptimeNanoseconds) / 1e6
}
func insert500(batched: Bool) -> Double {
    let path = NSTemporaryDirectory() + "perf-\(batched).db"
    try? FileManager.default.removeItem(atPath: path)
    var db: OpaquePointer?
    sqlite3_open(path, &db)
    sqlite3_exec(db, "CREATE TABLE t(a INTEGER)", nil, nil, nil)
    let t = ms {
        if batched { sqlite3_exec(db, "BEGIN", nil, nil, nil) }
        for i in 0..<500 { sqlite3_exec(db, "INSERT INTO t VALUES (\(i))", nil, nil, nil) }
        if batched { sqlite3_exec(db, "COMMIT", nil, nil, nil) }
    }
    sqlite3_close(db)
    return t
}
let each = insert500(batched: false), once = insert500(batched: true)
print("One transaction more than 5x faster:", each / once > 5)
  • Yes, by a large factor
  • No, they are about equal
  • No, the transaction is slower

Mental model. Mailing five hundred letters one trip at a time versus one bag. The letters are the same; the trips are the cost.

Disk cost is dominated by how many separate operations you issue and whether each one waits for the storage to confirm. Writing one record at a time is far slower than batching: in SQLite (and so Core Data and SwiftData underneath) each statement outside a transaction is its own durable commit, so 500 inserts become 500 commits, while wrapping them in one transaction commits once, which the check below measures at hundreds of times faster. The same idea applies elsewhere: save the Core Data context once per batch rather than per object, use batch insert and batch delete requests for bulk changes, coalesce rapid updates (a text field’s edits, location points) into one write after a short delay, and write a file once rather than appending in many small calls.

Other levers: use atomic writes (Data.write(to:options:.atomic) writes a temporary file and renames it) so a crash cannot leave a torn file, at the cost of an extra file operation, so do not use it for scratch data; open large read-only files with .alwaysMapped (memory mapping) so pages load on demand instead of copying everything into memory; keep large blobs out of a database and store a file path; and keep I/O off the main thread. The system also watches write volume: an app that writes an excessive amount in a day gets a disk-write exception diagnostic in MetricKit (and flash wear matters on a phone that lives for years), so log or analytics code that writes constantly is a real cost. Find the offenders with the Time Profiler, the File Activity and Disk instruments, and XCTStorageMetric.

Each autocommitted insert waits for the storage to confirm durability; one transaction pays that cost once.

Common traps

  • Saving a context or committing a statement inside a loop. Each iteration pays for a durable write; save after the batch.
  • Writing a log or analytics line to disk on every event. It adds up to write volume the system will flag; buffer and flush in batches.

They might follow up with

  • What does .atomic protect against? A crash during the write leaving a half-written file; the old file stays until the new one is complete.
  • Why map a big file? The system pages it in lazily and can drop the clean pages, so footprint stays low (Q162).

Related questions

  • 21.5.1 Which everyday operations are surprisingly expensive in Swift and iOS code, and how do you find and fix them?
  • 21.3.1 What does MetricKit give you in production, how do you subscribe, and what is the difference between metric…
  • 21.2.2 How do signposts and Points of Interest work, and how do you turn a measurement into an XCTest performance…
  • 3.7.5 What is memory footprint, and what gets an iOS app killed?

← Which everyday operations are surprisingly expensive in Swift and iOS…