RAW DATA

The scripts, the corpus, and every line of output

A measurement you cannot rerun is an opinion with a decimal point. Everything behind the pages on this site is here, in the form it was produced.

The machine

Apple M4, 10 cores, 16 GB RAM
macOS 26.3.1 (build 25D771280a), internal NVMe
zstd 1.5.7 · xz 5.8.3 · Apple gzip 475 · bzip2 1.0.8
SQLite 3.38.4 (as bundled with Python 3.11.0)

This is a laptop, not a server, and that matters when you read the absolute numbers: ten fast cores flatter the multi-threaded settings, and an internal NVMe flatters everything. The orderings should carry to slower hardware. The margins should not be quoted.

Scripts

• build-corpus.sh — builds the 282 MB compression corpus from stable public URLs: Wikimedia Commons photographs, a Blender Foundation video clip, the SQLite source tree, Project Gutenberg texts, and a SQLite database assembled from the OurAirports open dataset. Uses photo-urls.txt for the exact image list.
• compress-bench.py — runs thirteen codec settings against each corpus category, round-trips every result and asserts the restored size, emits JSON Lines.
• sqlite-durability-bench.py — thirty-six SQLite configurations of journal_mode, synchronous, fullfsync and transaction size, measured as insert throughput.
• make-charts.py — draws the SVG charts on the measurement pages directly from the result files. No chart library, no runtime JavaScript.
• render-pages.py — fills the tables and charts into the article templates from the result files, so a published page cannot drift away from the run it came from.

Results

• compress-results.jsonl — 65 measurements: 13 codec settings across 5 corpus categories. One JSON object per line, with raw and packed byte counts, ratio, and compression and decompression wall clock and throughput.
• sqlite-results.jsonl — 36 measurements, one JSON object per line, each stating the row count it was measured over.

Both files are JSON Lines, which means jq reads them without ceremony:

curl -s https://iotlinefair.com/bench/compress-results.jsonl \
  | jq -r 'select(.record=="measure" and .corpus=="photos")
           | "\(.codec)\t\(.ratio)"'

Reproducing it

curl -O https://iotlinefair.com/bench/build-corpus.sh
curl -O https://iotlinefair.com/bench/photo-urls.txt
curl -O https://iotlinefair.com/bench/compress-bench.py
chmod +x build-corpus.sh compress-bench.py
./build-corpus.sh corpus          # about 280 MB of downloads
./compress-bench.py corpus > my-results.jsonl

The corpus builder is idempotent and resumes, so an interrupted download can be restarted. It identifies itself in its User-Agent, as Wikimedia asks clients to; anonymous bulk fetches there get throttled to a stall.

The SQLite benchmark needs nothing but Python:

curl -O https://iotlinefair.com/bench/sqlite-durability-bench.py
python3 sqlite-durability-bench.py 20000 > my-sqlite-results.jsonl

Licence

The result files and the scripts are placed in the public domain under CC0. Use them, quote them, contradict them. The corpus contents belong to their own authors under their own licences — Wikimedia Commons images under their individual terms, the video clip under CC-BY to the Blender Foundation, the Gutenberg texts under the Project Gutenberg licence, the SQLite source in the public domain, and the OurAirports data as published by OurAirports.

If your numbers differ

They probably will, and that is the interesting case rather than a problem. Different hardware, a different corpus mix, a newer codec build — any of those moves the figures. Send what you got to hello@iotlinefair.com and it will be published alongside, with a correction to the original page if the original page was wrong.