How It Works

A quick tour of what happens when you run walrust watch.

SQLite Pages

SQLite stores everything in fixed-size pages (usually 4KB). Your database is just a file full of these pages - some hold table data, some hold indexes, some hold metadata. When you write data, SQLite modifies pages.

WAL Mode

In WAL (Write-Ahead Logging) mode, SQLite doesn’t modify the main database file directly. Instead, it appends changes to a separate -wal file. This is great for replication because we can watch the WAL file and capture every change before it gets folded back into the main database.

app.db      ← main database (pages)
app.db-wal  ← changes queue (walrust watches this)
app.db-shm  ← shared memory (ignore this)

The Sync Flow

┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│  Your App   │───▶│   WAL File  │───▶│   Walrust   │───▶ S3
│  (writes)   │    │  (changes)  │    │  (uploads)  │
└─────────────┘    └─────────────┘    └─────────────┘

1. Your app writes to SQLite
2. SQLite appends to the WAL file
3. Walrust detects the change (via inotify/kqueue)
4. Walrust packages changed pages into an LTX file
5. LTX file uploads to S3

LTX Format

LTX (Lite Transaction) is a compact format for storing SQLite page changes. Each LTX file contains:

• Header - page size, page count, transaction IDs
• Page data - the actual 4KB pages that changed
• Checksum - SHA256-based integrity check

Files are named by transaction ID range:

s3://bucket/mydb/
├── 00000001-00000001.ltx   ← snapshot (TXID 1)
├── 00000002-00000010.ltx   ← incremental (TXID 2-10)
├── 00000011-00000050.ltx   ← incremental (TXID 11-50)
└── manifest.json           ← index of all LTX files

The format comes from Litestream via the litetx crate. Walrust uses it because it’s well-designed and battle-tested.

Checksums

Every LTX file includes a checksum computed from SHA256, truncated to 64 bits. On restore, walrust verifies checksums to catch corruption. If something’s wrong, you’ll know before your data is toast.

GFS Retention

Left unchecked, you’d accumulate LTX files forever. Walrust uses Grandfather-Father-Son (GFS) rotation to keep storage bounded:

Tier	Default	Keeps
Hourly	24	Last 24 hours of snapshots
Daily	7	One per day for a week
Weekly	12	One per week for 12 weeks
Monthly	12	One per month beyond that

Run walrust compact to clean up old files, or use --compact-after-snapshot in watch mode.

Restore

Restoring is the reverse:

1. Download the latest snapshot LTX
2. Apply incremental LTX files in order
3. Verify checksums along the way
4. Output a complete, consistent database

Point-in-time restore works by stopping at a specific transaction ID or timestamp.

---

That’s it. WAL changes go in, LTX files come out, S3 stores them, GFS keeps it tidy.