Phase 4: Storage Deduplication

2026-02-15

When multiple tesseras share the same photo, the same audio clip, or the same +fragment data, the old storage layer kept separate copies of each. On a node +storing thousands of tesseras for the network, this duplication adds up fast. +Phase 4 continues with storage deduplication: a content-addressable store (CAS) +that ensures every unique piece of data is stored exactly once on disk, +regardless of how many tesseras reference it.

The design is simple and proven: hash the content with BLAKE3, use the hash as +the filename, and maintain a reference count in SQLite. When two tesseras +include the same 5 MB photo, one file exists on disk with a refcount of 2. When +one tessera is deleted, the refcount drops to 1 and the file stays. When the +last reference is released, a periodic sweep cleans up the orphan.

What was built

CAS schema migration (tesseras-storage/migrations/004_dedup.sql) — Three +new tables:

cas_objects — tracks every object in the store: BLAKE3 hash (primary key), +byte size, reference count, and creation timestamp
blob_refs — maps logical blob identifiers (tessera hash + memory hash + +filename) to CAS hashes, replacing the old filesystem path convention
fragment_refs — maps logical fragment identifiers (tessera hash + fragment +index) to CAS hashes, replacing the old fragments/ directory layout

Indexes on the hash columns ensure O(1) lookups during reads and reference +counting.

CasStore (tesseras-storage/src/cas.rs) — The core content-addressable +storage engine. Files are stored under a two-level prefix directory: +<root>/<2-char-hex-prefix>/<full-hash>.blob. The store provides five +operations:

put(hash, data) — writes data to disk if not already present, increments +refcount. Returns whether a dedup hit occurred.
get(hash) — reads data from disk by hash
release(hash) — decrements refcount. If it reaches zero, the on-disk file is +deleted immediately.
contains(hash) — checks existence without reading
ref_count(hash) — returns the current reference count

All operations are atomic within a single SQLite transaction. The refcount is +the source of truth — if the refcount says the object exists, the file must be +on disk.

CAS-backed FsBlobStore (tesseras-storage/src/blob.rs) — Rewritten to +delegate all storage to the CAS. When a blob is written, its BLAKE3 hash is +computed and passed to cas.put(). A row in blob_refs maps the logical path +(tessera + memory + filename) to the CAS hash. Reads look up the CAS hash via +blob_refs and fetch from cas.get(). Deleting a tessera releases all its blob +references in a single transaction.

CAS-backed FsFragmentStore (tesseras-storage/src/fragment.rs) — Same +pattern for erasure-coded fragments. Each fragment's BLAKE3 checksum is already +computed during Reed-Solomon encoding, so it's used directly as the CAS key. +Fragment verification now checks the CAS hash instead of recomputing from +scratch — if the CAS says the data is intact, it is.

Sweep garbage collector (cas.rs:sweep()) — A periodic GC pass that handles +three edge cases the normal refcount path can't:

Orphan files — files on disk with no corresponding row in cas_objects. +Can happen after a crash mid-write. Files younger than 1 hour are skipped +(grace period for in-flight writes); older orphans are deleted.
Leaked refcounts — rows in cas_objects with refcount zero that weren't +cleaned up (e.g., if the process died between decrementing and deleting). +These rows are removed.
Idempotent — running sweep twice produces the same result.

The sweep is wired into the existing repair loop in tesseras-replication, so +it runs automatically every 24 hours alongside fragment health checks.

Migration from old layout (tesseras-storage/src/migration.rs) — A +copy-first migration strategy that moves data from the old directory-based +layout (blobs/<tessera>/<memory>/<file> and +fragments/<tessera>/<index>.shard) into the CAS. The migration:

Checks the storage version in storage_meta (version 1 = old layout, version +2 = CAS)
Walks the old blobs/ and fragments/ directories
Computes BLAKE3 hashes and inserts into CAS via put() — duplicates are +automatically deduplicated
Creates corresponding blob_refs / fragment_refs entries
Removes old directories only after all data is safely in CAS
Updates the storage version to 2

The migration runs on daemon startup, is idempotent (safe to re-run), and +reports statistics: files migrated, duplicates found, bytes saved.

Prometheus metrics (tesseras-storage/src/metrics.rs) — Ten new metrics for +observability:

+ + + + + + + + + + +

Metric	Description
`cas_objects_total`	Total unique objects in the CAS
`cas_bytes_total`	Total bytes stored
`cas_dedup_hits_total`	Number of writes that found an existing object
`cas_bytes_saved_total`	Bytes saved by deduplication
`cas_gc_refcount_deletions_total`	Objects deleted when refcount reached zero
`cas_gc_sweep_orphans_cleaned_total`	Orphan files removed by sweep
`cas_gc_sweep_leaked_refs_cleaned_total`	Leaked refcount rows cleaned
`cas_gc_sweep_skipped_young_total`	Young orphans skipped (grace period)
`cas_gc_sweep_duration_seconds`	Time spent in sweep GC

Property-based tests — Two proptest tests verify CAS invariants under random +inputs:

refcount_matches_actual_refs — after N random put/release operations, the +refcount always matches the actual number of outstanding references
cas_path_is_deterministic — the same hash always produces the same +filesystem path

Integration test updates — All integration tests across tesseras-core, +tesseras-replication, tesseras-embedded, and tesseras-cli updated for the +new CAS-backed constructors. Tamper-detection tests updated to work with the CAS +directory layout.

347 tests pass across the workspace. Clippy clean with -D warnings.

Architecture decisions

BLAKE3 as CAS key: the content hash we already compute for integrity +verification doubles as the deduplication key. No additional hashing step — +the hash computed during create or replicate is reused as the CAS address.
SQLite refcount over filesystem reflinks: we considered using +filesystem-level copy-on-write (reflinks on btrfs/XFS), but that would tie +Tesseras to specific filesystems. SQLite refcounting works on any filesystem, +including FAT32 on cheap USB drives and ext4 on Raspberry Pis.
Two-level hex prefix directories: storing all CAS objects in a flat +directory would slow down filesystems with millions of entries. The +<2-char prefix>/ split limits any single directory to ~65k entries before a +second prefix level is needed. This matches the approach used by Git's object +store.
Grace period for orphan files: the sweep GC skips files younger than 1 +hour to avoid deleting objects that are being written by a concurrent +operation. This is a pragmatic choice — it trades a small window of potential +orphans for crash safety without requiring fsync or two-phase commit.
Copy-first migration: the migration copies data to CAS before removing old +directories. If the process is interrupted, the old data is still intact and +migration can be re-run. This is slower than moving files but guarantees no +data loss.
Sweep in repair loop: rather than adding a separate GC timer, the CAS +sweep piggybacks on the existing 24-hour repair loop. This keeps the daemon +simple — one background maintenance cycle handles both fragment health and +storage cleanup.

What comes next

Phase 4 continued — security audits, OS packaging (Alpine, Arch, Debian, +OpenBSD, FreeBSD)
Phase 5: Exploration and Culture — public tessera browser by +era/location/theme/language, institutional curation, genealogy integration +(FamilySearch, Ancestry), physical media export (M-DISC, microfilm, acid-free +paper with QR), AI-assisted context

Storage deduplication completes the storage efficiency story for Tesseras. A +node that stores fragments for thousands of users — common for institutional +nodes and always-on full nodes — now pays the disk cost of unique data only. +Combined with Reed-Solomon erasure coding (which already minimizes redundancy at +the network level), the system achieves efficient storage at both the local and +distributed layers.

+ +