diff options
| author | murilo ijanc | 2026-03-24 21:41:06 -0300 |
|---|---|---|
| committer | murilo ijanc | 2026-03-24 21:41:06 -0300 |
| commit | f186b71ca51e83837db60de13322394bb5e6d348 (patch) | |
| tree | cd7940eaa16b83d2cde7b18123411bfb161f7ebb /news/phase4-institutional-onboarding | |
| download | website-f186b71ca51e83837db60de13322394bb5e6d348.tar.gz | |
Initial commit
Import existing tesseras.net website content.
Diffstat (limited to 'news/phase4-institutional-onboarding')
| -rw-r--r-- | news/phase4-institutional-onboarding/index.html | 239 | ||||
| -rw-r--r-- | news/phase4-institutional-onboarding/index.html.gz | bin | 0 -> 5539 bytes |
2 files changed, 239 insertions, 0 deletions
diff --git a/news/phase4-institutional-onboarding/index.html b/news/phase4-institutional-onboarding/index.html new file mode 100644 index 0000000..cbd9ac3 --- /dev/null +++ b/news/phase4-institutional-onboarding/index.html @@ -0,0 +1,239 @@ +<!DOCTYPE html> +<html lang="en"> +<head> + <meta charset="utf-8"> + <meta name="viewport" content="width=device-width, initial-scale=1"> + <title>Phase 4: Institutional Node Onboarding — Tesseras</title> + <meta name="description" content="Libraries, archives, and museums can now join the Tesseras network as verified institutional nodes with DNS-based identity, full-text search indexes, and configurable storage pledges."> + <!-- Open Graph --> + <meta property="og:type" content="article"> + <meta property="og:title" content="Phase 4: Institutional Node Onboarding"> + <meta property="og:description" content="Libraries, archives, and museums can now join the Tesseras network as verified institutional nodes with DNS-based identity, full-text search indexes, and configurable storage pledges."> + <meta property="og:image" content="https://tesseras.net/images/social.jpg"> + <meta property="og:image:width" content="1200"> + <meta property="og:image:height" content="630"> + <meta property="og:site_name" content="Tesseras"> + <!-- Twitter Card --> + <meta name="twitter:card" content="summary_large_image"> + <meta name="twitter:title" content="Phase 4: Institutional Node Onboarding"> + <meta name="twitter:description" content="Libraries, archives, and museums can now join the Tesseras network as verified institutional nodes with DNS-based identity, full-text search indexes, and configurable storage pledges."> + <meta name="twitter:image" content="https://tesseras.net/images/social.jpg"> + <link rel="stylesheet" href="https://tesseras.net/style.css?h=21f0f32121928ee5c690"> + + + <link rel="alternate" type="application/atom+xml" title="Tesseras" href="https://tesseras.net/atom.xml"> + + + <link rel="icon" type="image/png" sizes="32x32" href="https://tesseras.net/images/favicon.png?h=be4e123a23393b1a027d"> + +</head> +<body> + <header> + <h1> + <a href="https://tesseras.net/"> + <img src="https://tesseras.net/images/logo-64.png?h=c1b8d0c4c5f93b49d40b" alt="Tesseras" width="40" height="40" class="logo"> + Tesseras + </a> + </h1> + <nav> + + <a href="https://tesseras.net/about/">About</a> + <a href="https://tesseras.net/news/">News</a> + <a href="https://tesseras.net/releases/">Releases</a> + <a href="https://tesseras.net/faq/">FAQ</a> + <a href="https://tesseras.net/subscriptions/">Subscriptions</a> + <a href="https://tesseras.net/contact/">Contact</a> + + </nav> + <nav class="lang-switch"> + + <strong>English</strong> | <a href="/pt-br/news/phase4-institutional-onboarding/">Português</a> + + </nav> + </header> + + <main> + +<article> + <h2>Phase 4: Institutional Node Onboarding</h2> + <p class="news-date">2026-02-15</p> + <p>A P2P network of individuals is fragile. Hard drives die, phones get lost, +people lose interest. The long-term survival of humanity's memories depends on +institutions — libraries, archives, museums, universities — that measure their +lifetimes in centuries. Phase 4 continues with institutional node onboarding: +verified organizations can now pledge storage, run searchable indexes, and +participate in the network with a distinct identity.</p> +<p>The design follows a principle of trust but verify: institutions identify +themselves via DNS TXT records (the same mechanism used by SPF, DKIM, and DMARC +for email), pledge a storage budget, and receive reciprocity exemptions so they +can store fragments for others without expecting anything in return. In +exchange, the network treats their fragments as higher-quality replicas and +limits over-reliance on any single institution through diversity constraints.</p> +<h2 id="what-was-built">What was built</h2> +<p><strong>Capability bits</strong> (<code>tesseras-core/src/network.rs</code>) — Two new flags added to +the <code>Capabilities</code> bitfield: <code>INSTITUTIONAL</code> (bit 7) and <code>SEARCH_INDEX</code> (bit 8). +A new <code>institutional_default()</code> constructor returns the full Phase 2 capability +set plus these two bits and <code>RELAY</code>. Normal nodes advertise <code>phase2_default()</code> +which lacks institutional flags. Serialization roundtrip tests verify the new +bits survive MessagePack encoding.</p> +<p><strong>Search types</strong> (<code>tesseras-core/src/search.rs</code>) — Three new domain types for +the search subsystem:</p> +<ul> +<li><code>SearchFilters</code> — query parameters: <code>memory_type</code>, <code>visibility</code>, <code>language</code>, +<code>date_range</code>, <code>geo</code> (bounding box), <code>page</code>, <code>page_size</code></li> +<li><code>SearchHit</code> — a single result: content hash plus a <code>MetadataExcerpt</code> (title, +description, memory type, creation date, visibility, language, tags)</li> +<li><code>GeoFilter</code> — bounding box with <code>min_lat</code>, <code>max_lat</code>, <code>min_lon</code>, <code>max_lon</code> for +spatial queries</li> +</ul> +<p>All types derive <code>Serialize</code>/<code>Deserialize</code> for wire transport and +<code>Clone</code>/<code>Debug</code> for diagnostics.</p> +<p><strong>Institutional daemon config</strong> (<code>tesd/src/config.rs</code>) — A new <code>[institutional]</code> +TOML section with <code>domain</code> (the DNS domain to verify), <code>pledge_bytes</code> (storage +commitment in bytes), and <code>search_enabled</code> (toggle for the FTS5 index). The +<code>to_dht_config()</code> method now sets <code>Capabilities::institutional_default()</code> when +institutional config is present, so institutional nodes advertise the right +capability bits in Pong responses.</p> +<p><strong>DNS TXT verification</strong> (<code>tesd/src/institutional.rs</code>) — Async DNS resolution +using <code>hickory-resolver</code> to verify institutional identity. The daemon looks up +<code>_tesseras.<domain></code> TXT records and parses key-value fields: <code>v</code> (version), +<code>node</code> (hex-encoded node ID), and <code>pledge</code> (storage pledge in bytes). +Verification checks:</p> +<ol> +<li>A TXT record exists at <code>_tesseras.<domain></code></li> +<li>The <code>node</code> field matches the daemon's own node ID</li> +<li>The <code>pledge</code> field is present and valid</li> +</ol> +<p>On startup, the daemon attempts DNS verification. If it succeeds, the node runs +with institutional capabilities. If it fails, the node logs a warning and +downgrades to a normal full node — no crash, no manual intervention.</p> +<p><strong>CLI setup command</strong> (<code>tesseras-cli/src/institutional.rs</code>) — A new +<code>institutional setup</code> subcommand that guides operators through onboarding:</p> +<ol> +<li>Reads the node's identity from the data directory</li> +<li>Prompts for domain name and pledge size</li> +<li>Generates the exact DNS TXT record to add: +<code>v=tesseras1 node=<hex> pledge=<bytes></code></li> +<li>Writes the institutional section to the daemon's config file</li> +<li>Prints next steps: add the TXT record, restart the daemon</li> +</ol> +<p><strong>SQLite search index</strong> (<code>tesseras-storage</code>) — A migration +(<code>003_institutional.sql</code>) that creates three structures:</p> +<ul> +<li><code>search_content</code> — an FTS5 virtual table for full-text search over tessera +metadata (title, description, creator, tags, language)</li> +<li><code>geo_index</code> — an R-tree virtual table for spatial bounding-box queries over +latitude/longitude</li> +<li><code>geo_map</code> — a mapping table linking R-tree row IDs to content hashes</li> +</ul> +<p>The <code>SqliteSearchIndex</code> adapter implements the <code>SearchIndex</code> port trait with +<code>index_tessera()</code> (insert/update) and <code>search()</code> (query with filters). FTS5 +queries support natural language search; geo queries use R-tree <code>INTERSECT</code> for +bounding box lookups. Results are ranked by FTS5 relevance score.</p> +<p>The migration also adds an <code>is_institutional</code> column to the <code>reciprocity</code> table, +handled idempotently via <code>pragma_table_info</code> checks (SQLite's +<code>ALTER TABLE ADD COLUMN</code> lacks <code>IF NOT EXISTS</code>).</p> +<p><strong>Reciprocity bypass</strong> (<code>tesseras-replication/src/service.rs</code>) — Institutional +nodes are exempt from reciprocity checks. When <code>receive_fragment()</code> is called, +if the sender's node ID is marked as institutional in the reciprocity ledger, +the balance check is skipped entirely. This means institutions can store +fragments for the entire network without needing to "earn" credits first — their +DNS-verified identity and storage pledge serve as their credential.</p> +<p><strong>Node-type diversity constraint</strong> (<code>tesseras-replication/src/distributor.rs</code>) — +A new <code>apply_institutional_diversity()</code> function limits how many replicas of a +single tessera can land on institutional nodes. The cap is +<code>ceil(replication_factor / 3.5)</code> — with the default <code>r=7</code>, at most 2 of 7 +replicas go to institutions. This prevents the network from becoming dependent +on a small number of large institutions: if a university's servers go down, at +least 5 replicas remain on independent nodes.</p> +<p><strong>DHT message extensions</strong> (<code>tesseras-dht/src/message.rs</code>) — Two new message +variants:</p> +<table><thead><tr><th>Message</th><th>Purpose</th></tr></thead><tbody> +<tr><td><code>Search</code></td><td>Client sends query string, filters, and page number</td></tr> +<tr><td><code>SearchResult</code></td><td>Institutional node responds with hits and total count</td></tr> +</tbody></table> +<p>The <code>encode()</code> function was switched from positional to named MessagePack +serialization (<code>rmp_serde::to_vec_named</code>) to handle <code>SearchFilters</code>' optional +fields correctly — positional encoding breaks when <code>skip_serializing_if</code> omits +fields.</p> +<p><strong>Prometheus metrics</strong> (<code>tesd/src/metrics.rs</code>) — Eight institutional-specific +metrics:</p> +<ul> +<li><code>tesseras_institutional_pledge_bytes</code> — configured storage pledge</li> +<li><code>tesseras_institutional_stored_bytes</code> — actual bytes stored</li> +<li><code>tesseras_institutional_pledge_utilization_ratio</code> — stored/pledged ratio</li> +<li><code>tesseras_institutional_peers_served</code> — unique peers served fragments</li> +<li><code>tesseras_institutional_search_index_total</code> — tesseras in the search index</li> +<li><code>tesseras_institutional_search_queries_total</code> — search queries received</li> +<li><code>tesseras_institutional_dns_verification_status</code> — 1 if DNS verified, 0 +otherwise</li> +<li><code>tesseras_institutional_dns_verification_last</code> — Unix timestamp of last +verification</li> +</ul> +<p><strong>Integration tests</strong> — Two tests in +<code>tesseras-replication/tests/integration.rs</code>:</p> +<ul> +<li><code>institutional_peer_bypasses_reciprocity</code> — verifies that an institutional +peer with a massive deficit (-999,999 balance) is still allowed to store +fragments, while a non-institutional peer with the same deficit is rejected</li> +<li><code>institutional_node_accepts_fragment_despite_deficit</code> — full async test using +<code>ReplicationService</code> with mocked DHT, fragment store, reciprocity ledger, and +blob store: sends a fragment from an institutional sender and verifies it's +accepted</li> +</ul> +<p>322 tests pass across the workspace. Clippy clean with <code>-D warnings</code>.</p> +<h2 id="architecture-decisions">Architecture decisions</h2> +<ul> +<li><strong>DNS TXT over PKI or blockchain</strong>: DNS is universally deployed, universally +understood, and already used for domain verification (SPF, DKIM, Let's +Encrypt). Institutions already manage DNS. No certificate authority, no token, +no on-chain transaction — just a TXT record. If an institution loses control +of their domain, the verification naturally fails on the next check.</li> +<li><strong>Graceful degradation on DNS failure</strong>: if DNS verification fails at startup, +the daemon downgrades to a normal full node instead of refusing to start. This +prevents operational incidents — a DNS misconfiguration shouldn't take a node +offline.</li> +<li><strong>Diversity cap at <code>ceil(r / 3.5)</code></strong>: with <code>r=7</code>, at most 2 replicas go to +institutions. This is conservative — it ensures the network never depends on +institutions for majority quorum, while still benefiting from their storage +capacity and uptime.</li> +<li><strong>Named MessagePack encoding</strong>: switching from positional to named encoding +adds ~15% overhead per message but eliminates a class of serialization bugs +when optional fields are present. The DHT is not bandwidth-constrained at the +message level, so the tradeoff is worth it.</li> +<li><strong>Reciprocity exemption over credit grants</strong>: rather than giving institutions +a large initial credit balance (which is arbitrary and needs tuning), we +exempt them entirely. Their DNS-verified identity and public storage pledge +replace the bilateral reciprocity mechanism.</li> +<li><strong>FTS5 + R-tree in SQLite</strong>: full-text search and spatial indexing are built +into SQLite as loadable extensions. No external search engine (Elasticsearch, +Meilisearch) needed. This keeps the deployment a single binary with a single +database file — critical for institutional operators who may not have a DevOps +team.</li> +</ul> +<h2 id="what-comes-next">What comes next</h2> +<ul> +<li><strong>Phase 4 continued</strong> — storage deduplication (content-addressable store with +BLAKE3 keying), security audits, OS packaging (Alpine, Arch, Debian, OpenBSD, +FreeBSD)</li> +<li><strong>Phase 5: Exploration and Culture</strong> — public tessera browser by +era/location/theme/language, institutional curation, genealogy integration +(FamilySearch, Ancestry), physical media export (M-DISC, microfilm, acid-free +paper with QR), AI-assisted context</li> +</ul> +<p>Institutional onboarding closes a critical gap in Tesseras' preservation model. +Individual nodes provide grassroots resilience — thousands of devices across the +globe, each storing a few fragments. Institutional nodes provide anchoring — +organizations with professional infrastructure, redundant storage, and +multi-decade operational horizons. Together, they form a network where memories +can outlast both individual devices and individual institutions.</p> + +</article> + + </main> + + <footer> + <p>© 2026 Tesseras Project. <a href="/atom.xml">News Feed</a> · <a href="https://git.sr.ht/~ijanc/tesseras">Source</a></p> + </footer> +</body> +</html> diff --git a/news/phase4-institutional-onboarding/index.html.gz b/news/phase4-institutional-onboarding/index.html.gz Binary files differnew file mode 100644 index 0000000..92beb89 --- /dev/null +++ b/news/phase4-institutional-onboarding/index.html.gz |