Provenance optimization: technical roadmap

Futuros' provenance today is derivable: the deep link is reconstructed from the citation_id, the prebuild gate battery (traceability, deep links, citations, scores, provenance, catalog, API — plus the full test suite) rejects any figure without a trail, and the trust receipt is a reproducible hash over a canonical body (Provenance and citations, The data pipeline). This page documents the next frontier: moving from "every figure has its source" to "every figure carries a machine-verifiable proof of origin that a third party can check without trusting us".

Everything that follows is roadmap, not live capability, except where marked otherwise. The order is deliberate: each phase reuses seams that already exist in the code (citation_id, stableStringify, receipt_hash, chain_anchor, refresh-meta.json, the check-* gates), so no phase demands a rewrite.

The target model: four properties

A figure with optimized provenance satisfies four independent properties, each with its mechanism:

Property	Question it answers	Mechanism
Integrity	Are these bytes the ones that were ingested?	Content addressing (SHA-256) of upstream responses
Authenticity	Who produced this artifact?	Ed25519 signatures over canonical bodies + build attestations
Reproducibility	Can I recompute it and get the same thing?	Hermetic, deterministic baking, inputs pinned by digest
Transparency	Could history have been rewritten afterwards?	Merkle-style append-only log with periodic external anchoring

The current discipline (deterministic citations + build gates) gives internal integrity. The following phases make it verifiable from the outside.

Phase A — Content-addressed acquisition layer

Today ingestion writes derived cells; the raw bytes of the upstream response are discarded. That makes it impossible to re-audit a transformation after the fact: if the World Bank revises a series, we cannot prove which version we saw.

The design: every adapter fetch persists its raw response as a content-addressed blob:

snapshots/
  sha256/ab/cd/abcd…ef        # raw response bytes (gzip)
  manifests/wb-sp-dyn-le00-in-2026-07-30.json

The fetch manifest records { url, method, relevant_request_headers, etag, last_modified, fetched_at, digest, adapter, source_key }. Every cell in parameter-cache/ gains a derived_from: [digest…] field — the list of exact blobs it was derived from. The set forms a Merkle DAG: cells → manifests → blobs; the root digest of a bake summarizes the entire input corpus in 32 bytes.

Immediate consequences:

• Retroactive re-auditing. Any transformation defect can be reproduced against the original bytes, not against whatever the upstream serves today.
• Upstream revision detection. A different etag/digest at the same URL distinguishes "the source revised its series" from "our pipeline changed" — two events that are indistinguishable today.
• Free deduplication. Content addressing collapses identical responses across runs; storage cost grows with real change, not with refresh cadence.

The existing seam: scripts/freshness-guard.py already implements snapshot→compare→repair for one-off audits; phase A generalizes it from audit tool to pipeline invariant.

Phase B — Queryable PROV-O lineage graph

/linaje narrates the origin of each series; phase B turns it into a formal graph using the W3C PROV-O vocabulary:

• Every phase-A blob is a prov:Entity; every adapter run or bake-* script is a prov:Activity (prov:used → inputs, prov:generated → outputs); the pipeline and its operators are prov:Agents.
• Every baked cell exports its chain as JSON-LD under the public API: GET /api/v1/prov/{indicator}/{iso3} returns the minimal subgraph connecting the on-screen figure to the digests of its origin blobs.
• The graph is derived, not hand-maintained: it is emitted as a byproduct of instrumenting the ~50 bake scripts, with a common wrapper that records (inputs read, outputs written, digest of each, script version).

The success criterion is falsifiable: for any figure visible on the platform, a client with no access to the repo can walk the JSON-LD from the citation_id down to a raw-bytes digest, and verify every edge with hashes.

Phase C — Reproducible baking + build attestations

Reproducibility turns lineage into something checkable, not merely declared:

1. Hermetic bakes. Every bake-*.ts declares its inputs by digest (phase A) and is forbidden from touching the network; the sources of non-determinism (timestamps, object iteration order, floats) are canonicalized — stableStringify already exists for the trust receipts (server/trust/ledger.ts: recursive serialization with sorted keys, over which receipt_hash = sha256(canonical body); the receipt is deterministic given the contribution quadruple + issued_at + nonce + salt, and therefore independently reproducible) and is promoted to the serializer of every baked artifact. Goal: bit-for-bit identical scores.json given the same input set.
2. in-toto/SLSA-style attestations. Every artifact under public/data/** and public/api/v1/** ships with a signed attestation: { subject: artifact_digest, materials: [input_digests], builder: {repo, commit, script, version}, invocation: {argv, allowed_env} }, signed with Ed25519 using a published build key. They are published under public/api/v1/attestations/.
3. Third-party verification. With (1) + (2), an external auditor — a university, a multilateral bank, a government running the sovereign mirror — recomputes the bake over the same blobs and compares digests. Trust stops being "we believe Futuros" and becomes "we recomputed it".

Honesty note: the HMAC issuer authentication of the Data Trust receipt is now built and env-gated (TRUST_RECEIPT_HMAC_KEY): with the key configured, the receipt becomes schema v2 and carries an HMAC-SHA256 signature over its receipt_hash. That authenticates the issuer but is not third-party verifiable, since verification requires the secret key; public verifiability with Ed25519 remains roadmap, and this phase's build signing key is its prerequisite.

Phase D — Transparency log with external anchoring

An append-only ledger operated by the same actor that writes to it proves little: we could rewrite history and re-serve it. Phase D makes that economically infeasible to forge:

• RFC 6962-style Merkle tree (Certificate Transparency) over three series: the citation registry, the trust ledger and the build attestations. Every new entry produces a signed checkpoint {tree_size, root_hash, timestamp}.
• Inclusion and consistency proofs. A contribution receipt gains an inclusion proof of ~log₂(n) hashes; any two checkpoints admit a consistency proof (nothing was deleted or rewritten between them). The existing receipt_hash is exactly the leaf the tree needs — zero schema migration. (And the receipt already seals salt_version — a public one-way label of the salt that produced the contributor_id — so a future rotation of TRUST_ID_SALT does not invalidate the verifiability or revocability of old receipts.)
• Periodic external anchoring. The root is anchored outside our infrastructure at a fixed cadence: RFC 3161 timestamping and/or OpenTimestamps (Bitcoin aggregation, ~zero cost), with the receipt's chain_anchor slot — reserved since day one — receiving the anchor identifier. Forging past history would also require forging the dated external anchor.
• Monitors. The signed checkpoint is published in the public API; any third party can run a monitor verifying consistency across checkpoints — the same distributed-audit model as CT.

The shape of the log, schematically:

Phase E — Content credentials on exported artifacts

Provenance today travels with the embed via the synthetic citation (Provenance); it is lost the moment someone exports a PNG/SVG/CSV and pastes it into a report. Phase E attaches C2PA Content Credentials to every export:

• The embedded C2PA manifest declares: issuer (Futuros), the citation_ids of every plotted series, the digest of the origin dataset, a signed timestamp, and the assertion that the render was produced from cited data with no manual editing.
• The CSV/XLSX files at /datos-abiertos gain an attestation sidecar (.csv + .csv.att.json) with the same content — the tabular format does not admit an embedded manifest.
• Result: a Futuros chart circulating on WhatsApp or in a ministerial deck remains verifiable — any C2PA verifier recovers the citations and the digest, and detects edits.

Phase F — Freshness SLOs, drift and typed absence

Phases A–E prove where a figure came from; phase F governs its temporal quality — and encodes the most expensive lesson from this project's data audits.

Freshness SLOs. refresh-meta.json already records cadence and expiry per source — today with last_refreshed / next_refresh_due at feed level (social:gdelt, doc:perplexity, …) and per-indicator tracking only for the WB/WGI subset (49 codes); the rest of the corpus (884 series in the source ledger) is tracked at feed level, with the vintage rollup in data-health.json. Phase F turns them into service-level objectives with a staleness budget: each source declares {cadence, grace, severity}; the gate goes from binary (expired/not) to an aggregate budget that can fail the build when the corpus as a whole ages past the threshold, even if no individual source is critical.

Statistical drift detection. On top of the phase-A blobs: when a refresh brings a revised series, a distributional drift test (PSI / Kolmogorov–Smirnov over the overlapping series) classifies the change as a minor revision (silent re-bake with a note), a structural revision (requires an entry in the source's changelog and re-verification of the derived cells) or a transport anomaly (blocks the refresh — it is probably our error or the upstream's, not new data). It complements the year-over-year anomaly detection already published in the data-health layer (|z| > 4 + a 10 pp floor, see Methodology).

Typed absence. The most dangerous recurring defect found in the pipeline's internal audits: a failed fetch written to disk as "there is no data". An empty cell must declare why it is empty, with a closed enum:

absence: "no_data"        # the source states no measurement exists
       | "not_collected"  # outside the source's declared scope
       | "fetch_failed"   # our transport failed — NOTHING is known about the value
       | "license_gated"  # it exists, but the license only allows citation
       | "revoked"        # it existed, the contributor revoked consent

The build gates treat fetch_failed as a refresh error, never as data: a previous value is never overwritten with a transport absence. The license_gated/revoked distinction connects to the license gate and makes revocation visible in the corpus instead of silent.

Phase G — ZK proof-of-record

Phases C and D leave the trust with two pieces: a deterministic commitment per record and a public log that includes that commitment. Phase G adds the third: zero-knowledge proofs that demonstrate a predicate about a record's contents without revealing the record.

The dependency chain, link by link:

1. Canonical body + receipt_hash (live today; hash-verifiable, unsigned). stableStringify produces a deterministic serialization of the receipt and receipt_hash = sha256(canonical body). It is the only link that exists today: a 32-byte commitment per contribution.
2. Merkle inclusion proof (phase D, roadmap). Places that commitment inside an append-only log with external anchoring. Without D, a ZK proof over a receipt says nothing about the corpus: it only speaks of a blob nobody else saw.
3. ZK proof-of-record (this phase). A circuit takes the canonical body as private witness, the receipt_hash plus the log root as public inputs, and proves statements of the form "a record included in checkpoint N exists whose privacy is aggregates_only and which has not been revoked", without exposing the contributor, the dataset or any other field.

The existing design makes the circuit tractable: the tree leaf is exactly the receipt_hash, and the canonical body is already a stable serialization with sorted keys, that is, a well-defined witness. Phase G does not ask for a receipt redesign; it asks that D operate first.

What it unlocks: selective disclosure (an auditor verifies a property of a sensitive record without seeing it), provable aggregates (the public counts per privacy mode go from claim to proof) and the direct prerequisite for Pillar 2's ZK roadmap (see below).

Honest status: no circuit, no prototype, no proof-system choice exists. It is the last phase of the cryptographic chain and does not start before D is in production.

Phase H — Sensitivity tiers and per-record encryption

The starting point, verifiable in server/trust/ledger.ts:

• PrivacyMode is declared metadata, not a technical control. The contributor picks open | aggregates_only | clean_room | private_compute; the mode governs scoring and how the contribution is aggregated, but today it activates no encryption and no cryptographic access control.
• Storage is plaintext. The ledger persists records unencrypted on the operator's backend.
• Pseudonymity comes from TRUST_ID_SALT. The contributor_id is a salted hash of the email, with salt_version sealed into the receipt so a salt rotation does not break revocation. That is pseudonymity, not encryption: it protects the identity from readers of the ledger, not the content from the operator.

Phase H closes the gap between what is declared and what is enforced with two moves:

1. Per-record sensitivity tiers. Every record is classified into a closed tier: T0 public, T1 pseudonymized (the current state), T2 sensitive, encrypted at rest, T3 never leaves the contributor (the "compute without possess" that private_compute already declares). The default tier derives from the declared PrivacyMode, which stops being just a label.
2. Per-record encryption with per-contributor keys. Tiers T2 and T3 use envelope encryption: one data key per record, wrapped by a per-contributor key. Revocation gains a cryptographic mechanism: destroying the key (crypto-shredding) renders the record unreadable without rewriting the log, compatible with phase D's append-only tree, where the receipt_hash remains as evidence that the record existed.

Dependencies: encryption at rest requires neither G nor D, but the verifiable evidence of a revocation (proving that the ability to decrypt was destroyed) does rely on phase D's log. And H is the data prerequisite for Pillar 2's ZK inference: without encrypted records there is no "inference over encrypted records".

One thread with Pillar 2: ZK inference over encrypted records

The roadmap of Pillar 2, Sovereign Model includes zero-knowledge inference over encrypted trust records. That item and phases G and H on this page are a single dependency chain, not parallel promises:

receipt_hash over canonical body (live) → Merkle inclusion proof (phase D) → ZK proof-of-record (phase G) + tier-encrypted records (phase H) → ZK inference over encrypted records (Pillar 2).

No link after the first exists today. If any page of this documentation were to present Pillar 2's ZK inference as a capability while G and H do not operate here, that would be a reportable defect in the same spirit as the falsifications log.

Sequence and dependencies

Phase	Depends on	Dominant cost	What it unlocks
A — Acquisition CAS	—	Blob storage (grows with change, not with cadence)	Re-auditing, revision detection, the DAG
B — PROV-O lineage	A	Instrumenting the ~50 bake scripts with a common wrapper	Machine-queryable, per-cell provenance
C — Reproducible bakes + attestations	A	Canonicalizing non-determinism; build-key management	Third-party verification; the receipt signature
D — Transparency log + anchoring	C (key)	Operating the log + monitors	Non-rewritable history; chain_anchor stops being null
E — C2PA credentials	C	Manifest tooling in the export pipeline	Provenance that survives outside the platform
F — SLOs + drift + typed absence	A (for drift)	Migrating the cell schema to the absence enum	Governed freshness; end of the "failed fetch = no data" defect class
G — ZK proof-of-record	D	Circuit design and proof-system choice	Selective disclosure; provable aggregates; Pillar 2's ZK prerequisite
H — Sensitivity tiers + per-record encryption	D (for revocation evidence)	Per-contributor key management; storage migration	Tiered encryption at rest; revocation by key destruction

What each phase makes verifiable — and who can check it without trusting Futuros:

Phase	What becomes verifiable	Verifier	Artifact that proves it
A	That the upstream bytes are exactly the ones ingested	Auditor with access to the blobs	SHA-256 digest + fetch manifest
B	The full chain cell → origin blob	Any HTTP client, no repo access	JSON-LD under /api/v1/prov/…
C	That the bake recomputes bit for bit	Third party running the sovereign mirror	Ed25519-signed attestation
D	That history was not rewritten	Continuous external monitor, CT-style	Inclusion/consistency proofs + dated anchor
E	The provenance of an export outside the platform	Any standard C2PA verifier	Embedded manifest or .att.json sidecar
F	Why a cell is empty and how much the corpus has aged	The build gates + the reader	Absence enum + staleness budget
G	A predicate about a record, without revealing the record	Any verifier holding the log checkpoint	ZK proof + inclusion proof
H	That a revocation destroyed the ability to decrypt	Auditor with the log and the key registry	Key-destruction event anchored in the log

A and F-typed-absence are the right starting points: A is the prerequisite for everything cryptographic, and typed absence eliminates today the defect class that has most often produced false-by-omission data. D and E have the highest external visibility but only make sense on top of C. G and H close the chain: G only exists on top of D, and Pillar 2's ZK inference only on top of G and H.

---

The rule governing the whole roadmap is the same as the rest of the platform: no phase is announced as a capability until its verification is executable by a third party. In the meantime, this page is the public commitment of where the optimization is headed — in the same falsifiable spirit as the falsifications log.