The data pipeline

Futuros does not query live APIs when someone opens a page. The entire corpus is ingested, baked and frozen as static JSON before deployment; the SPA only reads files. That makes the platform fast, cacheable, offline-capable — and, above all, auditable: every figure you see was verified at build time against its source record. This pipeline directly serves Pillar 1 — Data Trust: a corpus with provenance and a clean license is the one asset a regional model cannot copy.

The source registry

Every external source is a SourceRecord declared in scripts/sources/registry/. It defines where the data comes from and under which rules it may be published:

• AdapterEngine — the connector type: series, bulk, sdmx, ckan, ocds, event, entity, microdata, provider or meta. Every external API has exactly one adapter in adapters.ts (WHO GHO, UNESCO UIS, OpenAlex, Climate Watch, UN SDG, UNHCR, OWID, IMF DataMapper, generic SDMX/DBnomics…). Later waves live under scripts/sources/gap/ (one file per source). Wave F (wave-f/ + src/lib/gap-sources-f/) is the full-geo pass: JME wasting, UIS tertiary GER, DHS nutrition, CIVICUS civic space, Ember/IRENA fill, plus honest-gap probes (RICYT, ILIA/GIRAI/GDB, Chile INE.Stat health/innovation, ENDI) that re-check and return []. Refresh: scripts/refresh-wave-f.ts.
• LicensePosture — kind / redistribute / aggregates_only: what may be republished and what may only be cited.
• Probe verification — sources.ts is SOURCE_REGISTRY_V2, with probe-tested endpoints; regulation-sources.ts is the sibling registry for AI-law tracking. The nso/*.json files are 27 source lists, one per national statistics office.
• Verification anatomy — each record also declares its deep_link (the per-record URL stamped into citations.json), its cadence (cadence_days), a verify spec for sample-based re-verification (refetch-compare: re-request a sample and compare against what was stored) and an expected_volume sized for a ±50% volume sentinel. Note: the sentinel and the re-verifier are today a declared schema contract — no ingester runs them yet; the guard that does run on every refresh is the freshness guard.

The full schema of a SourceRecord (scripts/sources/registry/schema.ts) — each field with the rule validateRegistry() applies to it:

Field	Type / values	Rule in validateRegistry()
key	stable UPPER_SNAKE	regex ^[A-Z0-9_]+$; duplicate or invalid = error
tier · priority · status	primary/secondary/curated · P0/P1/P2 · wired/ready/blocked/research	the daemon's ingest:due tick drains ready rows
engine	one of the 10 AdapterEngine values	unknown engine = error
license	kind (open/cc-by/non-commercial/share-alike/cite-only) + redistribute + aggregates_only?	the license gate (below); aggregates_only + microdata engine = reminder warn: the adapter aggregates, never dumps rows
auth	none/env_key/registration/approval + env	env = environment variable the adapter reads
endpoint	URL pattern with {iso3}/{code} placeholders	missing = error
deep_link	per-record URL pattern	this is what gets stamped into citations.json
cadence_days	number	the freshness ledger compares against this
geos · geo_level	ISO3[] or "ALL" · country/adm1/adm2/point/supranational	geo outside the 26-strong set (25 countries + LATAM) = warn
pillars	dimension slugs	empty with engine ≠ provider = warn
expected_volume	number ≥ 0	negative or non-numeric = error; sizes the ±50% sentinel
verify	sample_n + method (refetch-compare/liveness-only/checksum)	missing block or sample_n < 0 = error

The adapters are failure-isolated: a dead endpoint returns [] instead of breaking the ingestion. Cadence (how many months between refreshes for each indicator) lives next to the source and materializes in the freshness ledger.

The license gate (at compile time)

The most important rule in the registry is not a runtime rule: it is a compile-time rule. validateRegistry() rejects any source whose license says redistribute: false while being wired to an engine that is not provider. The exact condition is threefold: redistribute === false and engine ≠ provider and status === "wired" — a cite-only source may stay declared as research or blocked without breaking the build; what it cannot do is be wired to an engine that produces values. Republishing values from a source that only allows citing is a compile error (assertRegistryValid() throws with the list of offending keys), not a warning someone could ignore. A source's license posture decides whether its data appears as a republished indicator, only as an aggregate, or only as a citation — and it does so before the build produces a single file.

The same gate applies to data contributed by third parties via /contribuir: nothing is redistributed beyond what its license allows. It is the mechanism that keeps the Data Trust honest.

The ingestion stages

scripts/ingest-all.ts orchestrates three stages, each failure-isolated (a failing stage does not drag down the others):

1. runMultisource() → parameter-cache/ cells + citations. This is the bulk: each indicator's value per country, with its citation_id.
2. runDocuments() → documents/ and news/: documents and news by dimension and country.
3. runSignals() → cited signals, LLM-extracted, in signals/.

When finished, it writes refresh-meta.json. There are additional domain ingesters for open budget, economic freedom, digital freedom, authoritarian drift, OGP, human freedom, press freedom, regulation, the observatory and the ADM1 subnational level.

There is also a fourth, cadenced path: the social pulse. ingest-social.ts (GDELT events + GKG + social RSS) and fetch-trends.ts feed the social/, social-trends/, attention/ and civic/ families, with their own bakes (bake-trends, bake-civic, bake-attention). It appears in refresh-meta.json as the source social:gdelt, with a post count.

And a fifth path: high-frequency (HF). ingest-hf.ts runs NASA POWER daily (one capital point per ISO3) and, in the same run, adapters for FIRMS, ClimateSERV CHIRPS, monthly Ember, Wikimedia, GHS, FAO ASIS, GFW and Meta Movement. The publication rule is strict: only nasa-power.json is committed and served. An empty store (observations: []) or a feed with no publishable rows is not written under public/data/hf-observations/ — a hollow 200 is a defect, not a "coming soon". check-hf-publish.ts blocks the build if any extra JSON appears or POWER comes back empty. POWER does not enter parameter-cache or observations.csv (that CSV already exceeds GitHub's 100 MB ceiling). The hf lane does not move generated_at.

The bake

scripts/bake-all.ts transforms the raw cache into the derived datasets the pages consume: narratives → frontier indicators → subnational → pulse briefs and contradictions → forecasts → shadows → people graph → composite scores → markets index → _index.json. There are 41 bake-*.ts scripts plus 10 build-* generators — close to 50 bake-and-generate scripts: scores, markets, V-Dem, exploration, insights, data health, signals, coalitions, ontology, trends, civic, attention, feeds, contributions, compute sovereignty, regional convergence, instability nowcast, triangulation, narratives and the people graph, plus generators like build-democracy-index and build-search-index.

Two domain generators index families you already saw in the dataset list: build-cases-index.ts regenerates the _index.json of cases/ (the per-country precedents and lessons, authored file by file) and gen-elections-index.mjs regenerates that of elections/ (the regional election calendar).

No fact is "baked" into a model: facts live in the data with their deterministic citation; the bake only precomputes what is derivable from them.

The baked public API

The pipeline has a fourth product besides the SPA's data: a static read-only API at public/api/v1, re-baked on every deploy by bake-api.ts — citations.csv/.json, freshness.json, contributions.json, futuros-data.xlsx and per-country and per-indicator records with M49/SDMX interoperability. It has its own gate: check-api.ts runs after the bake and verifies that the API mirrors the current data (for example, that its index's citation count matches citations.json). By construction, the API is never out of sync with the corpus. Full reference in Public API.

The dataset families (public/data/**)

The result is 68 families of static data. Among the main ones:

• parameters.json (the 10 axes), geographies.json (25 countries + LATAM, with ISO3; M49 codes do not live here — they are joined from an ISO3→M49 mapping owned by bake-api.ts when baking the public API).
• parameter-cache/ (indicator×country cells) and parameter-cache-adm1/ (thousands of subnational cells).
• hf-observations/ — only nasa-power.json is published (non-empty NASA POWER). Other HF feeds are ingested but not committed or served; an empty shell is a defect.
• citations.json — the central citation registry (several MB).
• documents/, news/, signals/, social/, social-trends/, attention/, shadows/.
• personas/ (profiles + _index + _graph), state-history/, country-history/.
• scores.json (composite), data-health.json, frontier-indicators/, compute-sovereignty/, agi-economy/, dependency/, climate-action/.
• pulse/, regional-convergence/, subnational-convergence/, instability-nowcast/, triangulation/, uncertainty/, insights/.
• pilots/, pilot-archetypes/, financing-instruments/, coalitions/, positions/, cases/.
• regulation/, constitutional/, elections/, democracy/, transparency/, ai-governance/.
• markets/, metrics/, ontology/, civic/, gender-gap/, trends/.
• search-index.json (Voyage index for the assistant), catalog.json, sdg-crosswalk.json, falsifications.json.

Freshness

public/data/refresh-meta.json is the freshness ledger, with two granularities: the indicators map tracks the World Bank / WGI subset per indicator (49 codes); everything else (884 series in the baked source ledger) is tracked at feed level in the sources map (197 entries such as social:gdelt or doc:exa, with kind, last_refreshed, next_refresh_due and counts like posts). The shape of a per-indicator entry:

Field	Example (SI.POV.GINI)	What it answers
source / source_url	World Bank Open Data	where did it come from?
cadence_months / next_refresh_due	12 / 2027-03-02	when is a refresh due?
last_refreshed	2026-07-28	when did we last request it?
last_observation_year	2025	how new is the data itself?
countries_with_data / total_countries	22 / 25	how much coverage?
source_frontier_year / source_frontier_checked_at	2025 / 2026-07-23	is there anything newer upstream?
freshness_status / freshness_note	current	verdict: current, source_not_updated or no_data

The distinction that carries the weight is last_refreshed vs last_observation_year: yesterday's pull with 2023 data is not a pipeline failure but provider lag — and source_frontier_year proves it, because it records the newest year the upstream publishes. That is what makes it possible to show a figure's vintage instead of pretending everything is up to date. The /confianza surface reads this ledger; /incertidumbre is something else — it exposes V-Dem confidence bands from uncertainty/vdem-ci.json, not this file.

The ledger also has an active guard, not just a screen. The multisource runner prunes and rewrites cells while ingesting, so a run with a downed or rate-limited adapter could silently erase cells that used to exist — the "my fetch failed, written to disk as there is nothing there" defect. freshness-guard.py prevents it with three subcommands over parameter-cache/:

snapshot <out.json>          # before the run
  for each parameter-cache/<pillar>__<ISO3>.json, for each indicator:
    snap[id][cell] = { vintage: vintage_year, value, obj: <full indicator> }

compare <before.json>        # after the run
  per indicator:  gone       (cells that disappeared)
                  regressed  (vintage went backwards: after < before)
                  advanced   (vintage moved forward)  ·  added (new cells)
  reports the total of lost + regressed cells

repair <before.json> --apply
  restores ONLY gone ∪ regressed, from the obj stored in the snapshot
  advanced and added are left intact — a genuine advance is never reverted
  without --apply it is a dry run: lists what it would restore, touches no disk

Subtle invariant of the --only=<substr> filter: the guard filters the baseline with the same terms as the live snapshot. Without that, every indicator outside the filter would look gone and repair would restore the entire platform to the baseline — including later deliberate fixes. Refresh runs are non-destructive by construction.

In the prebuild, bake-source-ledger.ts bakes the source ledger that feeds /confianza and stamps the top-level fields source_ledger_baked_at and source_ledger_series into refresh-meta.json; on data refreshes, recheck-wb-freshness.ts re-verifies World Bank vintages against the upstream and stamps wb_freshness_checked_at.

The prebuild gates

The prebuild is a chain of gates linked with && in package.json; if any of them fails, the build fails and nothing gets deployed:

* = advisory (--warn): reports and exits 0.

First the groundwork: two typechecks (tsc -p tsconfig.scripts.json for the ingestion and bake scripts, tsc -p tsconfig.server.json for the assistant server and the functions), the full test suite (bun test --timeout 15000 — 2,076 tests across 175 files, including the assistant's server, the MCP handler and the bakers) and build-positions-index --check. Then twenty-one named check-* checks, plus two gates in --check mode over scripts that normally write (build-positions-index, bake-trends-index). Three are advisory (--warn: they report and exit 0 — check-search-index, check-vintage, check-adm1-coverage); the rest fail the build.

Corpus provenance and coherence:

1. check-traced — every figure has its trace back to a source.
2. check-signals — the signals are well-formed and cited.
3. check-source-links — citation links point to the specific country + indicator record. The only exception is the LANDING_OK list inside the gate itself: a documented per-source opt-out (curated and internal layers that resolve to /metodologia, documentary layers EXA/GDELT/RSS/gazettes/signals and a few sources with no stable per-record URL whose honest destination is their own dataset page), each with its justification in the code. Everything else carries a deep link or the build fails.
4. check-pilots — pilot coherence.
5. check-persona-scope — no profile may be unreachable. The /personas country <select> is built from the canonical 25, so a profile filed under an iso3 outside that set was a complete, sourced record no visitor could filter to (audit D-5 found 50 of 1,384 that way: Barbados, the Bahamas and one Miami-based columnist under USA). They were not deleted, they were tiered; this gate makes the tiering binding.
6. check-citations — integrity of the citation registry.
7. check-parameter-cache-citations — every citation_id in parameter-cache/ resolves in citations.json, with no synthetic-citation fallback. The public API was already clean; this gate keeps the cache it is baked from clean too.
8. check-scores — coherence of the composite scores (see The Futuros Index).
9. check-provenance — rejects fabricated hashes or digests (truncated sha256 stubs) anywhere in public/data; born from the July 2026 data-accuracy audit.
10. colombia/check-real-data — separates real from synthetic in the Colombia corpus, so no demo figure ships without its banner.

Style, taxonomy and coverage:

11. check-emdash-drift — style drift (em dashes) does not reappear in already-swept content.
12. check-landing-stats — every number printed on public/landing.html is still the number the data says. The landing is a hand-edited artifact, the highest-traffic page and the only one with no component, dictionary or test behind it; its statistics were free to rot, and they did (the 2026-08-13 audit found it advertising 6,726 citations against a corpus of 6,855, and "300+ pilots" against an index of 337). The gate recomputes each figure from public/data/** the way the matching surface does, and prints the literal to paste. It checks six today: distinct primary sources, countries, axes, citations, personas and instruments.
13. check-catalog — the dataset catalog is complete and consistent.
14. check-search-index (advisory) — the semantic index covers the corpus.
15. check-law-term-index — the legal term index covers 100% of the regulation corpus (instrument_count from regulation/_index.json). Local index only: Voyage stays opt-in and fail-closed.
16. bake-trends-index --check — the baked trends index matches its inputs.
17. check-vintage (advisory) — how many published figures break the two-year vintage rule. The threshold and the year-resolution order come from src/lib/vintage.ts, the same module the headline strips render from, so the gate measures what the visitor sees. It exists because bake-data-health scores a cell by the median age of its indicators: the platform reported 2 stale cells out of 260 while 27% of individual figures broke the rule.
18. check-adm1-coverage (advisory) — the honesty gate for the subnational layer. "432 of 456 ADM1 units carry real data" was true only of the GDL universal overlay; at eje level no unit carried more than 5 of the 10 ejes on a real measurement, five ejes were 100% modelled, and 26 units had their cells written under slugs the app can never request. Nothing detected any of it, so it drifted silently for months.
19. check-i18n — ES/EN/PT coverage does not regress.
20. check-taxonomy — UI copy does not revert to the retired nouns for the ten sectoral measures (diez parámetros, ten dimensions, dez dimensões) nor point nav at /governance. The ten are ejes / axes / eixos.
21. check-hf-publish — only nasa-power.json may live under public/data/hf-observations/, and it must be a non-empty NASA POWER store. Empty shells (observations: []) and unpublished feeds (CHIRPS, FIRMS, Ember…) are a defect: they serve a 200 that looks live.
22. check-api — runs last, after the four prebuild bakes (bake-data-health, bake-source-ledger, bake-api, bake-feeds), and verifies with seven assertion groups that the static public API mirrors the current data.

The provenance gates in the strict sense are the subset check-traced, check-signals, check-source-links, check-citations, check-parameter-cache-citations and check-provenance — but they all share the same consequence: they are the reason "every figure with its source" is not a slogan but a compile-time invariant. How a citation resolves on screen is explained in Provenance and citations.

A pattern repeats across the newer gates (check-landing-stats, check-vintage, check-adm1-coverage, check-persona-scope): each was born from an audit that found a claim true when it was written and false when it was read, with nothing connecting it to the data. The gate is that connection. It is the same reason this documentation exists, and the reason it is audited against the code.