The dataset registry¶

Standardising a multi-GB catalogue is expensive and fully deterministic — so it's done once and reused. The registry (cosmotron_mcp/registry.py, Phase R1) is a SQLite-backed store of standardised datasets that outlives any one disposable workspace/ session.

Layout¶

$COSMOTRON_REGISTRY/                 # default ./database (cwd-relative), NOT inside workspace/
├── registry.db                      # SQLite — one row per dataset
└── datasets/<dataset_id>/           # standardised products + a copy of DATA_DESCRIPTION.md

Set COSMOTRON_REGISTRY to an absolute path for anything beyond local experimentation — the default resolves relative to the current working directory, which is fine for ./database living alongside workspace/ in a single repo checkout but not for a registry meant to be shared across checkouts.

Identity: `DATA_DESCRIPTION.md` content hash¶

Matching is by the content hash of a data directory's DATA_DESCRIPTION.md — a candidate, never an automatic identity. A human always confirms reuse vs. registering a new variant. This is why DATA_DESCRIPTION.md is described as the registry's identity key in Writing DATA_DESCRIPTION.md — two directories with the same catalogue but a different DATA_DESCRIPTION.md are different candidates, and a directory whose description changes stops matching its old registration.

The registration decision flow¶

There are two ways a dataset gets into the registry:

1. Register from scratch, before any session exists:

register_dataset(data_dir, name="my_survey_shear", version="v1")

Standardises the directory straight into the persistent store. Use this to seed the registry ahead of time, or to register a variant that doesn't match what a session just happened to standardise.

2. Register post-ingest — file-backed gate. The first successful ingest_to_session on a directory with a DATA_DESCRIPTION.md that has never been registered opens a kind="registration", gate_id="registration-<basename>" human gate with options ["register", "ignore"], and sets dataset_decision = "registration_pending". Every other pipeline tool (compute_*_from_session, plotting, covariance, SACC assembly — everything except the ingest/registry tools) refuses with ERROR: registration decision pending until authorise_gate closes the gate: "register" persists the products the session just standardised (no recompute); "ignore" records ephemeral and the workspace stays disposable. The gate is asked exactly once per session, never silently re-assumed.

Reusing a registered dataset¶

A later ingest_to_session call against a data directory that already matches a registered dataset opens a kind="registry", gate_id="registry-<basename>" gate BEFORE any standardisation, with options [<candidate_dataset_id_1>, ..., "register-new", "ignore"], and returns {"needs_input": ..., "gate_id": ..., "orchestrator_relay": ...}. Close it via authorise_gate(session_dir, gate_id, <one of the options>). On the next dispatch, ingest sees the recorded decision and reuses the stored products (linked id), standardises fresh (register-new / ignore), or opens the registration gate as appropriate.

Programmatic affordances that bypass the gate (used from CLI or when the task explicitly names an exact registered dataset):

# reuse the stored products — no recompute, manifest points at the store
ingest_to_session(session_dir, ..., reuse_dataset_id="<id>")

# this run is a distinct variant — register it as a new dataset in one call
ingest_to_session(session_dir, ..., register_as="my_survey_shear:v2")

# skip the registry entirely for this run
ingest_to_session(session_dir, ..., ignore_registry=True)

If the task already names an exact registered dataset, pass dataset_tag= with that string verbatim — a deterministic lookup with no candidate search and no human gate. It accepts either form:

a full dataset_id exactly as printed by list_datasets / cosmotron-data-registry list, e.g. dataset_tag="my_survey__galaxy_shear__v1" — this is the usual case, since the id is the only identifier those surfaces display;
a "<name>[:<version>[:<probe>]]" colon tag, e.g. dataset_tag="my_survey:v1:galaxy_shear".

A bare name ("my_survey") also resolves when it matches exactly one registered dataset; if it matches several, the call returns needs_input with dataset_candidates rather than guessing. On no match the error names the real registered ids — never retype or "correct" a tag by hand.

reuse_dataset_id/dataset_tag also work with append=True — a 3×2pt session's second-and-later catalogue reuses its own registered dataset with zero re-standardisation, merging into the session's existing manifest instead of overwriting it. The reuse decision persists per catalogue (keyed by catalogue_path's basename via session_context.json["dataset_decisions"]), so it never overwrites the primary catalogue's own decision.

Reuse is always answered through a file-backed gate at ingest. Before any standardisation runs, ingest_to_session scans the registry for candidates matching this catalogue's DATA_DESCRIPTION.md hash (narrowed by probe and basename). If one or more match, the tool opens a kind="registry", gate_id="registry-<basename>" human gate whose options enum lists every concrete candidate dataset_id plus "register-new" and "ignore", and returns an orchestrator_relay string. @cosmotron closes it via authorise_gate(session_dir, gate_id, <one of the options>) — a permission: ask picker dialogue with the concrete choices. Bootstrap never asks about the registry; the decision lives with the tool that consumes it.

Multi-catalogue reuse (a 3×2pt directory)¶

Each catalogue's reuse gate opens on its OWN ingest call, keyed by its own basename. A 3×2pt session dispatches @data_ingestor once per catalogue (primary + append=True for the second), and each call opens its own registry-<basename> gate when a candidate exists. Close each gate independently via authorise_gate. Decisions persist per-basename in session_context.json["dataset_decisions"]; the first catalogue's decision also mirrors into the scalar dataset_decision/dataset_id fields every other tool reads. Once every gate is closed, both ingest calls reuse their registered products with no re-standardisation.

Discovering what's registered¶

find_dataset(data_dir="/data/my_survey_shear")   # candidates matching a directory
find_dataset(name="my_survey_shear")              # candidates matching a name
list_datasets()                                    # everything registered

Variants are distinct datasets¶

Single-bin, tomographic, masked, and cosmic-shear ingests of the same underlying catalogue family are four different datasets — register each under a distinct name/version (probe is also part of the dataset id, so clustering vs. shear ingested from the same raw directory get distinct ids automatically). See also Catalogue-type profiles.

Duplicate `(description_hash, source_catalogue)` rows¶

Registering the same (description_hash, source_catalogue) tuple under a DIFFERENT name/version is intentional: each registration is a distinct candidate the reuse gate then offers, and the human picks one at authorise_gate time. So register_dataset does not de-duplicate on the tuple; two rows are a feature, not a corruption.

But un-intended duplicates — accreted from copy-paste, batch scripts, or a prior tree that predated register_session(prune_session_copy=True) — are worth noticing. Opening the registry (i.e. the first register/get/find_candidates/list_datasets call in a process) emits a UserWarning naming every pre-existing duplicate tuple with the concrete dataset ids sharing it, once per process. The store is never mutated; run cosmotron-data-registry dedup to consolidate.

Managing the registry — `cosmotron-data-registry`¶

An operator CLI (installed by the package; entry point cosmotron-data-registry, human-run) to inspect and manage the registry:

Command	What it does
`cosmotron-data-registry list`	Table of every dataset — id, probe, bins, objects, on-disk size, locations (which machines hold it), created.
`cosmotron-data-registry show <id>`	Full record: name/version/probe, `column_map`, `z_edges`, `stats`, per-bin product paths, and where each copy lives.
`cosmotron-data-registry rm <id> [--keep-products]`	Delete the registry row AND its local product store dir. Pass `--keep-products` to only drop the row (rarely useful — leaving multi-GB stores behind was the ses_0392 disk-orphan bug).
`cosmotron-data-registry push <id> <site>`	Materialise the dataset's products on a cluster (Spark → site), recording `locations[<site>]` — so remote compute reuses the cluster copy instead of re-staging per job (see below).
`cosmotron-data-registry pull <id> [<site>]`	Fetch a dataset's products back to the Spark canonical store (cluster → Spark), adding `locations["local"]`.
`cosmotron-data-registry register <data_dir> [...]`	Standardise a raw data directory into the registry — offline, no LLM. See below.
`cosmotron-data-registry dedup [--workspace DIR] [--apply]`	Reclaim disk: session-local copies byte-identical to a store product get their `manifest.json` rewritten to the store path and deleted; separately reports (never deletes) registered datasets that are byte-identical duplicates of each other. Dry-run by default — pass `--apply` to actually reclaim.

Agents get the same view through the MCP tools list_datasets (now includes probe + locations) and get_dataset(dataset_id) (the full record).

register_session(session_dir, name, version, ..., prune_session_copy=True) (the default for every register_session an agent calls) already stops NEW duplication going forward — once its store copy is verified and the session's manifest.json is rewritten to point at it, the session's own copy is deleted. dedup is for the disk that accumulated BEFORE that existed: a session that never resolved its reuse gate re-standardised its catalogue from scratch, or a register_session call predating prune_session_copy left its own copy sitting next to the store's forever.

Registering a dataset from the CLI — `register`¶

register <data_dir> wraps the same streaming, bounded-memory tools.ingest.register_dataset the agents use, so a well-described multi-GB catalogue can be registered directly from the operator's terminal. It resolves what to register in this order — the first that matches wins:

Explicit --name (+ column/probe flags) — a single ad-hoc dataset, e.g. cosmotron-data-registry register /data/my_survey --name my_survey_shear --probe galaxy_shear --ra-col RA --dec-col DEC ....
--spec FILE — a YAML or JSON file holding either a single mapping of register_dataset kwargs, a bare list, or {"datasets": [...]} for registering several catalogues (e.g. a 3×2pt directory's lens + source catalogues) in one call. CLI flags override values from the spec; --name may only be used to override a spec with exactly one entry.
An auto-discovered <data_dir>/dataset.yaml — read non-interactively if present (and --interactive wasn't forced). This file is also read by bootstrap_session itself (the MCP tool, not just this CLI) — it becomes session_context.json["catalogues"], the multi-catalogue plan a 3×2pt session's ingest and registry-reuse steps consume. Once both catalogues in a directory are registered (via this CLI or register_session), the SAME dataset.yaml is what lets bootstrap_session resolve the reuse gate PER CATALOGUE in one call — see "Multi-catalogue reuse" below — instead of a single ambiguous reuse answer that can never disambiguate two catalogues sharing one DATA_DESCRIPTION.md.
The interactive wizard — runs when stdin is a TTY (or --interactive forces it) and none of the above matched. It inspects the directory, proposes probe/column/tomography/mask/nside/n(z)/theory defaults for the user to confirm (deterministic detection, no LLM), loops over every catalogue file it finds (one dataset per probe for a 3×2pt directory), and writes the confirmed answers back to <data_dir>/dataset.yaml — so the next run against the same directory is reproducible and non-interactive.
Otherwise the command fails, naming exactly which flag/file/mode to supply.

Other flags: --force re-registers even if the dataset id already exists (the default is to skip a duplicate, not error); --non-interactive refuses to prompt at all, failing instead if no --name/--spec/dataset.yaml is available (for CI/scripted use).

Example — registering two catalogues from a spec file in one call:

# spec.yaml
datasets:
  - name: my_survey_lens
    version: v1
    probe: galaxy_density
    catalogue_path: lens.fits
  - name: my_survey_src
    version: v1
    probe: galaxy_shear
    catalogue_path: source.fits

cosmotron-data-registry register /data/my_survey --spec spec.yaml

Location-aware records (remote execution)¶

The registry stays canonical on the machine that runs the agents (e.g. the Spark), but each DatasetRecord tracks where its standardised products physically live, in a locations map — {"local": <store path>} for the normal case, plus a site name → cluster path entry once the dataset is materialised on a cluster (cosmotron-data-registry push <id> <site>, or a remote ingest that kept large products above the site's pull_threshold).

Reusing a registered dataset remotely. A reused dataset's manifest points at the Spark store, which a remote job can't read — so before running spectra/covariance on a cluster with a reused dataset, push it once: cosmotron-data-registry push <id> <site>. The remote submit then rewrites the manifest's product paths to the cluster copy (locations[<site>]) and skips re-staging the catalogue. If you forget, the submit fails loud naming the exact push command — never a cryptic remote error. A dataset that lives only on a cluster likewise refuses local consumption, naming the fix (run remotely, or cosmotron-data-registry pull <id>). Old records with no locations field read as {"local": ...}, so nothing needs migrating. See Data placement.

Full tool signatures: Ingest & catalogue reference.