Skip to content

Data placement & the dataset registry

Short answer: keep your raw catalogue wherever is convenient for you — on the Spark if it's a normal-sized download, on the cluster's own filesystem if it's a large or shared dataset already staged there. Either works; cosmoTRON stages exactly the files each remote step needs, automatically, either way. This page explains the mechanics so you can reason about what actually crosses the network and when.

Bare paths vs. site:path references

Every place cosmoTRON accepts a data path (bootstrap_session(data_dir=...), ingest_to_session's mask_path/nz_path/theory_path overrides, ...) accepts one of two forms:

  • A bare path (data/my_survey/, /home/you/data/cat.fits) — means "this lives on the machine driving the session" (the Spark, in the usual setup). This is the common case and needs nothing special.
  • A site:path reference (sunrise:/cfs/nobackup/mygroup/my_survey/) — means "this lives on the named cluster's filesystem already". site must be a name configured in sites.yaml.

Both forms round-trip through one parser (cosmotron_mcp.remote.locations.parse_ref) used everywhere a location is resolved — bootstrap, ingest staging, and the dataset registry all agree on the same syntax.

Bootstrapping from data that's already on the cluster

bootstrap_session(
    data_dir="sunrise:/cfs/nobackup/mygroup/glass_mock/",
    user_task="...",
)

When data_dir carries a site prefix, cosmoTRON enumerates that directory over a lightweight SSH ls (safe on a login node — it's a listing, not compute) instead of scanning locally, finds the catalogue + any DATA_DESCRIPTION.md/ancillary files, and records execution_site in session_context.json. The session is then pinned to that site (remote_state.json, written once, never hand-edited): every subsequent remote step for this session must target the same site — submitting to a different one is a hard error (start a new session to switch clusters).

Mixing sites across inputs (e.g. a catalogue on sunrise: but a mask file that's Spark-local) is refused with needs_input rather than guessed — a session lives on one site.

What actually gets staged, and when

Nothing is ever synced wholesale. Every remote-capable tool declares its own input closure — the exact files it reads (session context, manifest, the relevant per-bin Cℓ JSON, a mask, an n(z) file, ...) — and only those are pushed at submit time. jobs/, results/plots/, and unrelated map-scale FITS never cross the wire for, say, a covariance job.

Remote ingest

ingest_to_session(session_dir, jobspec={"backend": "slurm", "site": "sunrise"}) runs the standardisation itself as a Slurm job — a multi-GB FITS read is compute-node work, not something to do on a login node or ship back and forth. What happens:

  1. If the catalogue is a bare (Spark-local) path, it's pushed to the cluster once, alongside any other Spark-local ancillary files the ingest args reference (mask_path, nz_path, theory_path, ...). If it's already a site:path reference, it's already there — nothing to push.
  2. The job runs the real, deterministic ingest tool on the cluster, writing the standardised per-bin FITS + manifest.json there.
  3. On completion, manifest.json, catalogue_stats, and n(z)/ancillary outputs are always pulled back (they're small — KB to a few MB).
  4. Standardised per-bin FITS are pulled back too, up to the site's pull_threshold (default 2 GiB per file). A product larger than that stays on the cluster as the primary copy, and its artefact-registry entry is tagged location: "remote:<site>" instead.

Reusing what's already on the cluster

A later remote step (spectra, covariance) checks whether a given per-bin FITS is already present on the cluster before pushing it — so a spectra job dispatched right after a remote ingest never re-uploads the bins it just produced there. This falls out of the same input-closure staging, not a special case.

The dataset registry is location-aware

The persistent dataset registry (see The dataset registry for the full registration flow) stays canonical on the Spark, but a DatasetRecord now tracks where its standardised products physically live:

record.locations
# {"local": "/path/on/spark/database/datasets/my_survey__v1"}                 # normal case
# {"local": "...", "sunrise": "/cfs/nobackup/.../my_survey__v1"}              # both copies exist
# {"sunrise": "/cfs/nobackup/.../my_survey__v1"}                              # remote-only (over pull_threshold)

A dataset that only has a remote-only entry refuses local consumption with a legible error rather than a confusing missing-file crash:

dataset 'my_survey__v1' has no Spark-local copy — its products live only on
sunrise:/cfs/nobackup/.../my_survey__v1. Either run the step remotely
(jobspec={"backend": "slurm", "site": "sunrise"}), or pull the data to the
Spark first with `cosmotron-remote pull-data my_survey__v1`.

Both remedies it names are real: pass the same jobspec to whatever tool needs the data, or fetch it once with the operator command below.

Materialising a registered dataset on a cluster

The clean way to run reused registered datasets remotely: push the dataset's products to the cluster once, then remote spectra/covariance read the cluster copy (no per-job catalogue re-staging).

# push a registered dataset's products to a site (records locations[<site>])
cosmotron-data-registry push my_survey__v1 sunrise

# inspect where everything lives
cosmotron-data-registry list
cosmotron-data-registry show my_survey__v1

# pull a cluster-primary dataset back to the Spark canonical store
cosmotron-data-registry pull my_survey__v1

After push, a remote spectra/covariance submit that reuses my_survey__v1 rewrites the manifest to the cluster copy automatically. If you skip the push, the submit fails loud naming the exact push command — never a cryptic remote error. (cosmotron-remote push-data <site> <local> <remote_dir> still exists for staging an arbitrary file/dir outside the registry; cosmotron-remote pull-data is now a deprecated alias of cosmotron-data-registry pull.)

Summary — does it matter where you keep the data?

  • Normal-sized catalogue, not already on a cluster: keep it on the Spark. Nothing to think about — bootstrap_session/ingest_to_session read it locally, and any later remote step stages only what it needs.
  • Catalogue already on the cluster (shared dataset, or too large to duplicate): bootstrap with a site:path reference and run ingest with a jobspec targeting the same site — the standardisation happens where the data already is, and only small products (plus, up to pull_threshold, the per-bin FITS) come back.
  • Reusing a registered dataset on a cluster: cosmotron-data-registry push <id> <site> once, then remote steps read the cluster copy (no re-staging).
  • A dataset ends up registered remote-only: that's fine — the registry tells you exactly what to do (run remotely, or cosmotron-data-registry pull) rather than failing silently.