Remote execution — overview¶

cosmoTRON's numerical pipeline usually runs wherever the MCP server runs. On a DGX Spark that's fine for maps/spectra at moderate nside, but multi-GB catalogues, TJPCov covariances, and CosmoSIS/Firecrown chains want real CPU allocations on an HPC cluster. Remote execution lets any of those steps run as a Slurm job instead, while the session — session_context.json, manifest.json, results/, the artefact registry — stays exactly where it already is. Nothing changes about how you talk to the agents; you just add a jobspec to a tool call.

compute_all_cls_from_session(session_dir, jobspec={"backend": "slurm", "site": "sunrise"})
# → {"job_id": "...", "status": "pending", "mode": "async", "site": "sunrise",
#    "resource_class": "spectra", "next": "check_job"}

The async model is unchanged from local background jobs: submit returns a job_id immediately, you poll with check_job, results land back in the session automatically. No process lives on the cluster between polls — every check is a short-lived command from wherever the agent runs.

Which tools have a remote path¶

Tool	Resource class
`ingest_to_session`	`ingest`
`compute_cls_from_session`, `compute_all_cls_from_session`	`spectra`
`compute_systematics_cls_from_session`, `run_bmode_null_tests_from_session`	`spectra`
`compute_cterm`, `compute_psf_leakage`, `build_systematics_evidence`	`spectra`
`compute_covariance_from_session`, `compute_full_covariance_from_session`, `compute_validation_covariance_from_session`	`covariance`
`run_inference`	`inference`
`run_systematics_check_job` (already async by design)	`spectra`

run_power_spectrum and apply_template_deprojection are session-less low-level primitives (a bare fits_path / inline map arrays, nothing to build a session closure from) — they have no remote path. Use compute_cls_from_session(deproject_templates=[...]) instead, which does.

Every remote-capable tool takes the same jobspec shape and returns the same async receipt — {job_id, status, mode: "async", site, resource_class, next: "check_job"} — deliberately disjoint from a synchronous result so nothing downstream can mistake a submission receipt for actual data. Poll with check_job(session_dir, job_id).

Triggering it from a task (not raw Python)¶

The jobspec examples above are the low-level API. In normal use you never write one — you say in the task which steps run on the cluster, and the tools attach the jobspec themselves (the small local model never picks a site or a step). Two task-level signals, both parsed deterministically at bootstrap_session:

## remote: block — names which heavy step classes run remotely:
```
## remote:
site: sunrise
covariance, inference
```
## site: sunrise alone = all classes on that site. This resolves to session_context.json's pipeline_config.remote = {site, steps}, and each remote-capable wrapper auto-derives its own jobspec from it (keyed by the tool's resource class). See Writing TASK.md.

Optionally add a queue: line to steer every step in the block to a non-default sites.yaml queue key (e.g. a bigger-memory queue) instead of the resource class's own default:

## remote:
site: sunrise
queue: bigmem
covariance, inference

This is global to the whole block, like site: — there's no per-step queue syntax. It's a lower-precedence knob than the low-level jobspec fields below (an expert caller passing partition/qos directly still wins).

The step classes you can list (comma/newline separated, or all for every one). An unknown token fails loud at bootstrap, naming the valid set:

Class	Covers	Notes
`ingest`	`ingest_to_session`	Auto-forced when data is `site:path` — no need to list it for cluster-resident data.
`spectra`	`compute_cls_from_session`, `compute_all_cls_from_session`, `compute_systematics_cls_from_session`, `run_bmode_null_tests_from_session`, `compute_cterm`, `compute_psf_leakage`, `build_systematics_evidence`	The catalogue-bound systematics checks live here too.
`covariance`	`compute_covariance_from_session`, `compute_full_covariance_from_session`, `compute_validation_covariance_from_session`	Isolated, small inputs — the cleanest first remote test.
`inference`	`run_inference`	Needs the site's `env: FIRECROWN_DIR`/`COSMOSIS_STD_LIB_DIR` set + the cosmosis standard library built on the cluster once.
`all`	expands to the four above

run_power_spectrum / apply_template_deprojection are session-less primitives (bare arrays / fits_path, no session closure to stage) and have no remote path — use compute_cls_from_session(deproject_templates=...). - site:path data — bootstrap_session(data_dir="sunrise:/cfs/.../cat") pins the session to that site and forces ingest remote (a catalogue on the cluster can't be read locally). See Data placement.

Under the hood, when a step is pinned remote the tool returns an async receipt instead of sync numbers; the executor agents (data_ingestor, analysis_coder, systematics_checker) poll it through automatically — the same background-job flow inference already uses. You just get results in the session, wherever they were computed.

Two transports, one config file¶

A site in sites.yaml is one remote Slurm cluster, reached by one of two transports:

transport: {type: ssh} (the default) — the agent runs on one machine (e.g. the Spark) and the cluster is reached over SSH. Inputs are staged with rsync (only the files a tool actually needs — never a blind session sync), and results/artefacts are pulled/merged back on completion. This is the "Spark orchestrates, cluster computes" shape — see the Spark + Sunrise walkthrough.
transport: {type: local} — the agent runs on the cluster itself (e.g. an interactive GPU-node allocation, with the GPU serving the local model) and dispatches CPU-node Slurm jobs over the shared filesystem. sbatch/squeue/sacct become direct subprocess calls — no SSH, no rsync, no artefact-registry merge, because the CPU job's session tree is the agent's own tree. See NERSC & same-cluster agents.

Both shapes can coexist as separate entries in the same sites.yaml — pick per call via jobspec["site"].

`sites.yaml` — cluster configuration¶

Pydantic-validated YAML, loaded from the first file found in this order:

$COSMOTRON_SITES (explicit file path)
./sites.yaml (repo root — gitignored; holds a username + port)
~/.cosmotron/sites.yaml

sites.yaml holds roots only — actual data paths always come from the dataset registry or explicit site:path references (see Data placement), never hardcoded here. A copy-paste starting point (Sunrise SSH, NERSC sshproxy, and a same-cluster local shape) is committed at sites.yaml.example; cosmotron-remote add-site (below) probes a cluster and writes a validated entry for you.

Field	Required	Notes
`transport.type`	no	`ssh` (default) or `local` (Phase R6 — see Same-cluster execution).
`shared_filesystem`	only with `transport: local`	Must be `true` — the explicit acknowledgement that the session tree is shared with the CPU jobs.
`host` / `port` / `user`	yes for `ssh`; optional for `local` (default `localhost`)	SSH endpoint.
`fallback_hosts`	no	Tried on a connect timeout against `host`.
`auth`	yes for `ssh`; defaults to `none` for `local`	`{type: static_key, key_path: ~/.ssh/id_...}`, `{type: sshproxy, key_path, proxy_hint}` (NERSC — see NERSC), or `{type: none}` (same-cluster).
`remote_root`	yes for `ssh`; defaults to cwd for `local`	Acts as the remote repo root: holds the installed `cosmotron_mcp` package + its own `workspace/`. Every cosmoTRON path is repo-root-relative, so a session mirrored here resolves identically on both machines.
`scratch_root`	no	Large-data cache (catalogues, per-bin FITS).
`pull_threshold`	no	Per-file byte cutoff (default 2 GiB) above which a standardised product from a remote ingest stays on the cluster instead of being pulled back — see Data placement.
`env_setup`	yes (usually)	Lines run before every job to activate the env, e.g. `module load conda`, `conda activate cosmotron`.
`env`	for inference	Exported into every job. Remote `run_inference` requires `FIRECROWN_DIR`/`COSMOSIS_STD_LIB_DIR` — submit fails early, naming `sites.yaml`, if either is missing. Fix a `doctor` warning about either with `cosmotron-remote edit-site <site> --set-env FIRECROWN_DIR=<path> --set-env COSMOSIS_STD_LIB_DIR=<path>` (or just answer `doctor`'s own prompt), rather than hand-editing the YAML.
`queues`	yes (`default`)	Abstract queue name → concrete `{partition, qos, max_cpus, max_mem, max_walltime}`. `partition` is optional (NERSC selects via `qos` + a hardware constraint, no `-p`).
`slurm.sbatch_extra`	no	Verbatim extra `#SBATCH ...` lines (site quirks, e.g. NERSC's `-C cpu`/`-A <account>`).
`slurm.srun_args`	no	Extra args on the `srun` line for MPI jobs, e.g. `["--mpi=pmix"]`.
`slurm.poll_min_interval_s`	no	Rate-limits polls (default 10 s; same-cluster sites can go lower — local `squeue` is cheap).
`resources`	no	Per-site override of the shipped resource classes.

Resource classes. Tools declare an abstract need (probe, ingest, spectra, covariance, inference — see cosmotron_mcp/remote/resources.py); the site's queues map translates it into concrete partition/CPU/memory/ walltime. So a jobspec normally just names backend+site and gets sane resources without anyone being asked "how much memory?". A per-call jobspec field (cpus/mem/walltime/partition/qos/queue) overrides any default — queue names a different sites.yaml queue-map key (what the task-level ## remote: queue: <name> line above threads through to); qos overrides the resolved queue's QOS on its own, with or without an explicit partition/queue; partition is the lowest-level raw escape hatch.

`cosmotron-remote` — the operator CLI¶

A maintenance CLI (installed by the package; entry point cosmotron-remote), run by a human in a terminal — never by an agent (agents have no interactive channel). Provisioning is manual-once + documented, verified at submit time.

Command	What it does
`cosmotron-remote add-site`	Interactive onboarding wizard. Detects an active Slurm allocation (`$SLURM_JOB_ID`) and offers a same-cluster `transport: local` entry instead of SSH when appropriate. For an SSH site: connects in, probes `sinfo`/`scontrol`/`module`/`quota`/`groups`, asks only what a machine can't answer, submits a 1-minute hello job, and writes a validated `sites.yaml` entry.
`cosmotron-remote view <site>`	Read-only. Prints which `sites.yaml` file the site was loaded from and its full resolved config — transport/host/roots, `env_setup`, `env` (so a missing `FIRECROWN_DIR`/`COSMOSIS_STD_LIB_DIR` is visible directly), every queue's `partition`/`qos`/limits, and `slurm`/`resources`. Use this before `edit-site` to see current state.
`cosmotron-remote edit-site <site> [...]`	Adds/removes a queue or sets env vars on an existing site, re-validating through the same schema `add-site` uses before writing (a bad edit is never partially written). `--add-queue NAME --partition P [--qos Q] [--max-cpus N] [--max-mem M] [--max-walltime T]`, `--remove-queue NAME` (refuses to remove `default`), `--set-env KEY=VALUE` (repeatable) — e.g. `cosmotron-remote edit-site sunrise --set-env FIRECROWN_DIR=/path/to/firecrown`.
`cosmotron-remote doctor <site>`	One-shot contract check. SSH sites: reachable (incl. fallback), `sbatch` present, `remote_root` writable, `env_setup` runs, `cosmotron_mcp` importable, version matches local, `FIRECROWN_DIR`/`COSMOSIS_STD_LIB_DIR` resolve. Same-cluster sites: runs the gating `sbatch --wrap` probe first (some clusters forbid compute-node submission — this is how you find out), then `remote_root` writable + `env_setup` + import. Both dump `sinfo -l`/`scontrol show partition` for the operator. Run from an interactive terminal, a missing `FIRECROWN_DIR`/`COSMOSIS_STD_LIB_DIR` prompts inline for a value and writes it to `sites.yaml` on the spot (same write path as `edit-site --set-env`); non-interactive runs are unaffected and just report `[SKIP]`.
`cosmotron-remote sync-code <site>`	rsyncs `cosmotron_mcp/` + `pyproject.toml` + `README.md` to `remote_root` and `pip install -e`s it in the remote env. The Spark stays the single source of truth (uncommitted WIP is pushed; no git needed on the cluster).
`cosmotron-remote bootstrap <site>`	One-time provisioning, in order: create the conda env from the committed `envs/cosmotron-hpc.lock.yml`, then `sync-code` (rsync + `pip install -e` into that env), then run `doctor`. Idempotent.
`cosmotron-remote push-data <site> <local> <remote_dir>`	Stage a local file/dir to a site's scratch — the Spark→cluster data-staging direction (see Data placement).
`cosmotron-remote pull-data <dataset_id>`	Fetch a dataset's remote-primary products back to the Spark's canonical registry store, updating its `locations`.

Typical first-time setup:

cosmotron-remote add-site            # or hand-write sites.yaml from sites.yaml.example
cosmotron-remote bootstrap sunrise   # sync code + conda env + doctor
cosmotron-remote doctor sunrise      # re-verify any time

For the full step-by-step (including a real submitted job), see the Spark + Sunrise walkthrough.

Async jobs, wherever they run¶

Every remote job is tracked the same way a local background job is ({session_dir}/jobs/{job_id}/job.json): submit_job/check_job/ list_jobs/cancel_job all work identically regardless of backend. check_job additionally carries, for a remote job: site, slurm_state, stdout_tail, and poll_error/stale/merge_error — a transport error never corrupts job status; it's soft-surfaced so you can tell "still running, just couldn't poll it a moment ago" from "actually failed".

If an expired credential (an SSH key permission problem, or an expired NERSC sshproxy certificate) is the cause, poll_error/the submit error field carry an auth_required: ... prefix naming the fix — see NERSC & same-cluster agents.