Skip to content

data_ingestor

mode: subagent — always the first execution step, after the plan is approved.

What it does

Standardises the raw catalogue and writes manifest.json by calling ingest_to_session — one call handles inspect → column resolution → standardisation → per-bin stats → ancillary file discovery. For a 3×2pt task it dispatches once per probe (see Multi-format ingestion).

Registry decisions

Owns the entire dataset registry decision surface — bootstrap is registry-agnostic. Two file-backed human gates can open per catalogue:

  • registry-<basename> — opens BEFORE any standardisation when the catalogue's DATA_DESCRIPTION.md hash matches one or more registered datasets. options = [<candidate_dataset_id_1>, ..., "register-new", "ignore"].
  • registration-<basename> — opens AFTER standardising a previously unseen dataset. options = ["register", "ignore"].

ingest_to_session returns the gate_id string and an orchestrator_relay string alongside needs_input; the agent prints the relay verbatim so @cosmotron closes the gate via authorise_gate(session_dir, gate_id, decision) — the ONLY close path, carrying opencode's permission: ask on the primary agent. No free-text dataset_decision=/user_reply= kwargs remain on bootstrap_session or the ingest surface; the decision lives entirely in the gate file and the runtime confirmation dialogue.

Remote ingest. Data on a cluster (site:path data_dir) or a session that pins ingest remote makes ingest_to_session run as a Slurm job — the agent polls the async receipt to completion, unchanged otherwise. See Remote execution.