cosmotron_mcp.tools.ingest¶
ingest_to_session ¶
ingest_to_session(session_dir: str, survey: str | None = None, z_edges: list[float] | None = None, ra_col: str | None = None, dec_col: str | None = None, z_col: str | None = None, weight_col: str | None = None, e1_col: str | None = None, e2_col: str | None = None, psf_e1_col: str | None = None, psf_e2_col: str | None = None, probe: str | None = None, sigma_e: float | None = None, m_values: list[float] | None = None, sigma_m: float | list[float] | None = None, subtract_cterm: bool | None = None, selection: list[dict] | None = None, keep_zero_weight: bool = False, bin_col: str | None = None, bin_values: list[int] | None = None, mask_kind: str | None = None, mask_from_weights: bool = False, completeness_path: str | None = None, catalogue_path: str | None = None, append: bool = False, mask_path: str | None = None, nz_path: str | None = None, nz_paths: list[str] | None = None, theory_path: str | None = None, theory_paths: list[str] | None = None, theory_sacc_path: str | None = None, reuse_dataset_id: str | None = None, dataset_tag: str | None = None, register_as: str | None = None, ignore_registry: bool = False, jobspec: dict | None = None) -> dict
Deterministically standardise the session catalogue and write manifest.json.
One call replaces the whole hand-written ingest step: reads session_context for catalogue_path + pipeline_config, auto-detects RA/DEC/Z/weight columns, standardises the catalogue (single bin, or per-bin from z_edges), computes per-bin catalogue_stats incl. n_bar, discovers ancillary n(z)/theory/mask files, writes manifest.json, and registers artefacts.
Column overrides: pass ra_col / dec_col / z_col / weight_col when a column has a NON-STANDARD name (e.g. z_col="PHOTO_Z") or auto-detection is ambiguous — an explicit name overrides detection. Use the column names from DATA_DESCRIPTION.md when they are non-standard, rather than relying on auto-detection.
@data_ingestor should call ONLY this — never write a bespoke ingest script.
If a required column can't be resolved it returns {"needs_input": ...} naming
the fix (re-call with z_col=
Ancillary files: explicit mask_path/nz_path/theory_path override discovery; multiple masks auto-combine; for tomography, n(z) files are matched per bin automatically (or pass nz_paths=[one per bin]); per-bin theory Cℓ files must be passed explicitly as theory_paths=[one per bin] — they cannot be auto-matched reliably. If no theory files exist for this dataset, omit both theory_path and theory_paths.
theory_sacc_path is an ALTERNATIVE to nz_path/nz_paths/theory_path/
theory_paths, not a fifth independent source: when the dataset's n(z) and
theory Cℓ live bundled in one SACC file (.fits or .sacc/.h5) instead of
separate plain-text files, pass theory_sacc_path=
Shear (spin-2) catalogues: pass e1_col/e2_col (auto-detected from e1/e2/g1/g2 patterns otherwise) and probe="galaxy_shear" (or declare it in the task / profile). Per-bin sigma_e is the user value (sigma_e= arg or the per-probe pipeline_config block) when given, else the measured Heymans+2012 estimate; n_eff_arcmin2 is computed from the footprint. Both land in the manifest bin.
Real-catalogue capabilities (all named parameters — the tool owns the mechanics, the caller only names the choice; take it from DATA_DESCRIPTION.md / the task / the survey= profile, never invent it): - selection=[{col,op,value}, ...]: declarative quality/flag cuts (closed op set ==,!=,<,<=,>,>=,in,not_in,finite; AND-combined; NO eval). A resolved weight column adds an implicit weight>0 cut (zero-weight = non-detection) unless keep_zero_weight=True; selection=[] disables profile cuts. Recorded per catalogue with n_before_selection / n_after_selection counts. - bin_col=/bin_values=: tomography by an INTEGER bin-ID column (Euclid TOM_BIN_ID, DES ZBIN). Mutually exclusive with z_edges; omit bin_values to auto-enumerate distinct non-negative IDs (negatives excluded). - mask_from_weights=/mask_kind=: build the shear mask as per-pixel Σw (auto for maskless spin-2); weight masks are never element-wise-max combined; f_sky_eff recorded. completeness_path=: HEALPix completeness divided into counts inside the footprint (floor 1e-3) — the pseudo-Cℓ substitute for randoms. - psf_e1_col/psf_e2_col: carried through as PSF_E1/PSF_E2 (Phase-D α-leakage). - m_values=/sigma_m=: per-bin multiplicative shear bias + uncertainty; length must equal n_bins; also resolvable from a ## probes: m=/sigma_m= block or a profile calibration:. m is DIVIDED OUT of the shear ellipticities at ingest (2110.06947 Eq. 2), so the measured Cℓ / data vector are m-corrected; sigma_m is consumed by the covariance Eq. 20 marginalisation term (marginalise_sigma_m=). - subtract_cterm=: subtract the weighted-mean additive c-term from E1/E2 at ingest (2110.06947 Eq. 1). Default (None) resolves via the survey= profile, else OFF — so a synthetic/unprofiled shear session is unchanged; KiDS-style profiles set it true. Euclid-style _CORRECTED catalogues set it false (already corrected upstream). The Phase-D compute_cterm then measures the RESIDUAL c-term.
Multi-probe (3x2pt) sessions: call once per catalogue. The first call ingests
the session's primary catalogue (session_context.json["catalogue_path"], the
default when catalogue_path is omitted); each subsequent call passes
append=True with an explicit catalogue_path=
catalogue_path resolution (the default primary-catalogue pin, and any override — same rule as mask_path/completeness_path): tried in order as given verbatim (works when already absolute), then joined onto data_folder, then joined onto the repo root; the first that exists wins. A path that matches none of the three returns {"needs_input": ...} naming both roots tried and the files actually present in data_folder — never a bare FileNotFoundError. A bare basename from DATA_DESCRIPTION.md/dataset.yaml (e.g. "glass_lsst_y1_source_catalogue.h5") resolves correctly without needing an absolute path.
L1 registry (persistent reuse) — file-backed gate. When the data
directory has a DATA_DESCRIPTION.md matching one or more already-
registered datasets (narrowed by probe + catalogue basename), this tool
opens a kind="registry", gate_id="registry-<basename>" gate whose
options enum lists the concrete candidate dataset_ids plus
"register-new" and "ignore", and returns
{"needs_input": ..., "gate_id": ..., "gates_open": [...],
"orchestrator_relay": ..., "dataset_candidates": [...]}. The
orchestrator_relay string is self-contained (session_dir + gate_id +
options + action) — the subagent prints it verbatim; @cosmotron
closes the gate via authorise_gate(session_dir, gate_id, <one of
options>) (permission: ask picker). Idempotent: re-calling
ingest_to_session while the gate is open returns the SAME gate and
relay. The gate's side-effect writes dataset_decisions[basename]
(and, for the primary catalogue, the scalar
dataset_decision/dataset_id mirror) to session_context.json;
on the next call this tool sees the decision and either reuses the
stored standardised products (no recompute; spin/probe/sigma_e
restored) or standardises fresh. Clustering vs shear from the same
directory are distinct datasets (probe is part of the id). Per-bin
theory_paths are stored and reconstructed on reuse.
Reuse works on append=True too — the same per-basename gate opens
on the appended catalogue's own ingest call, keyed by its own basename;
the primary catalogue's decision is never overwritten. A 3×2pt session
with two already-registered catalogues costs zero re-ingestion.
dataset_tag (an exact registered dataset id / colon-tag from the
user's task) resolves and reuses directly — no candidate search, no
gate. It accepts EITHER a full dataset_id exactly as printed by
list_datasets / cosmotron-data-registry list (e.g.
"glass_450_shear__galaxy_shear__v2") OR a
"<name>[:<version>[:<probe>]]" colon tag (e.g.
"glass_450_shear:v2:galaxy_shear"). Pass whatever the task gives
you VERBATIM — never retype or "correct" it. needs_input names the
real registered ids on mismatch (never silently falls back to
standardising fresh). Mutually exclusive with
reuse_dataset_id/register_as.
Registration gate. A successful ingest of a NEW (never-before-
registered) dataset opens a kind="registration", gate_id=
"registration-<basename>" gate with options ["register", "ignore"]
and sets dataset_decision="registration_pending". Every downstream
pipeline tool then refuses via _require_resolved_dataset_decision
until authorise_gate closes the gate ("register" persists the
standardised products into the store; "ignore" records ephemeral).
Mechanical, not advisory — cannot be skipped.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
session_dir
|
str
|
Session directory (session_context.json must exist). |
required |
survey
|
str | None
|
Catalogue-type profile name/alias supplying column conventions and defaults. |
None
|
z_edges
|
list[float] | None
|
Redshift bin edges for tomography; mutually exclusive with
|
None
|
ra_col
|
str | None
|
Right-ascension column override. |
None
|
dec_col
|
str | None
|
Declination column override. |
None
|
z_col
|
str | None
|
Redshift column override. |
None
|
weight_col
|
str | None
|
Weight column override. |
None
|
e1_col
|
str | None
|
First shear-ellipticity column override (spin-2). |
None
|
e2_col
|
str | None
|
Second shear-ellipticity column override (spin-2). |
None
|
psf_e1_col
|
str | None
|
PSF first-ellipticity-component column; carried through
as |
None
|
psf_e2_col
|
str | None
|
PSF second-ellipticity-component column; carried
through as |
None
|
probe
|
str | None
|
|
None
|
sigma_e
|
float | None
|
Per-component shear shape-noise RMS override; else the measured Heymans+2012 estimate is used. |
None
|
m_values
|
list[float] | None
|
Per-bin multiplicative shear calibration; length must equal the number of bins. Divided out of the ellipticities at ingest. |
None
|
sigma_m
|
float | list[float] | None
|
Per-bin (or scalar) shear calibration uncertainty; feeds the covariance marginalisation term. |
None
|
subtract_cterm
|
bool | None
|
Subtract the weighted-mean additive c-term from
E1/E2 at ingest. |
None
|
selection
|
list[dict] | None
|
Declarative row-selection predicates,
|
None
|
keep_zero_weight
|
bool
|
Skip the implicit |
False
|
bin_col
|
str | None
|
Integer tomographic bin-ID column; mutually exclusive with
|
None
|
bin_values
|
list[int] | None
|
Explicit bin IDs to use with |
None
|
mask_kind
|
str | None
|
|
None
|
mask_from_weights
|
bool
|
Build the shear mask as per-pixel Σw instead of a supplied footprint file. |
False
|
completeness_path
|
str | None
|
HEALPix completeness map path (counts/ completeness inside the footprint, floor 1e-3). |
None
|
catalogue_path
|
str | None
|
Explicit catalogue path, required when |
None
|
append
|
bool
|
Add this catalogue as an additional probe to an already-ingested session (multi-probe / 3×2pt). |
False
|
mask_path
|
str | None
|
Explicit survey mask path, overriding discovery. |
None
|
nz_path
|
str | None
|
Explicit n(z) file path (single-bin), overriding discovery. |
None
|
nz_paths
|
list[str] | None
|
Explicit per-bin n(z) file paths, one per tomographic bin. |
None
|
theory_path
|
str | None
|
Explicit theory Cℓ file path (single-bin). |
None
|
theory_paths
|
list[str] | None
|
Explicit per-bin theory Cℓ file paths, one per tomographic bin — never auto-matched. |
None
|
theory_sacc_path
|
str | None
|
Path to a SACC file (.fits or .sacc/.h5) to extract n(z) + all-pairs theory Cℓ from, as an alternative to nz_path/nz_paths/theory_path/theory_paths — mutually exclusive with all four (needs_input if combined with any). |
None
|
reuse_dataset_id
|
str | None
|
Registry dataset id to reuse (from a prior
|
None
|
dataset_tag
|
str | None
|
A registered dataset to reuse, named EITHER as a full
|
None
|
register_as
|
str | None
|
|
None
|
ignore_registry
|
bool
|
Standardise fresh, skipping the registry entirely.
The former |
False
|
jobspec
|
dict | None
|
Optional |
None
|
Returns:
| Type | Description |
|---|---|
dict
|
An async receipt ``{job_id, status, mode: "async", run_dir, |
dict
|
session_dir, next: "check_job", ...}`` — never the manifest itself. |
dict
|
Poll |
dict
|
|
dict
|
a |
dict
|
file, or registry decision needs resolving before ingest can |
dict
|
proceed. |
Source code in cosmotron_mcp/server.py
2874 2875 2876 2877 2878 2879 2880 2881 2882 2883 2884 2885 2886 2887 2888 2889 2890 2891 2892 2893 2894 2895 2896 2897 2898 2899 2900 2901 2902 2903 2904 2905 2906 2907 2908 2909 2910 2911 2912 2913 2914 2915 2916 2917 2918 2919 2920 2921 2922 2923 2924 2925 2926 2927 2928 2929 2930 2931 2932 2933 2934 2935 2936 2937 2938 2939 2940 2941 2942 2943 2944 2945 2946 2947 2948 2949 2950 2951 2952 2953 2954 2955 2956 2957 2958 2959 2960 2961 2962 2963 2964 2965 2966 2967 2968 2969 2970 2971 2972 2973 2974 2975 2976 2977 2978 2979 2980 2981 2982 2983 2984 2985 2986 2987 2988 2989 2990 2991 2992 2993 2994 2995 2996 2997 2998 2999 3000 3001 3002 3003 3004 3005 3006 3007 3008 3009 3010 3011 3012 3013 3014 3015 3016 3017 3018 3019 3020 3021 3022 3023 3024 3025 3026 3027 3028 3029 3030 3031 3032 3033 3034 3035 3036 3037 3038 3039 3040 3041 3042 3043 3044 3045 3046 3047 3048 3049 3050 3051 3052 3053 3054 3055 3056 3057 3058 3059 3060 3061 3062 3063 3064 3065 3066 3067 3068 3069 3070 3071 3072 3073 3074 3075 3076 3077 3078 3079 3080 3081 3082 3083 3084 3085 3086 3087 3088 3089 3090 3091 3092 3093 3094 3095 3096 3097 3098 3099 3100 3101 3102 3103 3104 3105 3106 3107 3108 3109 3110 3111 3112 3113 3114 3115 3116 3117 3118 3119 3120 3121 3122 3123 3124 3125 3126 3127 3128 3129 3130 3131 3132 3133 3134 3135 3136 3137 3138 3139 3140 3141 3142 3143 3144 3145 3146 3147 3148 3149 3150 3151 3152 3153 3154 3155 3156 3157 3158 3159 3160 3161 3162 3163 3164 3165 3166 3167 3168 3169 3170 3171 3172 3173 3174 3175 3176 3177 3178 3179 3180 3181 3182 3183 3184 3185 3186 3187 3188 | |
ingest_template_maps ¶
ingest_template_maps(session_dir: str, templates: dict[str, str], kinds: dict[str, str] | None = None) -> dict
Ingest named systematics template maps (stellar density, seeing, dust, depth, PSF-ellipticity, ...) into the session manifest for compute_systematics_cls_from_session / deproject_templates=.
templates: {name: path} of HEALPix FITS maps (absolute, relative to
data_folder, or relative to the repo root). kinds: optional per-name
"density" (ud_grade power=-2, Cosmotheka convention — preserves the total
under resampling) or "dimensionless" (default — plain averaging). An
unknown kind returns needs_input.
Discovery is explicit-only — there is no filename auto-discovery for survey-property maps (no reliable naming convention exists); name them from DATA_DESCRIPTION.md.
Each map is reconciled to the session nside, written to
results/systematics/template_{name}.fits, recorded in
manifest["systematics_templates"][name] = {path, kind, native_nside,
source_path}, and registered as a systematics_template artefact with
provenance. Re-ingesting an existing name overwrites it.
Requires manifest.json to already exist (run ingest_to_session first) — else returns needs_input. All-zero map after nside reconciliation also returns needs_input.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
session_dir
|
str
|
Session directory (manifest.json must already exist). |
required |
templates
|
dict[str, str]
|
|
required |
kinds
|
dict[str, str] | None
|
Optional per-name |
None
|
Returns:
| Type | Description |
|---|---|
dict
|
The updated manifest's |
dict
|
|
dict
|
unknown, or a map is all-zero after nside reconciliation. |
Source code in cosmotron_mcp/server.py
register_dataset ¶
register_dataset(data_dir: str, name: str, version: str = 'v1', survey: str | None = None, z_edges: list[float] | None = None, probe: str | None = None, sigma_e: float | None = None, catalogue_path: str | None = None, ra_col: str | None = None, dec_col: str | None = None, z_col: str | None = None, weight_col: str | None = None, e1_col: str | None = None, e2_col: str | None = None, psf_e1_col: str | None = None, psf_e2_col: str | None = None, bin_col: str | None = None, bin_values: list[int] | None = None, selection: list[dict] | None = None, keep_zero_weight: bool = False, mask_path: str | None = None, mask_from_weights: bool = False, mask_kind: str | None = None, completeness_path: str | None = None, nz_path: str | None = None, nz_paths: list[str] | None = None, theory_path: str | None = None, theory_paths: list[str] | None = None, subtract_cterm: bool | None = None, m_values: list[float] | None = None, sigma_m: float | list[float] | None = None, nside: int | None = None, full_sky: bool | None = None) -> dict
Standardise a data directory into the persistent registry and record it.
The directory is ONE dataset (catalogue[s] + mask[s] + n(z) +
DATA_DESCRIPTION.md). Standardised products are copied into the persistent
store ($COSMOTRON_REGISTRY, default ./database) so they survive a
deleted workspace and can be reused across sessions via
ingest_to_session(reuse_dataset_id=...). Register variants (e.g. GLASS
single-bin vs tomographic vs cosmic-shear) under DISTINCT name/version —
they are different datasets. Pass probe="galaxy_shear" for a shear dataset:
the probe becomes part of the dataset id so clustering vs shear from the same
directory never collide. A multi-catalogue 3×2pt directory is registered ONE
PROBE AT A TIME — pass catalogue_path= (the catalogue for this probe) and a
distinct probe=.
All standardisation knobs mirror ingest_to_session and are forwarded into
the internal staging run (tomography, cuts, mask/shear/calibration, ancillaries).
nside/full_sky override the staging config — set nside to your intended
analysis resolution for a weight-mask (shear) dataset, since the stored Σw masks +
f_sky/n_eff are served verbatim on reuse.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
data_dir
|
str
|
Directory holding the catalogue(s), mask(s), n(z), and
|
required |
name
|
str
|
Dataset name. |
required |
version
|
str
|
Dataset version tag. |
'v1'
|
survey
|
str | None
|
Catalogue-type profile name/alias to apply. |
None
|
z_edges
|
list[float] | None
|
Redshift bin edges for tomography. |
None
|
probe
|
str | None
|
|
None
|
sigma_e
|
float | None
|
Per-component shear shape-noise RMS override (shear only). |
None
|
catalogue_path
|
str | None
|
Which catalogue in |
None
|
ra_col
|
str | None
|
Column override. |
None
|
dec_col
|
str | None
|
Column override. |
None
|
z_col
|
str | None
|
Column override. |
None
|
weight_col
|
str | None
|
Column override. |
None
|
e1_col
|
str | None
|
Shear-ellipticity column override. |
None
|
e2_col
|
str | None
|
Shear-ellipticity column override. |
None
|
psf_e1_col
|
str | None
|
PSF-ellipticity column override. |
None
|
psf_e2_col
|
str | None
|
PSF-ellipticity column override. |
None
|
bin_col
|
str | None
|
Integer-bin-ID tomography column. |
None
|
bin_values
|
list[int] | None
|
Integer-bin-ID tomography values. |
None
|
selection
|
list[dict] | None
|
Declarative row-selection predicates. |
None
|
keep_zero_weight
|
bool
|
Skip the implicit |
False
|
mask_path
|
str | None
|
Mask construction — explicit mask path. |
None
|
mask_from_weights
|
bool
|
Mask construction — build from catalogue weights. |
False
|
mask_kind
|
str | None
|
Mask construction — mask kind. |
None
|
completeness_path
|
str | None
|
HEALPix completeness map (clustering). |
None
|
nz_path
|
str | None
|
Ancillary-file override — n(z) file (single bin). |
None
|
nz_paths
|
list[str] | None
|
Ancillary-file override — n(z) files (per bin). |
None
|
theory_path
|
str | None
|
Ancillary-file override — theory Cl file (single bin). |
None
|
theory_paths
|
list[str] | None
|
Ancillary-file override — theory Cl files (per bin). |
None
|
subtract_cterm
|
bool | None
|
Phase-E shear calibration — subtract additive c-term. |
None
|
m_values
|
list[float] | None
|
Phase-E shear calibration — multiplicative bias values. |
None
|
sigma_m
|
float | list[float] | None
|
Phase-E shear calibration — multiplicative bias uncertainty. |
None
|
nside
|
int | None
|
Staging pipeline_config override. |
None
|
full_sky
|
bool | None
|
Staging pipeline_config override. |
None
|
Returns:
| Type | Description |
|---|---|
dict
|
|
dict
|
payload if a column or n(z) file can't be resolved. |
Source code in cosmotron_mcp/server.py
3244 3245 3246 3247 3248 3249 3250 3251 3252 3253 3254 3255 3256 3257 3258 3259 3260 3261 3262 3263 3264 3265 3266 3267 3268 3269 3270 3271 3272 3273 3274 3275 3276 3277 3278 3279 3280 3281 3282 3283 3284 3285 3286 3287 3288 3289 3290 3291 3292 3293 3294 3295 3296 3297 3298 3299 3300 3301 3302 3303 3304 3305 3306 3307 3308 3309 3310 3311 3312 3313 3314 3315 3316 3317 3318 3319 3320 3321 3322 3323 3324 3325 3326 3327 3328 3329 3330 3331 3332 3333 3334 3335 3336 3337 3338 3339 3340 3341 3342 3343 3344 3345 3346 3347 3348 3349 3350 3351 3352 3353 3354 | |
register_session ¶
register_session(session_dir: str, name: str, version: str = 'v1', probe: str | None = None, catalogue_path: str | None = None, prune_session_copy: bool = True) -> dict
Persist a session's already-standardised products into the registry.
The post-ingest companion to the registration-<basename> file gate
ingest_to_session opens on a first successful run: adopts the products
the session just produced (no re-standardising), copies them into the
persistent store ($COSMOTRON_REGISTRY, default ./database), and records
the dataset. Use a DISTINCT name/version per variant; probe defaults
from the manifest bins (clustering vs shear get distinct dataset ids).
Historical user_reply string authorisation is gone (2026-08-03
collapse). Authorisation now lives in the file-backed registration
gate: closing it via authorise_gate(session_dir,
"registration-<basename>", "register") triggers the gate side-effect
which invokes this function's core with the deterministic
machine-suggested name. Direct programmatic/CLI callers may still call
this tool without any authorisation dance — the guard is gone.
A multi-catalogue (3×2pt) session must be registered ONCE PER CATALOGUE —
pass catalogue_path=<basename of one manifest.catalogues[i].raw_path>
to scope this call to just that catalogue's bins. Omitting it on a session
with more than one ingested catalogue returns an error naming them,
rather than registering a mixed dataset under one probe id.
By default (prune_session_copy=True) the session's OWN copy of every
product just registered is deleted once the store copy is verified and
manifest.json is rewritten to point at it — a session registering the
same multi-GB catalogue it just standardised no longer keeps two copies
forever. Pass prune_session_copy=False to keep the old behaviour
(session copy retained alongside the store's).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
session_dir
|
str
|
Session directory whose products to persist. |
required |
name
|
str
|
Dataset name. |
required |
version
|
str
|
Dataset version tag. |
'v1'
|
probe
|
str | None
|
Override the probe recorded for the dataset id; defaults
from the manifest bins (or the matched catalogue's, when
|
None
|
catalogue_path
|
str | None
|
Which ingested catalogue this registration covers
(required when the session has more than one). A full path or
just the basename, matched against |
None
|
prune_session_copy
|
bool
|
Delete the session's own copy of each registered
product after verifying the store copy and rewriting
|
True
|
Returns:
| Type | Description |
|---|---|
dict
|
|
dict
|
or |
dict
|
|
Source code in cosmotron_mcp/server.py
3357 3358 3359 3360 3361 3362 3363 3364 3365 3366 3367 3368 3369 3370 3371 3372 3373 3374 3375 3376 3377 3378 3379 3380 3381 3382 3383 3384 3385 3386 3387 3388 3389 3390 3391 3392 3393 3394 3395 3396 3397 3398 3399 3400 3401 3402 3403 3404 3405 3406 3407 3408 3409 3410 3411 3412 3413 3414 3415 3416 3417 3418 3419 | |