NERSC & same-cluster agents

Two variations on the base remote-execution model, each solving a different constraint: NERSC's 2FA login (auth, not architecture), and running the agents themselves on a cluster instead of the Spark (a different transport, not new tool calls).

NERSC sshproxy

NERSC requires multi-factor authentication (password + a one-time code) for every login — including SSH. cosmoTRON never automates that step; it can't, by design (no interactive channel inside a tool call), and doing so would be exactly the kind of MFA circumvention NERSC's Appropriate Use Policy prohibits. Instead it uses NERSC's own sanctioned tool for this pattern: sshproxy mints a short-lived SSH certificate (24h by default) after one interactive MFA login; every SSH call afterward, for the life of that certificate, is non-interactive.

You run this in a terminal — never the agent:

sshproxy.sh -u <your NERSC username>

This writes a key + certificate pair (e.g. ~/.ssh/nersc + ~/.ssh/nersc-cert.pub). Configure the site with auth: {type: sshproxy}:

sites:
  nersc:
    host: perlmutter.nersc.gov
    user: <your NERSC username>
    auth:
      type: sshproxy
      key_path: ~/.ssh/nersc
      proxy_hint: "sshproxy.sh -u <your NERSC username>"   # shown verbatim if the cert needs refreshing
    remote_root: /global/homes/<u>/<your NERSC username>/cosmotron
    scratch_root: $SCRATCH        # resolve via `add-site`'s probe, don't hardcode a literal path
    env_setup:
      - module load conda
      - conda activate cosmotron
    slurm:
      # Perlmutter selects via QOS + a hardware constraint, not `-p`.
      sbatch_extra:
        - "#SBATCH -C cpu"
        - "#SBATCH -A <your NERSC account, e.g. m1234>"
    queues:
      default:
        qos: regular
      debug:
        qos: debug

Every tool call and check_job works exactly as on any other site. The one difference: when the certificate expires (~once a day), the next call that needs it fails cleanly instead of hanging or corrupting job state:

• At submit: the job fails immediately with error: "auth_required: sshproxy certificate expired/expiring: ... — sshproxy.sh -u <you>".
• At poll: status is left unchanged (the job is very likely still running fine on the cluster — only checking on it failed), and poll_error/stale carry the same auth_required: ... message.

Either way: run sshproxy.sh -u <you> again in a terminal, then re-submit or re-poll — check_job resumes cleanly, no corrupted state, no lost job.

cosmotron-remote doctor nersc re-verifies the whole contract (cert validity included) any time.

Same-cluster execution

The scenario above assumes the agent runs somewhere else (the Spark) and reaches the cluster over SSH. If instead the agents themselves run on the cluster — e.g. an interactive GPU-node allocation, with the GPU serving the local Ollama model — dispatching the CPU work (NaMaster/TJPCov/CosmoSIS) as separate Slurm jobs needs neither SSH nor rsync: the agent and the CPU job already share the same filesystem.

Configure this as transport: {type: local}:

sites:
  sunrise-local:
    transport:
      type: local
    shared_filesystem: true      # required — the whole point is a shared tree
    auth:
      type: none                 # nothing to authenticate on the same machine
    # host/remote_root default to localhost / the process cwd — leave unset
    # unless your session workspace lives somewhere other than cwd.
    env_setup:
      - module load conda
      - conda activate cosmotron
    slurm:
      poll_min_interval_s: 5     # local squeue/sacct is cheap — shorter is fine
    queues:
      default:
        partition: Solar         # the CPU partition (different nodes than the GPU allocation)
      bigmem:
        partition: CoPS

Tool calls are identical to the SSH case — same jobspec={"backend": "slurm", "site": "sunrise-local"} shape on the same tools:

compute_all_cls_from_session(session_dir, jobspec={"backend": "slurm", "site": "sunrise-local"})

Under the hood: sbatch/squeue/sacct run as direct subprocess calls (no SSH handshake), and staging/pull-back are no-ops — the CPU job writes its results directly into the session tree the agent already sees, so there's nothing to copy and no artefact-registry merge to run.

A single sites.yaml can hold both an ssh entry (for driving this cluster from the Spark) and a local entry (for when the agent runs on the cluster itself) — pick whichever fits per call.

The one thing to verify first

Not every cluster allows a compute node to submit Slurm jobs — some restrict sbatch to login/submit nodes, in which case transport: local is not viable there at all (a different design — e.g. a submission proxy watching a spool directory — would be needed instead). Before relying on this, from inside your actual allocation:

sbatch --wrap='echo ok' && squeue -u "$USER"

cosmotron-remote doctor sunrise-local runs exactly this check as its first step, so it's easy to re-verify — and cosmotron-remote add-site, run from inside an active allocation ($SLURM_JOB_ID set), detects this and offers to set up a local entry automatically, running the same probe as the proof step instead of an SSH hello-job.