Simulation archive design
This document describes the on-disk format and core abstractions for a RIFT simulation archive. The goals are:
Self-contained. An archive is a directory you can copy, ship, and read months later without access to the originating source tree.
Cheap to query. Listing what’s in the archive, what’s complete, and basic per-sim summaries should not require opening every output file. There is a single index.jsonl for that.
Two-tier queueing. The queue used to request simulations is independent of the queue used to run them. A typical OSG setup is “Condor DAG on the submit host requests sims; a Slurm cluster runs them.” A test setup might be “local request, local run.” The archive does not care.
Frozen code. The user-supplied generator is captured at archive creation time and stored under code/. Workers exec the frozen copy. The archive is portable without the original repository.
The existing classes in BaseManager.py / CondorManager.py / SlurmManager.py evolve toward this design; the new schemas live alongside them in database.py (stub, this commit) and can be adopted incrementally.
On-disk layout
<base_location>/
manifest.json # archive-level metadata (schema, name, code refs, queue config)
code/ # frozen interfaces — required to *run* workers, not to read summaries
generator.py # entrypoint: generator.run(params, sim_dir) -> writes output
summarizer.py # OPTIONAL: summarize(sim_dir) -> dict (used post-completion)
requirements.txt # OPTIONAL: package versions known at archive creation
README # human note: what this archive is, how it was built
index.jsonl # one JSON object per line, one line per sim — canonical and cheap
sims/
<sim_name>/ # one subdirectory per sim (sim_name is the archive's internal id)
params.json # canonical input parameters
status.json # FSM state + per-queue tracking + history
summary.json # summarizer output (written when status -> complete)
output.<ext> # raw output written by the worker; format up to the generator
logs/
worker.stdout
worker.stderr
run.log # generator-internal logging if any
request_queue/ # workspace for the queue that *requests* sims
dags/ # condor DAGs (when request_queue.kind == 'condor')
*.sub
run_queue/ # workspace for the queue that *runs* sims
workers/ # generated per-sim worker scripts
submit_files/
state.json # cached cluster ids, last-poll timestamps, etc.
Anything that’s “configuration we may want to overwrite atomically” lives at the archive root; anything per-sim lives under sims/<sim_name>/. Anything queue-system-specific lives under request_queue/ or run_queue/ — the archive itself stays neutral.
Manifest
manifest.json at the archive root, written once at archive creation and updated only when archive-level configuration changes (e.g. a queue swap). Schema:
{
"schema_version": 1,
"name": "GW190521_grid_v3",
"created_at": "2026-04-26T15:00:00Z",
"rift_version": "0.0.18.0",
"code": {
"generator": "code/generator.py",
"generator_entrypoint": "generator:run", // module:callable
"summarizer": "code/summarizer.py",
"summarizer_entrypoint":"summarizer:summarize",
"requirements": "code/requirements.txt"
},
"request_queue": { "kind": "condor", "extra": { } },
"run_queue": { "kind": "slurm", "extra": { "partition": "compute" } },
"params_schema": { "type": "scalar", "dtype": "float" }, // hint only
"summary_schema": { "fields": { "value": "float", "wall_time_s": "float" } }
}
The split between code.generator_entrypoint (“module:callable”) matches the setuptools convention and works equally well for a single-file generator and a multi-file package frozen under code/.
Index
index.jsonl is one record per sim. It is the fast, canonical view: listing the archive, filtering by status, computing aggregate summaries all use this file and never touch sims/<sim_name>/ directly. The full per-sim story lives in status.json + summary.json; the index is a projection.
{"name":"1","status":"complete","params":0.5,"summary":{"value":0.7071}}
{"name":"2","status":"running", "params":1.5,"summary":null}
{"name":"3","status":"ready", "params":2.5,"summary":null}
Append-only with periodic compaction. A run that registers many sims appends one line per sim under a single index lock; status updates compact (rewrite) the file under the same lock. Compaction is cheap because the file is JSON-lines and rarely larger than a few MB even for 100k sims.
Per-sim status
sims/<sim_name>/status.json. Contains the FSM state plus per-queue tracking and a transition log. The FSM is the same as today’s QUEUE_STATES: ready -> submit_ready -> running -> complete | stuck.
{
"name": "1",
"params": 0.5,
"status": "running",
"history": [
{ "status": "ready", "ts": "2026-04-26T15:00:00Z" },
{ "status": "submit_ready", "ts": "2026-04-26T15:00:05Z" },
{ "status": "running", "ts": "2026-04-26T15:00:30Z" }
],
"request_queue": { "kind": "condor", "cluster_id": 12345, "proc_id": 0, "dag_node": "sim_1" },
"run_queue": { "kind": "slurm", "job_id": 98765, "host": "compute-7" },
"output": { "path": "sims/1/output.npy", "size": null, "sha256": null },
"started_at": "2026-04-26T15:00:30Z",
"completed_at": null
}
The request_queue and run_queue blocks are independent — the same sim can be tracked in two systems at once. kind matches the value in the manifest’s queue spec.
Frozen code
At archive creation, the user supplies one of:
A Python callable. The archive captures inspect.getsource(fn) and writes code/generator.py with a run(params, sim_dir) entrypoint. Closure captures and unusual import scopes are not supported; the function must be self-contained.
A path to a `.py` file. Copied verbatim. The user is responsible for naming the entrypoint per the manifest.
A spec dict. { “module_path”: “…”, “entrypoint”: “mod:fn”, “extra_files”: [“aux.py”, “data.csv”] }. All listed files are copied into code/; module_path becomes code/generator.py (or whatever the spec names).
A worker invocation is then approximately:
python -c "import sys; sys.path.insert(0, 'code'); \
from generator import run; run(params=..., sim_dir='sims/1')"
— so the archive is runnable without RIFT being installed, given a Python interpreter and whatever the generator’s own deps are. The optional code/requirements.txt documents those.
Reading summaries does not need code/ at all — summary.json is plain JSON. Reading outputs may need code/ if the format is opaque (e.g. pickled objects), so a generator that writes .npy/.h5/ JSON is preferred.
Similarity / dedup
Users come back. They request “the same” sim again. Sometimes “the same” means bit-identical params; sometimes it means “the same up to floating-point noise”; sometimes it means “the same up to a domain- specific tolerance” (mass within 1e-6 solar masses, spin within 1e-3, etc.). The archive should not redo work in any of these cases.
Two user-supplied callables, both optional, both frozen alongside the generator under code/:
def same_q(params_a, params_b) -> bool:
"""Reflexive, symmetric, transitive equality on parameters.
Defaults to exact equality (params_a == params_b)."""
def lookup_key(params) -> "JSON-serializable":
"""Maps params to a coarse bucket for fast dedup.
Must be consistent with same_q: same_q(a, b) == True implies
lookup_key(a) == lookup_key(b). Defaults to str(params)."""
`lookup_key` must be JSON-serializable, not merely hashable. The bucket key is a persisted value — written to index.jsonl, with the dedup buckets rebuilt from that file on every Archive construction. So the real requirement is that it survive the archive’s JSON normalization unchanged. That is both stricter and weaker than hashability: a frozenset is hashable but cannot be persisted, while a plain list can be persisted but is not hashable.
The engine normalizes on the way in and canonicalizes the same way on the way out, so these are handled rather than silently breaking dedup:
tuples — JSON has no tuple type, so a tuple key returns as a list; both canonicalize to the same form.
dict keys — JSON coerces them to strings, and not via str(): True/False/None become “true”/”false”/”null”, float infinities “Infinity”. Keys colliding once coerced ({True: ‘a’, “true”: ‘b’}) collapse last-wins, consistently on both sides.
A key JSON cannot represent at all raises at register() with a message naming this contract, rather than surfacing as a sorted() TypeError from inside the index write.
The safest choice is a string: it round-trips to itself, sorts, and cannot collide by coercion. RIFT’s own gw_pe_synthetic returns a tuple, which the normalization handles.
These together give O(1) average lookup: bucket by lookup_key, then run same_q against only the (typically zero or one) entries in that bucket. The archive keeps an in-memory {lookup_key: [sim_name, …]} map, rebuilt from index.jsonl on rehydration.
Each row in index.jsonl carries its lookup_key so the dedup index can be rebuilt without ever calling the user’s code (useful when inspecting an archive on a machine that doesn’t have the generator’s runtime deps).
Archive.register(params) becomes idempotent under same_q: it returns the existing sim’s name when a match is found, registers a fresh sim otherwise. Archive.find_existing(params) -> Optional[str] is exposed for callers that want to look without registering.
The manifest grows two optional code.* entries:
"code": {
"generator": "code/generator.py", "generator_entrypoint": "generator:run",
"same_q": "code/same_q.py", "same_q_entrypoint": "same_q:same_q",
"lookup_key":"code/lookup_key.py", "lookup_key_entrypoint":"lookup_key:lookup_key"
}
Typical usage, gravitational-wave style:
def lookup_key(p):
# Round into a coarse grid; tolerance below same_q's threshold.
return (round(p["mtotal"], 4),
round(p["q"], 4),
round(p["chi1z"], 3),
round(p["chi2z"], 3))
def same_q(a, b):
return (abs(a["mtotal"] - b["mtotal"]) < 1e-3 and
abs(a["q"] - b["q"]) < 1e-3 and
abs(a["chi1z"] - b["chi1z"]) < 1e-2 and
abs(a["chi2z"] - b["chi2z"]) < 1e-2)
Refinement (iterative improvement)
A simulation is rarely “done” in a binary sense. For GW parameter inference the lnL at a single (mc, eta) is computed via a Monte Carlo integral; the answer can always be made tighter by running more samples. A user might come back and say “this point doesn’t have the precision I need — improve it.”
The archive models this as a level counter per sim:
Each sim has a target_level (what’s been requested) and a current_level (what’s been computed). They’re both integers, starting at 0.
A sim’s output is a sequence of per-level files: sims/<name>/level_1.json, level_2.json, … Each level is self-contained (or builds on its predecessors — that’s the generator’s choice).
The summarizer can fold all completed levels into a single rolling summary (e.g. weighted average of lnL across levels with errors that shrink with sample count).
The FSM extends:
ready -> never run, target_level >= 1, current_level == 0
submit_ready -> queued for the run system (next level to compute is current_level + 1)
running -> a level is being computed
complete -> current_level == target_level
refine_ready -> output exists (current_level >= 1) but target_level > current_level
stuck -> failure
Archive.register(params, target_level=N) is idempotent on bump: - doesn’t exist: register fresh with target_level=N - exists at current_level >= N: return name, no work - exists at current_level < N: bump target_level to max(existing, N),
transition to refine_ready, return name
Archive.refine(name, target_level=N) is the explicit form for callers that already know the sim_name.
The generator signature is (params, sim_dir, level, prev_levels): - level: integer, the level being computed right now - prev_levels: list of paths to outputs from levels 1..level-1, so
the generator can read prior work and extend it (e.g. a Monte Carlo generator can accumulate samples from earlier levels)
Each register_simulation invocation in the run queue may compute one or many levels. The simplest run queue computes one level per submit and re-submits the sim for the next level. A smarter run queue might batch level-1-through-N for a sim into a single condor job to amortize startup cost.
Status records grow a levels[] list:
{
"name": "1",
"params": {"mc": 25.0, "eta": 0.22},
"status": "complete",
"target_level": 4,
"current_level": 4,
"levels": [
{ "level": 1, "output_path": "sims/1/level_1.json", "completed_at": "..." },
{ "level": 2, "output_path": "sims/1/level_2.json", "completed_at": "..." },
{ "level": 3, "output_path": "sims/1/level_3.json", "completed_at": "..." },
{ "level": 4, "output_path": "sims/1/level_4.json", "completed_at": "..." }
],
"summary": { "lnL_mean": -123.4, "lnL_stderr": 0.05, "n_samples": 4000 }
}
Two-tier queueing
Two interfaces, composed by the archive. The same archive can mix and match: (LocalRequestQueue, LocalRunQueue) for tests, (CondorRequestQueue, SlurmRunQueue) for an OSG-fed cluster, (CondorRequestQueue, CondorRunQueue) if the request and run queues are the same condor pool, etc.
class RequestQueue:
"""Decides which sims should next be sent to the run queue, and
tracks their state in the request system. The request queue does
NOT execute sims; it forwards them to the run queue."""
kind: str
def submit_pending(self, archive) -> list[str]: ... # 'ready' -> 'submit_ready'/'running'
def poll(self, archive) -> dict[str, str]: ... # name -> request-side status
class RunQueue:
"""Actually runs sims (writes the output file)."""
kind: str
def build_worker(self, archive, sim_name) -> str: ... # path to per-sim worker script
def submit(self, archive, sim_names) -> list[tuple[str, str]]: ... # (name, run_queue_id)
def poll(self, archive, sim_names) -> dict[str, str]: ... # name -> run-side status
In a production OSG setup the implementations look like: - CondorRequestQueue.submit_pending: builds a DAG over all ready
sims, calls condor_submit_dag, stamps submit_ready.
CondorRequestQueue.poll: queries the schedd via _htcondor_module (htcondor or htcondor2), updates request_queue.cluster_id etc.
SlurmRunQueue.build_worker: writes run_queue/workers/<name>.sh that loads code/generator.py and calls run(params, sim_dir).
SlurmRunQueue.submit: sbatch per worker, capture job ids.
SlurmRunQueue.poll: squeue –json parse → status.
The archive composes them and owns the FSM transitions: queue implementations report state, the archive applies it.
Topology: dual condor
A common production setup at OSG is two condor pools: a “request” pool on a dedicated submit host that runs the long-lived planner DAG, and a “run” pool (typically OSG itself, or a remote site) that runs the actual workers. The archive doesn’t care that both happen to be condor — they’re configured as two separate (RequestQueue, RunQueue) instances. Concretely:
+-------------------------------+ +---------------------------------+
| request pool (e.g. CIT submit)| | run pool (e.g. OSG / remote site)|
| | | |
| meta-DAG | | sub-DAG (one per scout pass) |
| ┌─────────────┐ | | ┌─────────────┐ |
| │ scout │ | | │ sim_a lvl 1 │ |
| │ (read │ | | │ sim_b lvl 1 │ |
| │ archive, │ | | │ sim_c lvl 2 │ (refine) |
| │ pick work) │ | | │ sim_a lvl 2 │ (refine) |
| └─────┬───────┘ | | └─────┬───────┘ |
| │ | | │ |
| ┌─────▼───────┐ -- submit ──┼────────┼──> writes outputs to shared FS |
| │ submit_run │ via | | │ |
| │ (condor_ │ condor_ | | ┌─────▼───────┐ |
| │ submit_dag │ submit_dag | | │ workers │ |
| │ -name run) │ -name run | | │ (run gen) │ |
| └─────┬───────┘ | | └─────────────┘ |
| │ | | |
| ┌─────▼───────┐ | +---------------------------------+
| │ wait/poll │ <-- htcondor.Schedd(run_pool).query(...)
| │ collect │
| └─────┬───────┘
| │
| ┌─────▼───────┐
| │ iterate │ -- back to scout
| └─────────────┘
+-------------------------------+
shared filesystem (the archive)
<base>/manifest.json, index.jsonl, sims/...
The shared filesystem typically only spans the submit nodes of each pool (and is most often the same machine for both pools, or two machines NFS-cross-mounted). Execute hosts on the run pool — OSG machines, leased EC2 nodes, etc. — generally have no view of the archive directory. The framework’s job is to ferry exactly the files each (sim, level) worker needs onto the execute host, run there, and ferry the level output back.
Topology: SLURM
The SLURM equivalent of DualCondorRunQueue is SlurmRunQueue (paired with SlurmRequestQueue). Two important differences from condor:
Shared filesystem at compute nodes is assumed. SLURM has no built-in transfer_input_files/transfer_output_files analog. The framework treats the archive directory as visible from the worker (the standard HPC model with NFS-mounted home/scratch). Sites without that property should arrange a Singularity container with bind-mounts (singularity exec -B <archive>:<archive> … python3 …) in the queue’s prelude snippet, or use sbcast to broadcast a manifest to /tmp on the workers.
Inter-job dependencies use `–dependency=afterok:<jobid>` rather than DAGMan’s PARENT/CHILD edges. SlurmRunQueue.submit chains each sim’s level submissions; level N’s sbatch waits for level N-1 to exit cleanly.
The per-(sim, level) submit script is a normal #!/bin/bash script with #SBATCH directives at the top, a prelude shell snippet (e.g. module load python/3.11, virtualenv activate), cd <sim_dir>, and a final exec python3 <archive>/run_queue/workers/bootstrap.py …. The bootstrap is the same one condor uses; the only adaptation is the –code-dir argument (passed as the absolute path to <archive>/code because the worker’s cwd is <sim_dir>, not a flattened sandbox).
OOM auto-recovery is not built into the SLURM submit description. SLURM’s standard idiom is –requeue plus operator-driven scontrol update jobid=… ReqMem=…; the archive’s Archive.unstick(name, bump_memory=True) covers this from the framework side. For sites that want fully automatic retry, wrap the sbatch in a re-submit harness in the queue’s prelude.
Configuration in the manifest’s run_queue.extra:
partition: compute # required
time_limit: "02:00:00" # default 01:00:00
nodes: 1
ntasks: 1
cpus_per_task: 1
request_memory: 4096 # MB
account: ligo-research # or env SLURM_ACCOUNT
qos: normal # or env SLURM_QOS
prelude: |
module load python/3.11
source /home/me/rift-env/bin/activate
partition_extra: # extra `--key=value` directives
gres: gpu:1
exclusive: ""
Per-sim overrides supported via Archive.set_resources(name, …): request_memory, time_limit, partition, plus extra_sbatch_directives (a dict; merged into the queue’s partition_extra).
Configuration defaults
DualCondorRunQueue reads the following from the constructor first, then falls back to environment variables, then to hard-coded defaults:
The getenv default is an allowlist rather than True because getenv = True is blocked by many sites (CIT among them). The allowlist is the OSG-blessed convention from docs/source/osg.rst. Sites that permit blanket env forwarding can set RIFT_GETENV=True in the submit-side environment, pass getenv=’True’ to the DualCondorRunQueue constructor, or set run_queue.extra.getenv in the manifest.
The env-var fallback matches the legacy CondorManager behavior, so existing LIGO/IGWN submit hosts that already export LIGO_ACCOUNTING and LIGO_USER_NAME work without manifest changes. Manifest values in run_queue.extra override constructor defaults, which override env vars — manifest > constructor > env > hard-coded.
OSG site-selection knobs (+DESIRED_SITES, +UNDESIRED_SITES, Requirements, etc.) are passed via the constructor’s extra_condor_cmds dict (or run_queue.extra.extra_condor_cmds in the manifest). The bindings appear verbatim as additional key = value lines in every per-(sim, level) submit description.
Because those lines are emitted last, a key that the queue already writes would replace its line rather than extend it — and condor_submit reports success either way. transfer_input_files, transfer_output_files, transfer_output_remaps and periodic_release are therefore refused in extra_condor_cmds (case-insensitively; HTCondor command names are). Each has an append-only alternative:
universe is not refused. Setting container_image selects the container universe for you, but declaring universe yourself is legal and sometimes necessary (local, scheduler, grid).
Containers
container_image selects HTCondor’s container universe and supplies the image in one argument. Not because the two can disagree — measured against condor 25.13.1, vanilla + container_image, container + container_image, and declaring no universe at all produce byte-identical job ads, and the JDL says universe “can either be optionally set to container or not declared at all”. It is one argument because that is one fewer thing to state, not because stating it twice is dangerous:
DualCondorRunQueue(
container_image="osdf:///ospool/ap41/data/<user>/supernu-v2.sif",
when_to_transfer_output="ON_SUCCESS",
)
Both are keyword-only, and last in the signature: they arrived after the constructor’s positional sequence was already in use, and inserting them in the middle would have rebound every positional argument after them.
The reference is not resolved or fetched. An osdf:// or docker:// URL is not readable from the submit host, so requiring that would refuse the ordinary OSG case; only a value that could corrupt the submit file is rejected.
use_singularity / singularity_image remain, for sites that honour only the legacy +SingularityImage form. Setting both is refused: emitted together, which one takes effect is decided by the site.
when_to_transfer_output takes HTCondor’s three legal values — ON_EXIT (default), ON_EXIT_OR_EVICT, ON_SUCCESS. NEVER is not one of them: condor accepts it with rc=0 and silently materialises ON_EXIT, so a caller setting it to suppress transfer gets transfer.
ON_EXIT_OR_EVICT is refused by this queue, and the refusal explains why. HTCondor: “If a file listed in transfer_output_files does not exist at eviction time, the job will go on hold.” This queue always emits an explicit transfer_output_files led by level_<N>.json, which the bootstrap writes only after the generator returns — so every mid-run eviction would hold the job rather than reschedule it, which is worse than the ON_EXIT behaviour the setting gets reached for. It also spools the evicted sandbox to the access point’s SPOOL, a shared volume that is easy to fill. To preserve partial work, use HTCondor self-checkpointing (checkpoint_exit_code / transfer_checkpoint_files), which does not require the final outputs to exist.
The trap is specific to listing an output that only exists on success, not to ON_EXIT_OR_EVICT itself. A submitter whose transfer_output_files names only paths created early — a working directory, say — can use it safely. This queue cannot, because it always leads that list with the level marker. If that ever changes, this refusal should lift with it.
container_image, a non-default when_to_transfer_output, and the extra_transfer_* lists are all emitted by build_worker — which submit() bypasses when subdag_factory is set, because the sub-DAG writes its own submit descriptions. Combining any of them with a factory is therefore refused, in the constructor and again in submit() (both are plain attributes, so a constructor-only check is walked past by assigning either one afterwards). Put the setting in the DAG the factory generates. For ON_EXIT_OR_EVICT the refusal does double duty: the sub-DAG path never reaches the check in build_worker, so accepting it would route the one value this queue rejects around its own rejection.
extra_periodic_release takes a single-line ClassAd expression for sites whose pool holds jobs for reasons the queue does not model — an opportunistic pool produces transient holds a dedicated cluster never sees. While auto_release_on_oom is on, the term is scoped away from whatever codes oom_hold_codes names, so oom_max_retries remains a real cap and request_memory cannot be multiplied without bound; the term governs every other hold code. With the OOM policy off it governs all of them.
The OOM policy is site configuration
auto_release_on_oom releases a job held for running out of memory and raises its request. Three parts of that are properties of the site, not of HTCondor, and are arguments rather than constants:
34 is the unambiguous memory code. 26 is `SystemPolicy` — it means whatever the site’s SYSTEM_PERIODIC_HOLD expressions say it means. On the clusters this policy came from that is usually memory; on an OSG access point it may be an anti-thrash limiter whose precondition is a high NumJobStarts, in which case releasing it with a bigger memory request fights the pool’s own protection. Sub-codes exist for the finer case: every SYSTEM_PERIODIC_HOLD at a site reports one hold code, so only the sub-code separates “over memory” from “restarted too many times”.
The counter is site-dependent too. NumJobStarts counts execution attempts, so preemption spends the budget. NumHolds counts holds of every kind, including input-transfer failures that increment it while the job has never run. Neither is “the number of memory holds” everywhere.
These are two alternatives, not one configuration: an exclusion on a code oom_hold_codes does not list is refused, since it could not have had any effect. Either disown 26 entirely —
DualCondorRunQueue(
auto_release_on_oom=True,
oom_hold_codes=(34,), # 26 means something else here
oom_retry_counter="NumHolds",
)
— or keep it and carve out the sub-codes the limiter reports:
DualCondorRunQueue(
auto_release_on_oom=True,
oom_hold_codes=(34, 26),
oom_hold_subcode_exclusions={26: (100, 101)}, # 26, minus the limiter
oom_retry_counter="NumHolds",
)
Which site is which is not recorded here. That belongs in whatever inventory you already keep about your own infrastructure; a table of site facts in this file would be stale immediately and wrong for every site it did not name. condor_config_val -dump | grep SYSTEM_PERIODIC_HOLD on the access point is what answers it.
DualCondorRunQueue(
auto_release_on_oom=True,
extra_periodic_release="(HoldReasonCode =!= 1) && (NumJobStarts < 50)",
)
Hyperpipeline / glue.pipeline integration
Many existing RIFT workflows are built with glue.pipeline.CondorDAG (see RIFT/misc/dag_utils.py, the various bin/create_*_pipeline* scripts, and the hyperpipeline). The simulation archive must drop into that ecosystem without forcing a top-level rewrite: the user adds one node to their DAG that says “ensure simulation X is present at level N”, and downstream nodes simply add it as a parent.
The integration is two pieces:
`request_sim` CLI (RIFT.simulation_manager.cli.request_sim). A small executable that opens an archive, calls Archive.register, dispatches run-pool work via the configured request_queue if needed, and (in –ensure mode) blocks until the requested level is satisfied. Three modes: –ensure (the DAGMan-friendly default), –check (poll-once and exit), –submit-async (dispatch and return). Exit code drives DAGMan node success.
`RequestSimulationJob` (RIFT.simulation_manager.glue_compat). A glue.pipeline.CondorDAGJob subclass whose executable is the CLI above. Construct once per archive/profile; call make_node(params, target_level) to get a CondorDAGNode that the user attaches to their existing DAG. Per-node values arrive as standard glue macros ($(macro_params), $(macro_target_level)).
A consumer’s hyperpipeline-style flow looks like:
from glue import pipeline
from RIFT.simulation_manager.glue_compat import RequestSimulationJob
dag = pipeline.CondorDAG(log='hyperpipe.dag.log')
sim_job = RequestSimulationJob(
archive_path='/data/archives/lnL_grid',
accounting_group='ligo.dev.o4.cbc.pe.lalinferencerapid',
accounting_group_user='albert.einstein',
timeout=3600,
)
sim_job.set_sub_file('request_sim.sub')
sim_job.write_sub_file()
for params in points_of_interest:
sim_node = sim_job.make_node(params=params, target_level=4)
dag.add_node(sim_node)
downstream_node.add_parent(sim_node) # blocks until ready
dag.set_dag_file('hyperpipe')
dag.write_concrete_dag()
The CLI runs on the submit node (where the archive lives). The parent DAG can therefore live on a different pool from the run pool; the request-sim node only needs the archive’s submit-node filesystem. When the CLI dispatches further work via the configured request_queue, the file-transfer machinery from the previous section ferries the inputs out to execute hosts and the outputs back.
glue.pipeline is treated as an optional dependency of the simulation_manager package: importing glue_compat is safe without it (only the RequestSimulationJob constructor raises a clear error if glue is actually missing). The CLI itself has no glue dependency.
Lightweight example
examples/sim_database_hello.py registers three sims (params 0.5, 1.5, 2.5), uses LocalRequestQueue + LocalRunQueue (which run the generator inline, no schedd), inspects manifest.json and index.jsonl, and reloads the archive in a fresh Archive(…) call to confirm everything round-trips. This is the smallest end-to-end exercise of the design and should always pass without any cluster.