Containers and multi-architecture deployment

RIFT runs its compute jobs (ILE, CIP) inside a Singularity/Apptainer container on HTCondor pools such as the OSG. Historically the environment variable SINGULARITY_RIFT_IMAGE names a single image, and every job is pinned to it:

export SINGULARITY_RIFT_IMAGE=/cvmfs/singularity.opensciencegrid.org/.../rift:production

That still works exactly as before. This page documents two additions:

  • a container *family* — point SINGULARITY_RIFT_IMAGE at a YAML manifest describing several images that target different GPU compute capabilities, and let HTCondor pick the right one per matched machine; and

  • a multi-target build that produces such a family from one template.

For an operator workflow that inventories a target GPU pool and creates container-cache warmup jobs, see survey_scan.

Note

If SINGULARITY_RIFT_IMAGE is a plain .sif path or a single osdf:// URL, behavior is unchanged — the manifest machinery is never engaged. A manifest is recognized purely by its .yaml / .yml suffix.

Deploying a container family

Set SINGULARITY_RIFT_IMAGE to a manifest file instead of a single image:

export SINGULARITY_RIFT_IMAGE=`pwd`/rift_container_family.yaml

Everything else — util_RIFT_pseudo_pipe.py, --use-singularity, --use-osg — is identical. When the pipeline builds the ILE/CIP submit files it reads the manifest and emits an expression-valued container selection (see What the pipeline generates below).

Manifest format

version: 1

# Machine ClassAd attribute the selection expression tests.
# Default GPUs_Capability (see "GPU attribute names" below).
capability_attr: GPUs_Capability

# Catch-all image (innermost else of the selection); MUST be CPU-safe,
# since it is also used when no GPU capability is advertised.
fallback: default

containers:
  # Broadly-compatible image for older machines. On CVMFS: referenced in
  # place and lazy-fetched (only the selected image is ever pulled), never
  # transferred.
  - label: default
    image: /cvmfs/singularity.opensciencegrid.org/oshaughn/rift_container_default.sif
    cuda_capability_min: 3.5     # inclusive lower bound for this image
    cuda_capability_max: 8.0     # informational; null = open-ended
    note: "cupy-cuda11x, ubuntu22.04/cuda11.8"

  # Newer image for higher-capability GPUs. Delivered via osdf: only the
  # matched machine fetches it (selective transfer).
  - label: modern
    image: osdf:///igwn/staging/oshaughn/rift_containers/rift_container_modern.sif
    cuda_capability_min: 8.0
    cuda_capability_max: null
    note: "cupy-cuda12x, ubuntu22.04/cuda12.4"

Field

Meaning

version

Manifest schema version (currently 1).

capability_attr

Machine ClassAd attribute the selection expression tests (default GPUs_Capability).

fallback

label of the catch-all image (innermost else); must be CPU-safe.

containers[].label

Human id; also referenced by fallback.

containers[].image

A CVMFS/local path (referenced in place, lazy-fetched) or an osdf:// URL (selectively transferred).

containers[].cuda_capability_min

Inclusive lower capability bound for this image.

containers[].cuda_capability_max

Informational upper bound (null = open-ended).

containers[].note

Free text.

A starting manifest lives at containers/rift_container_family.yaml in the source tree.

Warning

Keep the family consistent. A single SINGULARITY_BASE_EXE_DIR is applied to every image in the family — the ILE/CIP jobs locate the executable as SINGULARITY_BASE_EXE_DIR + <exe name>, with no per-image override. Every image in a manifest must install RIFT’s executables at the same in-container path (and share a common layout / Python / entrypoints). Build them from the same rift_container.def.in template (build_family.sh does this); do not hand-mix images with different internal layouts. The same applies to SINGULARITY_BASE_EXE_DIR_HYPERPIPE if you use hyperpipe.

What the pipeline generates

For a manifest, the ILE (and CIP) Condor submit files get:

  • ``MY.SingularityImage`` — an unquoted ifThenElse expression that selects the highest-capability image the matched machine can run, with the fallback image as the innermost else (used when the machine’s capability is below every threshold):

    ifThenElse(TARGET.GPUs_Capability >= 8.0, "./rift_container_modern.sif", "/cvmfs/.../rift_container_default.sif")

  • Selective transfer — only osdf:// images are fetched, and only on the machine that selected them, via a single HTCondor $$() match-time token appended to transfer_input_files. CVMFS/local images are referenced in place and never transferred, so the whole family is never pulled:

    $$([ (TARGET.GPUs_Capability >= 8.0 ? "osdf:///.../rift_container_modern.sif" : "") ])

    request_disk is not auto-sized — set it to your largest single transferred image.

  • ``require_gpus`` floor — Capability >= <lowest min across the family>, combined (&&) with any RIFT_REQUIRE_GPUS you set. Both apply; neither is dropped. This stops jobs matching a GPU that no image in the family supports.

  • A capability-defined ``Requirements`` clause — TARGET.GPUs_Capability =!= undefined. The selection above is deliberately not undefined-guarded: a slot that does not advertise the machine-level capability rollup could be anything (including a Blackwell that hard-fails on the older fallback image), so the safe action is to not match it rather than guess. Measured on the CIT pool, a large fraction of GPU slots satisfy the per-GPU require_gpus floor yet do not advertise the rollup attribute; without this clause those jobs go on hold with “Cannot expand $$ expression”.

CPU-only jobs are handled differently. CIP requests no GPU, so its matched slot advertises no capability at all and a capability-keyed selection cannot resolve — it would hold the job. CIP therefore collapses to a single fixed container: the manifest fallback image (hence the requirement that the fallback be CPU-safe), quoted as a plain literal, with no $$() token and no capability Requirements clause.

Choosing a delivery mode (legacy vs container universe)

The expression-valued MY.SingularityImage above is evaluated on the execute side. That works on a local HTCondor pool, but OSPool glidein pilots read SingularityImage as a literal string, so an ifThenElse lands verbatim and the job holds. Two opt-in modes solve this; pick one with an environment variable at DAG-build time.

Mode

Behaviour

(default)

Legacy: universe = vanilla + expression-valued MY.SingularityImage. Correct on a local/CIT pool. Not OSG-safe.

RIFT_CONTAINER_UNIVERSE=1

Recommended for OSG. universe = container and container_image = $$([ ifThenElse(...) ]). $$() is HTCondor’s documented match-time (schedd-side) machine-ad substitution, so the pilot only ever sees a literal image URL. No MY.SingularityImage, no MY.SingularityBindCVMFS, no $$() transfer token — the image is delivered by container_image itself. GPU access is automatic under request_gpus. Works on CIT-local too.

RIFT_CONTAINER_RUNTIME_SELECT=1

Older fallback, ILE only. Condor runs a generated rift_container_select.sh on the bare execute node; the wrapper reads the real GPU capability from nvidia-smi, fetches only the matching image (stashcp/pelican), and re-execs the command under apptainer exec --nv. Detects the actual device rather than trusting an advertised attribute, at the cost of running outside a container.

Under asimov, set the variable from the blueprint rather than the shell:

scheduler:
  singularity image: /path/to/rift_container_family.yaml
  singularity base exe directory: /usr/local/bin/
  environment variables:
    RIFT_CONTAINER_UNIVERSE: 1

Note

With a family manifest whose images are osdf:// URLs, the pipeline also enables the matching transfer credential automatically (use_oauth_services = scitokens, or igwn for igwn+osdf:). The single-image code path keys that off the SINGULARITY_RIFT_IMAGE string itself, which for a manifest is just a .yaml path — so the manifest’s image URLs are inspected instead.

Warning

Why the container-universe selector names basenames, not URLs. condor_submit parses container_image before any $$ expansion and derives the job ad’s ContainerImage — the name the image will have in the job scratch dir — as the text after the last /. A selector containing full paths is therefore cut in half, and the fragment that survives is not a valid image name. Submitting that form to the IGWN pool holds the job at the execute point:

PREPARE_JOB (prepare-hook) failed (reported status 001):
Unable to download or build singularity image cutest_busybox_...sif") ])

So the selector emits basenames only (no /), the whole $$ token survives into ContainerImage, and the schedd expands it at match time (MATCH_EXP_ContainerImage = "rift_container_modern.sif"). The image itself is delivered by the comma-free $$() transfer token, and MY.TransferInput is pinned so condor_submit does not append the basename selector to TransferInput as a bogus extra input file. Verified end to end on an OSPool glidein against $CondorVersion: 25.11.1.

Consequence: every image in a family used with container universe must be a transferable URL. An image referenced in place (a CVMFS or local path) can only be named by its full path, which would reintroduce the truncation, so build_container_image_select() raises ContainerManifestError for such a family. Stage those images at a URL, or use RIFT_CONTAINER_RUNTIME_SELECT=1.

Per-host ILE executable and memory

A family entry may also choose the ILE executable and the memory request. For example, to run the JAX ILE on GPUs with at least 16 GB and the standard ILE elsewhere:

fallback: legacy
containers:
  - label: legacy
    image: osdf:///igwn/.../rift_cuda12.sif
    cuda_capability_min: 6.0
  - label: jax_big
    image: osdf:///igwn/.../rift_cuda12.sif   # entries may share an image
    cuda_capability_min: 6.0
    select_requirements: TARGET.GPUs_GlobalMemoryMb >= 16000
    ile_exe: integrate_likelihood_extrinsic_jax
    request_memory: 16000                     # MB

select_requirements is ANDed into the entry’s capability test as (expr) =?= true, so a slot that does not advertise an attribute it reads falls through to the next entry. At equal cuda_capability_min an entry with select_requirements is tested first. It needs cuda_capability_min, may not contain , or /, and the fallback may not set it. Entries without ile_exe or request_memory use the job defaults.

A GPU ILE job then carries one ClassAd selector per choice, all over the same branches: the image, request_memory, MY.RIFTILEExe, and MY.RIFTILEProfile (the label). The executable is passed as a trailing --rift-ile-exe=$$([MY.RIFTILEExe]) argument, which ile_pre.sh (or a small ile_dispatch.sh when there is no frame prescript) removes before running it. HTCondor carves the slot at the selected memory, and keeps the expressions so a job rematched after eviction selects again. To see what each job got:

condor_history <cluster> -af MATCH_EXP_RIFTILEProfileMatched MemoryProvisioned

Limits:

  • Opt-in per job (ile_profiles=True). Only the ILE, ILE_puff and ILE_fetch jobs of create_event_parameter_pipeline_BasicIteration use it. ILE_extr keeps the default executable, because its output converter depends on which executable ran.

  • GPU jobs and HTCondor backends only.

  • Not available with RIFT_CONTAINER_RUNTIME_SELECT, which picks the image after the slot is carved and cannot evaluate select_requirements.

  • The build-time checks util_RIFT_pseudo_pipe.py applies to --use-jax-ile (calibration marginalization, OSG) do not see a manifest ile_exe. Check that the run’s ILE options are supported by every executable the family names.

  • Select on attributes that are the same for the partitionable and the dynamic slot (GPU capability, GPU memory, TotalCpus), not on Memory.

  • condor_qedit <job> RequestMemory <N> replaces the expression with a constant.

GPU attribute names

Two different ClassAd namespaces are involved, and they are kept separate:

  • The image selection ifThenElse reads the machine ad. The default attribute is GPUs_Capability. Override it per run with the environment variable RIFT_GPU_CAPABILITY_ATTR, or per manifest with capability_attr. Verify what your pool advertises:

    condor_status -constraint 'TotalGPUs > 0' -autoformat GPUs_DeviceName GPUs_Capability GPUs_GlobalMemoryMb

    Not every GPU host advertises this; on such hosts the expression collapses to the fallback image and the require_gpus floor does the steering.

  • The ``require_gpus`` floor uses the require_gpus sub-ad attribute Capability (unprefixed — not TARGET., not GPUs_).

Note

These mechanisms have been validated on a real HTCondor pool + GPU: the attribute names, the require_gpus floor (matching a compatible GPU and excluding an incompatible one), the $$() match-time image selection, and tolerance of the empty-result case for a manifest that mixes CVMFS and osdf entries. The remaining item for a first real OSG run is that the pilot evaluates the expression-valued MY.SingularityImage and honors a relative ./name.sif produced by it.

Building a container family

The build lives under containers/:

  • rift_container.def.in — an Apptainer definition template with @@BASE_IMAGE@@ / @@CUPY_PKG@@ placeholders.

  • build_family.sh — renders one .def per build-matrix entry and runs apptainer build. The first matrix entry keeps the current production base image, so the family always includes a broadly-compatible image for older machines.

  • requirements-container.txt — the shared, unpinned pip dependency set (the cupy wheel is the only per-entry difference).

# render the per-entry .def files only (no apptainer needed)
containers/build_family.sh --render-only ./container_family

# render and build each .sif (requires apptainer)
containers/build_family.sh ./container_family

# on shared clusters (e.g. CIT), build with --fakeroot to avoid the
# unprivileged proot engine (whose mksquashfs step fails):
containers/build_family.sh --fakeroot ./container_family

Each run also writes a rift_container_family.generated.yaml stub: fill in each image: with where you published the .sif (a CVMFS path or osdf:// URL) and you have a deployable manifest.

Note

On clusters without setuid apptainer or unprivileged user namespaces, a plain build falls back to the proot engine and fails at mksquashfs (ptrace(TRACEME): Operation not permitted). Use --fakeroot (needs /etc/subuid + /etc/subgid entries), or --sandbox to build a directory that skips mksquashfs and convert it to a .sif later on a capable host. See the build-troubleshooting section of containers/README.md.

The top-level rift_container.def is unchanged and remains the default single-image build.

Catching dependency breakage early (CI canary)

The container ships an unpinned dependency set and clones RIFT at build time, so a fresh upstream release (for example swig>=4.4.0) can silently break RIFT and only surface when a container rebuild fails. Two non-blocking CI jobs guard against this — container-dep-canary (installs the same unpinned containers/requirements-container.txt set and runs the import check) and container-swig-canary (exercises the pixi swig-post44 lane). They run on every push/PR and on a weekly schedule, so a breaking upstream release is flagged even with no RIFT commit.