Using the pipeline

Basics

For optional analytic fits of the joint extrinsic posterior, see Posterior compression. The final CIP stage determines the intrinsic spin prior used by those fits; the data rows come from the extrinsic export.

The primary user-interface for this code is a command line tool util_RIFT_pseudo_pipe.py which is available after following the installation instructions. For detailed documentation, see util_RIFT_pseudo_pipe.py. To see the help for this tool, run

$ util_RIFT_pseudo_pipe.py --help

If you leave options unspecified, the pipeline will often try to make guesses about the options you want to use, based on previous experience performing real GW inference. We strongly recommend most users fully specify the options involved via an ini file. To compare the options for ini file use versus command line specified:

$ util_RIFT_pseudo_pipe.py --use-ini my.ini --use-coinc coinc.xml
# or, for a freestanding example
$ util_RIFT_pseudo_pipe.py --gracedb-id G329473 --approx IMRPhenomD --calibration C01 --make-bw-psds --l-max 2 --choose-data-LI-seglen

Note that the code will use selected environment variables to identify optional external dependencies necessary for various features. The most important feature for most users is their accounting access information

export LIGO_USER_NAME=albert.einstein
export LIGO_ACCOUNTING=ligo.dev.o4.rift

For eccentric PE (--assume-eccentric), preliminary CIP iteration groups use --eccentricity-prior uniform to explore the likelihood. The pipeline option --eccentricity-prior log_uniform selects the prior for every Z run-to-convergence subdag and the final CIP argument group. A convergence subdag retains that target prior even when followed by a final extrinsic-sampling group; all of its child CIP jobs use the same prior. The default, uniform, keeps all groups uniform. A schedule containing only one group uses the requested final prior throughout. Inspect args_cip_list.txt before submission to see the prior assigned to each group. This scheduling policy also applies when extra CIP arguments specify an eccentricity prior; direct CIP invocations continue to use the prior passed to them.

When you run util_RIFT_psuedo_pipe.py, the pipeline will create a directory structure as follows:

long_directory_name_here/
   -> local.cache
   -> iteration_0_ile/
   -> iteration_0_cip/
   -> iteration_0_con/
   -> iteration_0_test/
   -> iteration_1_ile/
   -> ...
   -> ILE.sub
   -> CIP*.sub
   -> marginalize_intrinsic_parameters_BasicIterationWorkflow.dag

The directory structure and DAG workflow are created by the create_event_parameter_pipeline_BasicIteration executable, which is called internally by util_RIFT_pseudo_pipe.py. For detailed documentation on this pipeline generator, see create_event_parameter_pipeline_BasicIteration.

Inside each iteration file is a logs subdirectory. The various iterations will be initially be empty, except for the logfile locations. The top level directory contains several *.sub submission scripts, along with the top-level dag submission script.

The standard RIFT pipeline only works within an HTCondor scheduling environment. To submit the workflow, use

$ condor_submit_dag marginalize_intrinsic_parameters_BasicIterationWorkflow.dag

Before you submit a workflow, however, we recommend you first confirm you’ve set it up correctly by running one of the worker jobs interactively from the command line. This is a great way to catch common configuration errors. Within the directory, there should be a script called command-single.sh. This contains a single worker job, so you can simply run this script to confirm that your worker jobs will proceed smoothly.

$ ./command-single.sh

This command will run anywhere; however, it will only test the GPU configuration if you run it on a machine with a GPU, like pcdev13 or pcdev11 at CIT. You’ll see a lot of output about reading in data, defining parameters, et cetera. Wait until you start seeing large arrays of numbers interspersed with the words Weight entropy (after histogram). At this point you may kill the script and submit the DAG as described above. Feel free to watch the job progress by:

$ watch condor_q

The workflow loosely consists of two parts: worker ILE jobs, which evaluate the marginalized likelihood; and fitting/posterior jobs, which fit the marginalized likelihood and estimate the posterior distribution. Other nodes help group the output of individual jobs and iterations together.

As your run proceeds, files will begin to appear in your directory. A description of some of the files is as follows:

overlap-grid-0.xml.gz: The initial grid used in the iterative analysis. You’re free to use any grid you want (e.g., the output of some previous analysis), and the workflow can also do the initial grid creation.
ILE.sub: The submit file for the individual worker ILE jobs. This contains the command line arguments passed to integrate_likelihood_extrinsic. If something is going wrong when your ILE jobs run, this file is a good place to check to ensure the code is using the settings you intended.
CIP.sub: The submit file for the individual fitting jobs.
iteration_*: Directories holding the output of each iteration, including log files.

As the workflow progresses, you’ll see the following additional files

consolidated_*: These files (particularly those ending in .composite) are the output of each iteration’s ILE jobs. Each file is a list of intrinsic parameters and the value of the marginalized likelihood at those parameters. (The remaining files provide provenance for how the .composite file was produced.)
output-grid-?.xml.gz: These files are inferred intrinsic, detector-frame posterior distributions from that iteration, expressed as an XML file.
posterior-samples-*.dat: These files are reformatted versions of the corresponding XML file, using the command convert_output_format_ile2inference. This data format should be compatible with LALInference and related postprocessing tools. The final output posterior samples are used to create PP plots. Corner plots for a user specified number of iterations are also created using these files.

Understanding ILE and CIP

ILE.sub

The ILE.sub file contains the call to and arguments for integrate_likelihood_extrinsic_batchmode. This step integrates the extrinsic parameters of the prefactored likelihood function. The marginalized likelihood is calculated during individual parallel worker jobs to be passed to the next (CIP) step.

CIP.sub

The file called CIP.sub contains the call to and arguments for util_ConstructIntrinsicPosterior_GenericCoordinates.py. During this step, the log-likelihoof data is loaded in and the peak is fitted using some particular coordinate system. This is passed as an input to the Monte Carlo sampler where samples are drawn from the posterior distribution. These samples become the inputs for the successive iteration.

Initialization: PSDs and grids

The RIFT pipeline by default can attempt to generate a PSD using existing tools. However, most users will want to supply a contemporary PSD for analysis on real GW events. RIFT uses an XML format, requiring some conversion for the input PSDs.

Strongly recommended dependencies

We strongly recommend you install cuda and cupy, and properly define your environment variables for such an install

# should be provided by igwn
export CUDA_DIR=/usr/local/cuda  # only needed for GPU code
export PATH=${PATH}:${CUDA_DIR}/bin  # only needed for GPU code

Additional environment variables are needed if you want to use waveforms through a non-lalsimulation interface. Such waveforms include the python implementation of surrogate waveforms; NR waveforms; or the C++ implementation of TEOBResumS. While we provide the necessary environment variables below, please contact one of the developers for appropriate settings, and keep in mind some surrogates and/or simulations and/or waveforms may be provided in advance of publication or release to the broader community.

export NR_BASE=/home/oshaughn/unixhome/PersonalNRArchive/Archives/
export GW_SURROGATE= # your installation of gwsurrogate goes here
export PYTHONPATH=${PYTHONPATH}:${GW_SURROGATE}

util_RIFT_pseudo_pipe.py help

For reference, here is the full output of

$ util_RIFT_pseudo_pipe.py --help

usage: util_RIFT_pseudo_pipe.py [-h] [--use-production-defaults] [--use-subdags] [--use-ini USE_INI] [--use-rundir USE_RUNDIR]
                             [--use-online-psd-file USE_ONLINE_PSD_FILE] [--use-coinc USE_COINC] [--manual-ifo-list MANUAL_IFO_LIST] [--online]
                             [--extra-args-helper EXTRA_ARGS_HELPER] [--manual-postfix MANUAL_POSTFIX] [--gracedb-id GRACEDB_ID] [--gracedb-exe GRACEDB_EXE]
                             [--use-legacy-gracedb] [--internal-use-gracedb-bayestar] [--event-time EVENT_TIME] [--calibration CALIBRATION] [--playground-data]
                             [--approx APPROX] [--use-gwsurrogate] [--l-max L_MAX] [--no-matter] [--assume-nospin] [--assume-precessing]
                             [--assume-nonprecessing] [--assume-matter] [--assume-lowlatency-tradeoffs] [--assume-highq] [--assume-well-placed]
                             [--internal-marginalize-distance] [--internal-marginalize-distance-file INTERNAL_MARGINALIZE_DISTANCE_FILE]
                             [--internal-distance-max INTERNAL_DISTANCE_MAX] [--internal-correlate-default]
                             [--internal-force-iterations INTERNAL_FORCE_ITERATIONS] [--internal-flat-strategy] [--internal-use-amr]
                             [--internal-use-amr-bank INTERNAL_USE_AMR_BANK] [--internal-use-amr-puff] [--internal-use-aligned-phase-coordinates]
                             [--external-fetch-native-from EXTERNAL_FETCH_NATIVE_FROM] [--internal-propose-converge-last-stage] [--add-extrinsic]
                             [--batch-extrinsic] [--fmin FMIN] [--fmin-template FMIN_TEMPLATE] [--data-LI-seglen DATA_LI_SEGLEN] [--choose-data-LI-seglen]
                             [--fix-bns-sky] [--ile-sampler-method ILE_SAMPLER_METHOD] [--ile-n-eff ILE_N_EFF] [--cip-sampler-method CIP_SAMPLER_METHOD]
                             [--cip-fit-method CIP_FIT_METHOD] [--cip-internal-use-eta-in-sampler] [--ile-jobs-per-worker ILE_JOBS_PER_WORKER] [--ile-no-gpu]
                             [--ile-force-gpu] [--fake-data-cache FAKE_DATA_CACHE] [--spin-magnitude-prior SPIN_MAGNITUDE_PRIOR]
                             [--force-chi-max FORCE_CHI_MAX] [--force-mc-range FORCE_MC_RANGE] [--force-eta-range FORCE_ETA_RANGE]
                             [--force-hint-snr FORCE_HINT_SNR] [--force-initial-grid-size FORCE_INITIAL_GRID_SIZE] [--hierarchical-merger-prior-1g]
                             [--hierarchical-merger-prior-2g] [--link-reference-pe] [--link-reference-psds] [--make-bw-psds] [--link-bw-psds]
                             [--use-online-psd] [--ile-retries ILE_RETRIES] [--general-retries GENERAL_RETRIES]
                             [--ile-runtime-max-minutes ILE_RUNTIME_MAX_MINUTES] [--fit-save-gp] [--cip-explode-jobs CIP_EXPLODE_JOBS]
                             [--cip-explode-jobs-last CIP_EXPLODE_JOBS_LAST] [--cip-quadratic-first] [--n-output-samples N_OUTPUT_SAMPLES]
                             [--internal-cip-cap-neff INTERNAL_CIP_CAP_NEFF] [--internal-cip-temper-log] [--internal-ile-sky-network-coordinates]
                             [--internal-ile-freezeadapt] [--internal-ile-adapt-log] [--manual-initial-grid MANUAL_INITIAL_GRID]
                             [--manual-extra-ile-args MANUAL_EXTRA_ILE_ARGS] [--verbose] [--use-quadratic-early] [--use-gp-early] [--use-cov-early] [--use-osg]
                             [--use-osg-file-transfer] [--condor-local-nonworker] [--condor-nogrid-nonworker] [--use-osg-simple-requirements]
                             [--archive-pesummary-label ARCHIVE_PESUMMARY_LABEL] [--archive-pesummary-event-label ARCHIVE_PESUMMARY_EVENT_LABEL]

Expand below for a description of each of the optional arguments: