Pipelines

A tutorial shows how to do an analysis; a pipeline does it the same way every time and leaves a folder that a person, a script or an agent can read without running the code again. virgil.pipeline wraps virgil's public functions in fixed, named stages. Each stage writes its own outputs, an interrupted run resumes, and every run ends with deterministic quality checks and an executed quicklook notebook.

A pipeline adds no science. Every number it reports comes from a public virgil function that a tutorial also calls, so any result can be reproduced by hand. The binary-search, contrast-limits and limb-darkening tutorials end with an "As a pipeline" section that does exactly that and asserts that the numbers agree.

Install the extra (it pulls in the plotting extra, h5py, nbformat, nbclient and ipykernel):

pip install 'virgil-astro[pipeline]'

import virgil never imports the pipelines; virgil.pipeline loads on first use.

Running a pipeline

import virgil as vg
from virgil.pipeline import BinaryPipeline, load

data = vg.OIData("hd1234.oifits")
res = BinaryPipeline(data, output="runs/hd1234", sigma=3.0).run()

res.summary["companion"]["sep_mas"]
res.checks            # a list of Check(name, status, value, threshold, message)
print(res.describe())

res = load("runs/hd1234")   # later, in a fresh session; nothing is recomputed
model = res.model()         # the best-fit model, a real virgil object

A pipeline takes the data, a model template, a run folder and keyword settings. Settings resolve as the class defaults (BinaryPipeline.defaults()), then a JSON config file (BinaryPipeline.from_config(path)), then keyword arguments. run(through="limits") stops after a stage, which is how a non-detection skips the fit and the sampler.

A run resumes by default: a completed stage whose outputs exist is skipped, and once a stage reruns every later stage reruns. A resumed run refuses, with ConfigMismatchError, when the settings, the data hash or the model template differ from those stored in the folder. run(resume=False) starts again.

The stages of BinaryPipeline

The fit is on the quoted errors unless error_scale="fit", and chi-squared per independent datum is always reported on the quoted errors. Priors are group-invariant: uniform in position (dra and ddec on a box of half-width max_sep_mas) and log-uniform in flux and in error scales. Both templates sample the same dra, ddec and flux, so they give the same prior on the sky; a BinaryModelAngular template reports sep and pa derived from them, with no wrap at 0 or 360 degrees in the fit. The flux range is a setting (flux_range).

The search and the limits always use a point companion in Cartesian offsets; the model template (BinaryModelAngular or BinaryModelCartesian) sets the parameters of the fit and the posterior.

StarPipeline: stellar diameters

from virgil.pipeline import StarPipeline

res = StarPipeline(data, output="runs/star").run()   # uniform and limb-darkened, compared
res = StarPipeline(data, "uniform", output="runs/ud").run()
res.summary["star"]["diam_mas"], res.model(), res.model("uniform")

model= is a short name, "uniform" (UniformDisk) or "limb_darkened" (QuadraticLimbDarkenedDisk with Kipping's q1, q2), or an instance of one of those classes (its q1 and q2 are the starting values). The default fits both and compares them. Each fitted model is saved under models/<name>/, with models/best the preferred one: the limb-darkened model only if its delta chi-squared exceeds the BIC penalty of its two extra parameters. res.model(), res.model_values(), res.fit_result(), res.samples() and res.sample_stats() take an optional name= to pick one.

Priors are group-invariant: log-uniform in the diameter over diam_range_mas (by default from a tenth of the resolution limit to four times the field of view) and uniform on [0, 1] in q1 and q2, which cover exactly the physical quadratic laws. The fit is on the quoted errors, and chi-squared per independent datum is reported on them for each model.

The run folder

Everything is JSON, HDF5, NumPy .npz, PNG, OIFITS or one notebook, with no pickles. Each file is written atomically, so a crash never leaves a half-written file under its final name, and nothing is written outside the run folder.

Reading a run: load and Result

load(path) returns a Result. Nothing is recomputed: each accessor reads the files the run wrote.

Checks

Checks are pure functions of plain numbers. The same run always gives the same checks, in a fixed order, each with a status of pass, warn or fail. The first is always the chi-squared per independent datum of the model on the quoted errors: a fitted error scale makes chi-squared close to one by construction, so it is never the headline number.

Warnings raised inside stages

A stage runs under warnings.catch_warnings(record=True) with the filter set to "always", so a warning raised by a virgil function inside a stage never reaches the terminal or the notebook. Each distinct message is recorded once per stage, with its stage, category and message, in stages/<name>/report.json and in the warnings list of summary.json; describe() counts them. The known messages are also turned into checks, matched by keywords so that rewording a message does not break the mapping. These checks appear only when the warning occurred, after the checks above, and always warn:

Deprecation notices are recorded in warnings but raise no check. The quicklook notebook's own kernel runs on the CPU (JAX_PLATFORMS=cpu, no memory preallocation), so it does not compete with the run for a GPU, and its stderr goes to quicklook.log.

Stability

A pipeline is either stable or provisional; the tier is the class attribute STABILITY, and it is written to run.json and summary.json.

A stable pipeline keeps the following within a schema version (virgil-pipeline-run-v1):

• the constructor and run signatures, the stage names and their order, and the setting names and defaults;
• every key of summary.json and run.json, and its meaning;
• the check names, their order and their thresholds;
• the file names in the run folder.

Changes are additive: new settings whose defaults keep the old behaviour, new summary keys, new checks appended at the end, and new files. Anything else needs a new schema version, and load refuses schemas it does not know. Snapshot tests pin the signatures, defaults, summary keys and check order.

A provisional pipeline writes the same folder layout and uses the same Result, but its stages, settings and summary keys may change while the virgil functions underneath settle. It emits one FutureWarning per session.

Further pipelines (orbit and imaging, both provisional) will be added to this table as they land.

Command line

virgil-pipeline runs and inspects pipelines from a shell.

virgil-pipeline binary hd1234.oifits -o runs/hd1234 --set sigma=3 --set num_chains=2
virgil-pipeline binary hd1234.oifits -o runs/hd1234 --through limits --fresh
virgil-pipeline star star.oifits -o runs/star --model limb_darkened
virgil-pipeline info runs/hd1234        # status and checks
virgil-pipeline validate runs/hd1234    # complete, and every file matches its hash
virgil-pipeline quicklook runs/hd1234   # rebuild the quicklook notebook

--set KEY=VALUE overrides a setting (the value is parsed as JSON where it can be). info and validate read only JSON and hashes and do no JAX work, although import virgil still imports JAX.

API reference

The classes and functions are documented on the pipeline API page.