Skip to content

virgil.detection

Detection statistics of a companion search over a grid: the profile likelihood ratio Δχ², the grid-marginalized log Bayes factor and the best flux SNR. They are traceable in the data, so they can be computed for many simulated observations under jax.lax.map with one compilation, which is what empirical false-alarm probabilities and ROC curves need.

The Monte Carlo: gaussian_null and bootstrap_null simulate the null hypothesis (Gaussian noise from the errors, or a residual bootstrap of the data), rescale_errors first brings the null to χ²_r = 1, injection_grid lays out companions to inject, and injection_recovery runs the search on null and injected simulations with one compiled kernel. Its DetectionMC result gives empirical false-alarm probabilities, thresholds, ROC curves, completeness maps and contrast curves, and saves, loads and concatenates for array jobs.

Detection statistics for companion searches, for ROC curves.

detection_statistics reduces a companion search over a grid to three numbers that measure how strongly the data prefer a companion to none:

  • delta_chi2, the profile likelihood ratio 2 [max log L − log L₀] over the grid, with the companion flux constrained to be non-negative;
  • log_bayes_factor, the log evidence ratio of "a companion somewhere on the grid" to "no companion", marginalized over the grid;
  • max_snr, the largest best-fit flux over its Laplace uncertainty, the significance map of the composition tutorial.

The null hypothesis is the same model with the companion flux set to zero and every other parameter as given, so fixed known components (a resolved star, a disk, a companion already found) stay in both hypotheses.

The function is traceable in the data: inside jax.jit or jax.lax.map over simulated observations it compiles once, which is what Monte Carlo estimates of false-alarm rates and completeness need.

local_nsigma converts delta_chi2 to Wilks's Gaussian-equivalent significance at a single position. It ignores the look-elsewhere effect of searching a grid, so it overstates the significance of the best of many positions; calibrate with simulations.

The simulations:

  • gaussian_null and bootstrap_null build simulators, simulate(key, scene=None) -> OIData: Gaussian noise from a template's errors, or a residual bootstrap of real data about the null scene. rescale_errors first scales errors so that the null scene has χ²_r = 1.
  • injection_grid lays out companions to inject, and injection_recovery runs the statistics on null and injected simulations with one compiled kernel.
  • Its result, a DetectionMC, holds plain NumPy arrays and gives empirical false-alarm probabilities, thresholds, ROC curves, completeness maps and contrast curves; it saves, loads and concatenates, so that array jobs can be merged.

DetectionMC dataclass

Statistics of simulated null and injected companion searches.

Made by injection_recovery; plain NumPy, with no JAX inside.

Attributes:

Name Type Description
null dict[str, ndarray]

Per null draw: delta_chi2, log_bayes_factor, max_snr, the diagnostics flux_peak_steps and converged_fraction (of the flux optimizer over grid positions), and the best position and flux as best_<name> (e.g. best_dra, best_flux), where name is the last dotted part of each grid key.

injected dict[str, ndarray]

The same per injected draw, plus the injected values under their names (e.g. dra, ddec, flux).

meta dict

JSON-serializable: the grid, fingerprints (hashes of every field, static or not) of the model, the null scene and the template, the noise model, match_radius, the numbers of draws, the seeds and the virgil version.

Notes

Conventions: the false-alarm probability of a value is the fraction of null draws at or above it; at a threshold t a draw is detected when its statistic exceeds t (and, with match_radius, its best position matches the injection). Separations are hypot(dra, ddec) of the injections, or their sep for angular grids (sep in mas, pa in degrees), in mas; fluxes are relative to the primary, as in absil_limits.

The single-position null of delta_chi2 is ½δ₀ + ½χ²₁, so a local 3σ (local_nsigma = 3, delta_chi2 = 9) is a one-sided FAP of 0.135% (scipy.stats.norm.sf(3)), not the two-sided 0.27%; over a grid the empirical FAP of that value is larger.

null instance-attribute

injected instance-attribute

meta instance-attribute

n_null property

Number of null draws.

n_injected property

Number of injected draws.

__init__(null, injected, meta)

__repr__()

separations()

Separations of the injections in mas.

hypot(dra, ddec) for Cartesian grids, or the injected sep for angular ones (sep, pa in degrees).

matched()

Whether each injection's best position is within match_radius.

All True when meta["match_radius"] is None. Angular positions (sep, pa in degrees) are compared on the sky.

false_alarm_probability(stat, value, *, confidence=0.95)

Empirical false-alarm probability of value, with an interval.

Parameters:

Name Type Description Default
stat str

"delta_chi2", "log_bayes_factor" or "max_snr".

required
value float or array - like

Observed statistic(s).

required
confidence float

Confidence of the binomial interval.

0.95

Returns:

Type Description
fap, lower, upper : numpy.ndarray

fap = (k + 1) / (n + 1) for k of the n null draws at or above value: the Monte Carlo p-value, never 0, and conservative. lower and upper bound the true probability P(null ≥ value) given k of n, with the exact (Clopper–Pearson) binomial interval.

threshold(stat, fap, *, n_boot=200, seed=0)

Threshold of stat at false-alarm probability fap.

Parameters:

Name Type Description Default
stat str

The statistic.

required
fap float

False-alarm probability (e.g. 1.35e-3, the one-sided Gaussian tail at 3σ).

required
n_boot int

Bootstrap resamples of the null draws, for the error.

200
seed int

Seed of the bootstrap.

0

Returns:

Type Description
threshold, error : float

The 1 - fap quantile of the null draws (linear interpolation), and its bootstrap standard deviation.

Warns:

Type Description
RuntimeWarning

If fewer than one null draw is expected above the threshold (fap * n < 1): it is then about the largest null value and underestimates the true threshold.

Notes

The bootstrap runs in batches of resamples, holding about 2²² values at a time, so its memory does not grow with n_boot × n.

detected(stat, fap)

Whether each injection is detected at false-alarm probability fap.

Its statistic exceeds :meth:threshold and, with match_radius, its best position matches the injection.

roc(stat, flux=None, sep_bin=None)

ROC curve: true- against false-positive rate over thresholds.

Parameters:

Name Type Description Default
stat str

The statistic.

required
flux float or (float, float)

Only the injections of this flux (to relative precision 1e-6), or with lo <= flux < hi.

None
sep_bin (float, float)

Only the injections with lo <= separation < hi (mas).

None

Returns:

Type Description
fpr, tpr, thresholds : numpy.ndarray

For each threshold t (+inf, then every distinct value of the statistic in decreasing order), the fraction of null draws with a statistic ≥ t, and of the selected injections with a statistic ≥ t (and matched, with match_radius). fpr rises from 0 to 1.

auc(stat, flux=None, sep_bin=None)

Area under the :meth:roc curve (0.5 for no discrimination).

completeness(stat, fap, sep_bins=None, flux_bins=None)

Detection fraction of the injections by separation and flux.

Parameters:

Name Type Description Default
stat str

The statistic.

required
fap float

False-alarm probability of the detection threshold.

required
sep_bins array - like

Bin edges (mas, and flux). By default each distinct injected separation or flux (to six significant figures) is its own bin, which suits an injection_grid.

None
flux_bins array - like

Bin edges (mas, and flux). By default each distinct injected separation or flux (to six significant figures) is its own bin, which suits an injection_grid.

None

Returns:

Type Description
dict

completeness (n_sep × n_flux, NaN in empty bins), n (the injections per bin), sep and flux (the distinct values, or the bin centres: arithmetic, except geometric for flux bins whose two edges are positive), and threshold.

contrast_curve(stat, fap, completeness=0.5, sep_bins=None, flux_bins=None)

Flux detected with a given completeness, against separation.

In each separation bin of :meth:completeness, the detection fraction is made non-decreasing in flux (a running maximum) and interpolated linearly in log flux to where it first reaches completeness.

Parameters:

Name Type Description Default
stat str

The statistic.

required
fap float

False-alarm probability of the detection threshold.

required
completeness float

Target detection fraction, e.g. 0.5 or 0.9.

0.5
sep_bins array - like

As for :meth:completeness.

None
flux_bins array - like

As for :meth:completeness.

None

Returns:

Type Description
sep, flux : numpy.ndarray

Separations (mas) and the companion flux relative to the primary, the units of absil_limits (convert with flux_to_contrast or flux_to_delta_mag). NaN where the positive injected fluxes do not bracket the target: it is never reached, or already exceeded at the lowest flux.

save(path)

Write to path, an .npz with the metadata as JSON.

load(path) classmethod

Read a file written by :meth:save.

concatenate(results) classmethod

Merge results of one experiment run with different seeds.

Parameters:

Name Type Description Default
results sequence of DetectionMC

E.g. one per array job. They must agree on the grid, the model, the null scene, the template, the noise model and match_radius, and their seeds must all differ (one seed repeats the same draws).

required

Returns:

Type Description
DetectionMC

All the null and injected draws, with every seed in meta.

Raises:

Type Description
ValueError

If the metadata are incompatible or a seed repeats.

detection_statistics(model, data, grid, *, flux_param=None, batch_size=None)

Detection statistics of a companion search over a grid.

Parameters:

Name Type Description Default
model SourceModel or class

Companion model, as for likelihood_grid: a template whose parameters at the paths in grid are varied (e.g. a System with "comp.dra"), or a class called with grid's keys (e.g. BinaryModelCartesian). The null hypothesis is this model with flux_param set to zero and everything else as given.

required
data OIData

Data to search. It may be traced (e.g. a simulation built inside jax.lax.map with OIData.with_model); the function then compiles once for all of them.

required
grid dict[str, array - like]

Grid axes, as for optimized_flux_grid: the coordinate keys (e.g. dra, ddec in mas) and a non-negative flux axis. The flux axis is both the starting grid of the flux optimizer and the flux prior of log_bayes_factor.

required
flux_param str

The key of grid holding the companion flux. By default, the one key whose last part is flux.

None
batch_size int

Number of grid points evaluated at once, as for likelihood_grid.

None

Returns:

Type Description
dict[str, Array]

Scalars:

  • delta_chi2: 2 [max over positions and fluxes ≥ 0 of log L − log L₀], always ≥ 0. At each position the flux is the best grid flux refined by BFGS (as in optimized_flux_grid); where the unconstrained best flux is negative the constrained one is 0, so that position adds nothing.
  • log_bayes_factor: log of the prior-weighted mean of L / L₀ over the full grid, logsumexp(log L − log L₀ + log w). The prior weights w sum to 1 and follow each axis's own spacing: trapezoid weights in the grid index (1 inside, ½ at the two ends) along every axis. So the prior is uniform over the searched box for evenly spaced coordinates, and uniform in log flux between the axis's ends for a log-spaced flux axis (uniform in flux for a linear one). Positive values favour a companion.
  • max_snr: the largest unconstrained best flux over its Laplace uncertainty, over positions (NaN-safe).
  • one entry per key of grid (e.g. dra, ddec, flux): the position where delta_chi2 is reached and the non-negative best flux there. If delta_chi2 is 0 the flux is 0 and the position is the first grid position.
  • flux_peak_steps: a diagnostic, the full width at half maximum of the likelihood peak in flux at that position (2.355 times the Laplace uncertainty) divided by the local spacing of the flux axis. Below about 2 the flux axis does not resolve the peak and log_bayes_factor is inaccurate.

Warns:

Type Description
RuntimeWarning

With concrete (untraced) data only: if the flux axis does not resolve the likelihood peak (flux_peak_steps < 2), if the best flux lies above the flux axis, or if the flux optimizer did not converge at some positions. Under jit/lax.map nothing is checked; inspect flux_peak_steps instead.

Notes

Every statistic uses the full likelihood grid, evaluated once. The flux axis must resolve the likelihood peak in flux for log_bayes_factor to approximate the evidence integral, and the coordinate spacing must resolve it in position (not checked); a coarse grid still gives a valid test statistic when its null distribution is simulated, but not an accurate evidence. A strong detection has a narrow peak, so a log-spaced axis needs many points per decade: the peak's relative width (FWHM over flux) is about 2.4/SNR, so two steps across it at SNR 10 need a step of 0.12 in ln(flux), about 20 points per decade.

The trapezoid weights make log_bayes_factor a trapezoid-rule integral over a prior with fixed bounds (the axes' ends), so it converges as the grid is refined. Equal weights per point would put the prior's edges half a step beyond the axes' ends, and, since L / L₀ stays near 1 far from a companion, would change the result at first order in the step.

delta_chi2 at one fixed position follows ½δ₀ + ½χ²₁ under the null (Chernoff 1954, the flux being on its boundary at 0); searching a grid makes it larger (the look-elsewhere effect). See local_nsigma.

Examples:

Statistics for many simulated null observations, compiled once:

>>> keys = jax.random.split(jax.random.PRNGKey(0), 100)
>>> stats = jax.lax.map(
...     lambda key: detection_statistics(
...         model, template.with_model(null_scene, key=key), grid
...     ),
...     keys,
... )

local_nsigma(delta_chi2)

Local (Wilks) significance of delta_chi2, in Gaussian sigma.

The significance a delta_chi2 would have at one position fixed in advance: under the null hypothesis it then follows ½δ₀ + ½χ²₁ (one flux, bounded at zero), whose upper tail at delta_chi2 is the one-sided Gaussian tail at sqrt(delta_chi2). This is nsigma with one degree of freedom.

It is local: it does not account for the look-elsewhere effect of taking the best of many grid positions, so for a search it overstates the significance. The global false-alarm probability needs simulations of the null over the same grid.

Parameters:

Name Type Description Default
delta_chi2 float or array - like

Profile likelihood ratio from detection_statistics.

required

Returns:

Type Description
array - like

Local significance in Gaussian sigma (sqrt(delta_chi2) up to rounding; it saturates at about 13σ in float32).

gaussian_null(null_scene, template, *, error_scale=1.0)

A simulator of the null hypothesis with Gaussian noise.

Each draw is template.with_model(scene, key=key) (OIData.with_model): the scene's prediction plus Gaussian noise from the template's errors, closure phases correlated through cp_noise, and draws of the template's gains and closure-phase offsets if it has them.

Parameters:

Name Type Description Default
null_scene SourceModel

The scene without a companion.

required
template OIData

Sampling and errors to simulate. Its values are not used.

required
error_scale float

The true noise as a multiple of the template's errors. The draws still carry the template's errors, as real data analysed with mis-estimated errors would, so error_scale ≠ 1 tests how the statistics fare when the errors are wrong. (To change the errors themselves, pass template.with_error_scale(...) or the result of rescale_errors.) It is a static Python float: a new value compiles anew.

1.0

Returns:

Type Description
callable

simulate(key, scene=None) -> OIData, observing scene (by default null_scene) with fresh noise for each key. It is an equinox Module, a pytree whose arrays jax.jit traces, and is traceable in key and scene, so it runs under jax.lax.map over keys.

Examples:

>>> simulate = gaussian_null(BinaryModelCartesian(0, 0, 0), template)
>>> data = simulate(jax.random.PRNGKey(0))

bootstrap_null(null_scene, data, *, method='sign_flip')

A simulator of the null hypothesis by residual bootstrap.

The residuals of data about null_scene's prediction are whitened, sign-flipped ("sign_flip", a wild bootstrap) or resampled with replacement ("resample"), re-coloured and added back to the prediction of the scene being simulated:

  • visibilities are independent per sample: whitened as r / σ;
  • unprojected phases are wrapped into [−π, π) first, as in OIData.residuals. Uncorrelated phases are whitened as Δ / σ, which equals the likelihood's chord 2 sin(Δ/2) / σ to O(Δ³); both are odd in Δ, so a sign flip flips the likelihood's whitened residual exactly;
  • closure phases from four or more telescopes are whitened with data.cp_noise (ClosureNoise.whiten) into independent combinations, which are flipped or resampled and re-coloured with ClosureNoise.colour, so the triangles' correlations survive;
  • projected phases (kernel or DISCO phases) are linear and not wrapped.

Sign flipping keeps each whitened residual's magnitude, so a heteroscedastic or mis-estimated error survives into every draw, and it removes any mean offset; it is the default. Resampling mixes the residuals of different samples (within the visibilities, and within the phases).

Parameters:

Name Type Description Default
null_scene SourceModel

The scene without a companion, fitted to data.

required
data OIData

The real data, concrete. Data with gains, closure-phase offsets or extra observables are not supported.

required
method (sign_flip, resample)

How to draw new whitened residuals.

"sign_flip"

Returns:

Type Description
callable

simulate(key, scene=None) -> OIData: scene's prediction (by default null_scene's) plus bootstrapped residuals, with data's errors. Like gaussian_null it is an equinox Module, traceable in key and scene.

Notes

Caveats:

  • It needs a null that fits: residuals from a poor fit carry the misfit into every draw.
  • A real companion in the data is part of the residuals, so it appears, scrambled, in every null draw and biases the null distribution upward (and the injections' too).
  • Only the closure-phase covariance model is kept: correlations between visibilities, between baselines beyond the closure triangles, or between frames are lost.
  • That covariance model is Kammerer et al.'s equal-baseline-noise approximation (see virgil._closure): with unequal errors within a group, the part of a real residual outside its column space is not recovered, and the re-coloured residuals are not exact closures of baseline phases.

rescale_errors(null_scene, data)

Scale the errors so that the null scene has χ²_r = 1.

The visibility errors and the phase errors are scaled separately, each by s = sqrt(χ² / n): χ² is the sum of squares of that block of whitened_residuals of null_scene (including the periodic penalty rows of correlated closure phases), and n its independent observables, as in n_independent (the independent closure-phase combinations, not the triangles). Without gains, the whitened residuals scale as 1 / s, so afterwards each block has χ² / n = 1 exactly; with gains, approximately.

Parameters:

Name Type Description Default
null_scene SourceModel

The scene without a companion, fitted to data.

required
data OIData

Data to rescale, concrete (not traced).

required

Returns:

Name Type Description
data OIData

A copy with d_vis and d_phi scaled. Extra observables (data.extras) keep their errors.

factors dict[str, float]

The scale factors, {"vis": s_vis, "phi": s_phi} (s_phi is 1 for data without phases).

Notes

n does not subtract the parameters fitted to obtain null_scene; with few data and several fitted parameters, the factors are biased low by about p / (2n).

injection_grid(separations, fluxes, n_pa, key)

Companions to inject: every separation and flux at random PAs.

Parameters:

Name Type Description Default
separations array - like

Separations in mas (non-negative).

required
fluxes array - like

Companion fluxes relative to the primary (non-negative; 0 gives null draws among the injections).

required
n_pa int

Number of position angles per (separation, flux), each drawn uniformly in [0°, 360°).

required
key Array or int

Random key (or integer seed) for the position angles.

required

Returns:

Type Description
dict[str, ndarray]

dra, ddec (mas) and flux, each of length len(separations) * len(fluxes) * n_pa, ordered by separation, then flux, then draw. Positions follow virgil's convention: PA from North through East, dra = sep sin PA (East positive), ddec = sep cos PA (North positive).

injection_recovery(model, null_scene, template, grid, key, *, n_null, injections=None, noise='gaussian', match_radius=None, flux_param=None, draw_batch=1, chunk_size=None, batch_size=None, progress=True)

Detection statistics of simulated null and injected observations.

Every draw simulates an observation (null, or with a companion injected), runs the companion search of detection_statistics on it, and keeps the statistics and the best position. One compiled kernel, (key, injection) -> statistics, serves every draw, null and injected, mapped with jax.lax.map(..., batch_size=draw_batch), so memory stays bounded (there is no vmap over grid × draws). The default grid batch_size is divided by draw_batch, and draw_batch is capped at that default, so the working set stays within the usual budget of one search whatever draw_batch is.

Parameters:

Name Type Description Default
model SourceModel or class

Companion model, as for detection_statistics: a template with dotted paths or a class called with grid's keys. Injections are this model at the injected coordinates and flux; null draws are this model at zero flux.

required
null_scene SourceModel

The scene without a companion. It must predict the same data as model with zero companion flux, the null hypothesis that the statistics test (checked on template).

required
template OIData

For noise="gaussian", the sampling and errors to simulate; for noise="bootstrap", the real data.

required
grid dict[str, array - like]

The search grid.

required
key Array or int

Random key, or an integer seed. Draw i of the null (or of the injections) uses fold_in(fold_in(key, 0 or 1), i), so results do not depend on draw_batch or chunk_size.

required
n_null int

Number of null draws (may be 0).

required
injections dict[str, array - like]

Companions to inject, e.g. from injection_grid: one array per grid key, found by its full name ("comp.dra") or its last part ("dra"), all of one length. None for no injections.

None
noise (gaussian, bootstrap)

Null noise model: gaussian_null or bootstrap_null of template and null_scene with their defaults, or a simulator built by either (for error_scale or method="resample") from template itself (checked by fingerprint), whose null scene must predict the same data as null_scene.

"gaussian"
match_radius float

In mas, finite and non-negative. When set, an injection counts as detected only if its best position lies within match_radius of the injected one. Needs coordinate keys ending in dra and ddec (Cartesian, mas) or sep and pa (mas and degrees, as in BinaryModelAngular). Stored in meta; the best positions are stored either way.

None
flux_param str

The flux key of grid, as for the grid tools.

None
draw_batch int

Draws evaluated together (vectorized) within jax.lax.map. When batch_size is omitted, it is capped at the default grid batch size, so that draw_batch searches of at least one grid point each never exceed the budget of one search.

1
chunk_size int

Draws per call of the compiled kernel, rounded up to a multiple of draw_batch. The null and the injected draws run in chunks of this size, with a progress bar over chunks. By default it is the smaller of 64 and the larger number of draws, and the last chunk of each run is exactly as long as the draws that remain (one extra compilation, and no wasted draws). If you set it, the last chunk is padded to chunk_size instead, so the compilation is reused across calls with different numbers of draws, at the cost of up to chunk_size - 1 discarded draws.

None
batch_size int

Grid points evaluated at once within each search, as for detection_statistics. An explicit value is used as given for each of the draw_batch searches evaluated together, so the working set is draw_batch times larger; the default is divided by draw_batch instead.

None
progress bool

Show a tqdm.auto progress bar, if tqdm is installed.

True

Returns:

Type Description
DetectionMC

The statistics of every draw, the injections, and metadata.

Examples:

>>> grid = {"dra": np.linspace(-150, 150, 16),
...         "ddec": np.linspace(-150, 150, 16),
...         "flux": np.geomspace(1e-4, 0.03, 32)}
>>> injections = injection_grid([60, 100], [1e-3, 3e-3], 100, 1)
>>> mc = injection_recovery(
...     BinaryModelCartesian, BinaryModelCartesian(0, 0, 0), template,
...     grid, 0, n_null=1000, injections=injections,
... )
>>> mc.threshold("delta_chi2", 1.35e-3)