`virgil.crossval`

Provisional

May change in any 0.4.x release without a deprecation cycle.

K-fold cross-validation of a hyperparameter, generic over the data. See the cross-validation guide.

K-fold cross-validation of a hyperparameter, generic over the data.

Provisional

May change in any 0.4.x release without a deprecation cycle.

The unit of work is cv_task: fit with one hyperparameter setting on the training points of one fold, then score the held-out points. It returns a small JSON-friendly record, so the (setting, fold) pairs can run in one process (cross_validate) or as separate cluster array tasks whose records aggregate (or aggregate_files) combines.

The data protocol. The data enter only through make_term(mask), which returns a likelihood term for fit's likelihoods using the points where the boolean mask (length N) is true and ignoring the rest, without changing array shapes, so that fits for different masks share one compilation. The term is called on the fitted values and returns whitened residuals: m consecutive blocks of N point residuals (m = 2 for the real and imaginary parts of a visibility, m = 1 for a real datum), zero at masked points. If it has has_log_norm true, log_norm(values) is the normalization that fit adds to ½χ², of the unmasked points only. ComplexVisData.term satisfies this; so can a term for any other data.

The score of a fold is the negative log likelihood of its held-out points under the training fit, up to constants::

½ Σ r² + log_norm

summed over the held-out points, at the fitted values including the fitted error scales. The record also holds the raw χ² (Σ r² with every error scale set to 1, if unit_scale is given) and, if groups are given, the same sums per group.

`cv_task(setting, k, fold, *, fit_kwargs, make_term, init=None, groups=None, unit_scale=None, fit_ft=None, score_ft=None, dtype='float64', fit_options=None)`

Fit on the training points of fold k and score the held-out points.

Parameters:

Name	Type	Description	Default
`setting`	`float or JSON - friendly`	The hyperparameter value (e.g. a regularization weight λ).	required
`k`	`int`	The held-out fold label (a value in `fold`).	required
`fold`	`(N,) int array`	The fold label of every point, e.g. from `dartboard_folds`; the caller loops over folds and settings.	required
`fit_kwargs`	`callable`	`fit_kwargs(setting)` returns the mask-independent arguments of `fit` (`model`, `priors`, `data=()`, `regularizers=...`). Vary the setting through array leaves (a regularizer's weight, a prior's scale), not static structure, so every setting reuses one compilation.	required
`make_term`	`callable`	`make_term(mask)`: the likelihood term of the points where `mask` is true (fitted on the training mask, scored on the held-out one); pass `data=()` in `fit_kwargs`, or the held-out points leak into the fit. Same function for every task.	required
`init`	`dict`	Starting values for the fit, the same for every task.	`None`
`groups`	`(N,) int array`	A label per point (uv annulus, spectral window): the record breaks the held-out sums down by group.	`None`
`unit_scale`	`callable`	`unit_scale(values)` returns the fitted values with every error scale set to 1, for the raw χ²; `chi2` is `None` without it.	`None`
`fit_ft`	`dict`	`set_ft_defaults` settings in force while fitting and scoring (`chi2="auto"` terms stay direct unless you set `toeplitz_min_points`). Keep them the same in every task of a process: they are read when a function is first traced.	`None`
`score_ft`	`dict`	`set_ft_defaults` settings in force while fitting and scoring (`chi2="auto"` terms stay direct unless you set `toeplitz_min_points`). Keep them the same in every task of a process: they are read when a function is first traced.	`None`
`dtype`	`(float64, float32)`	Precision of the fit and the scoring.	`"float64"`
`fit_options`	`dict`	Further arguments of `fit` (`method`, `max_steps`, ...).	`None`

Returns:

Type	Description
`dict`	JSON-friendly record: `setting`, `fold`, `score` (−ln L of the held-out points), `chi2` (raw, s = 1), `n_test`, `n_train`, `loss`, `converged`, `steps`, `stop`, and with `groups` `group_labels`, `group_score`, `group_chi2`, `group_n`.

`cross_validate(settings, n_folds, fold, **task_kwargs)`

Run every (setting, fold) in this process and aggregate the records.

task_kwargs are those of cv_task after setting, k and fold. init may also be a function of the setting. aggregate options (larger_is_simpler) go in a aggregate_options dict.

Returns:

Type	Description
`dict`	`aggregate`'s summary, with the records under `"records"`.

`aggregate(records, larger_is_simpler=True)`

Combine the records of every (setting, fold) into the CV curve.

Parameters:

Name	Type	Description	Default
`records`	`sequence of dict`	Records of `cv_task` covering each (setting, fold) pair exactly once.	required
`larger_is_simpler`	`bool`	Whether a larger setting is the more regularized (the default, for a penalty weight λ), for the one-standard-error choice.	`True`

Returns:

Type Description

dict

settings (sorted when all numbers, else in order of appearance), folds; fold_scores of shape (n_settings, n_folds); score (the sum over folds), mean and sd (over folds, ddof = 1), se (sd/√n_folds); chi2, the raw χ² summed over folds (None if absent); best (index of the smallest mean; a setting whose mean is NaN ranks last, with a warning), one_se (index of the most regularized setting whose mean is within one se of the best's) and edge, true if either is the first or last of a numeric grid; and, with groups, groups (labels) and group_score, group_chi2 and group_n of shape (n_settings, n_groups) summed over folds, with group_best, the index of each group's smallest score (-1 for a group whose scores are all NaN). A group's curve can differ from the total: it shows which part of the data prefers which setting. Without a numeric grid, one_se and edge are None.

Raises:

Type	Description
`ValueError`	If there are no records, records are missing or duplicated, or every setting has a NaN score.

`aggregate_files(paths, **options)`

aggregate the records in JSON files.

`fold_masks(fold, k)`

Training and held-out masks of fold k: fold != k and fold == k.

fold is the integer fold of every point (e.g. from dartboard_folds). The masks are disjoint and together cover all points. Raises if fold k is empty.

`save_record(record, path)`

Write one task's record as JSON.

`load_records(paths)`

Read records from JSON files holding a record or a list of them.