API reference¶
Generated from the docstrings.
Pipelines¶
imgad.pipeline.AnomalyDetectionPipeline ¶
AnomalyDetectionPipeline(extractor: Union[FeatureExtractor, str, Dict[str, Any]], detector: Union[BaseDetector, str, Dict[str, Any]] = 'knn', preprocessing: Optional[Iterable[Step]] = None, cache_dir: Optional[Union[str, PathLike]] = None)
Bases: ThresholdMixin
Parameters¶
extractor: a :class:FeatureExtractor, a HF model id, or a config dict {"type": "hf", ...}
detector: a :class:BaseDetector, a detector name, or a spec dict {"name": "knn", "k": 1}
preprocessing: list of steps (see :mod:imgad.preprocessing), e.g. ["l2", "pca:0.95"]
cache_dir: optional directory to cache per-image features (only for file-path inputs)
fit ¶
Fit on normal images only. With a zero-shot detector images may be empty ([]: zero-shot,
text prompts only) or a few normal images (few-shot).
save ¶
Save detector + preprocessing + extractor config (model weights are re-loaded from the Hub).
load
classmethod
¶
load(path: Union[str, PathLike], device: Optional[str] = None, batch_size: Optional[int] = None, cache_dir: Optional[Union[str, PathLike]] = None, token=None) -> 'AnomalyDetectionPipeline'
Load a trained pipeline. device/batch_size/cache_dir override the saved values
(e.g. a model trained on GPU can be used on CPU). Feature caching is off at inference unless
cache_dir is given. token: Hugging Face access token if the extractor is a gated/private
model (otherwise HF_TOKEN or the stored login are used; tokens are never saved in the file).
Files saved by :class:~imgad.anomalib_backend.AnomalibPipeline are also accepted (an
AnomalibPipeline is returned), so every loader in imgad works with both kinds.
imgad.anomalib_backend.AnomalibPipeline ¶
AnomalibPipeline(model: str = 'patchcore', image_size: Union[int, Tuple[int, int]] = 256, calibration_split: float = 0.2, threshold_quantile: float = 0.95, batch_size: int = 32, train_batch_size: Optional[int] = None, num_workers: int = 0, device: Optional[str] = None, max_epochs: Optional[int] = None, seed: Optional[int] = 0, trainer_kwargs: Optional[Dict[str, Any]] = None, intensity: str = 'auto', **model_kwargs)
Bases: ThresholdMixin
Anomalib model wrapped with imgad's API (fit / score / predict / save / load).
Parameters¶
model: anomalib model name: "patchcore", "padim", "efficient_ad",
"fastflow", "stfpm", "reverse_distillation", ... (one-class models)
image_size: model input size (int or (h, w)); images are resized (no crop), so maps
stay aligned with the full image
calibration_split: fraction of the normal training images held out (not used for training)
to set the image/pixel thresholds and the normal_percentile reference.
0 -> use in-sample training scores (optimistic, e.g. for PatchCore).
threshold_quantile: quantile of the held-out normal scores used as image threshold (0.95 ->
~5% false alarms). The pixel threshold is the same quantile of the
per-image maximum of the held-out normal maps.
batch_size / train_batch_size: inference / training batch sizes (efficient_ad trains with 1)
max_epochs: epochs for trainable models (ignored by PatchCore/PaDiM, which use 1)
trainer_kwargs: extra lightning.Trainer arguments (e.g. {"max_steps": 1000})
intensity: rule mapping images with more than 8 bits (16-bit, DICOM...) to 8 bits; see
:mod:imgad.imageio
**model_kwargs: passed to the anomalib model, e.g. backbone="resnet18",
layers=["layer2", "layer3"], coreset_sampling_ratio=0.01
fit ¶
Train on normal images only, then set thresholds on the held-out calibration split.
score ¶
Image-level anomaly scores (higher = more anomalous).
score_and_maps ¶
(scores, maps). size: None = model resolution, "original" = each image's own
size, or (h, w). Maps are raw (unnormalised) float arrays; higher = more anomalous.
predict_masks ¶
Binary defect masks: map > pixel_threshold_.
calibrate_pixel_threshold ¶
calibrate_pixel_threshold(images=None, masks: Optional[Sequence[MaskLike]] = None, maps: Optional[Sequence[ndarray]] = None, max_pixels: int = 5000000, show_progress: bool = True) -> Dict[str, float]
Set pixel_threshold_ to maximise pixel-level F1 on a validation set with ground-truth
masks (None = defect-free image). Pass images or precomputed maps.
save ¶
Save config + trained weights (state_dict, incl. e.g. the PatchCore memory bank) + thresholds. Unlike feature pipelines, the file contains the model weights: no download is needed to load it.
imgad.pipeline.load_pipeline ¶
load_pipeline(path: Union[str, PathLike], device: Optional[str] = None, batch_size: Optional[int] = None, token=None)
Load any saved imgad pipeline (feature + detector, or anomalib).
Inference¶
imgad.inference.AnomalyPredictor ¶
Thin inference wrapper around a trained :class:AnomalyDetectionPipeline or
:class:~imgad.anomalib_backend.AnomalibPipeline.
load
classmethod
¶
load(path: Union[str, PathLike], device: Optional[str] = None, threshold: Optional[float] = None, batch_size: Optional[int] = None, token=None) -> 'AnomalyPredictor'
token: Hugging Face access token for gated/private extractors (never stored).
predict ¶
predict(images: Union[ImageLike, Sequence[ImageLike]], recursive: bool = True, show_progress: bool = True, return_maps: bool = False) -> List[Prediction]
Predict on a path to an image or folder, a PIL image / numpy array, or a list of any of these.
return_maps=True (anomalib models) adds each image's anomaly map at its original size.
predict_iter ¶
predict_iter(images: Union[ImageLike, Sequence[ImageLike]], chunk_size: int = 256, recursive: bool = True, return_maps: bool = False) -> Iterator[Prediction]
Stream predictions chunk by chunk (constant memory for very large folders).
save_maps ¶
save_maps(results: Iterable[Prediction], out_dir: Union[str, PathLike], alpha: float = 0.5) -> List[Prediction]
Write, for each prediction with a map, <name>_heatmap.png (overlay on the image when the
source is a file) and <name>_mask.png (pixels above the pixel threshold). Fills
r.heatmap / r.mask so :meth:save_results lists the files.
save_results
staticmethod
¶
Write predictions to .csv or .json (chosen by extension).
imgad.inference.Prediction
dataclass
¶
Prediction(source: str, score: float, is_anomaly: bool, threshold: float, normal_percentile: float, anomaly_map: Optional[ndarray] = None, anomalous_area: Optional[float] = None, heatmap: Optional[str] = None, mask: Optional[str] = None)
Result for one image.
score: raw anomaly score (higher = more anomalous; scale depends on the detector)
is_anomaly: score > threshold
threshold: decision threshold used
normal_percentile: fraction of normal training images with a lower score (0-1). Comparable
across detectors: 0.5 ~ typical normal image, 0.999 ~ extremely unusual.
anomaly_map: (return_maps=True, anomalib models) float map, higher = more anomalous
anomalous_area: fraction of pixels above the pixel threshold (only with maps)
heatmap / mask: files written by :meth:AnomalyPredictor.save_maps
to_dict ¶
Serialisable fields (the map array is left out; unset optional fields are omitted).
Feature extractors¶
imgad.features.base.FeatureExtractor ¶
Bases: ABC
Maps a list of images to an (n_images, dim) float32 array.
Subclasses implement :meth:_extract_batch (list of PIL RGB images -> np.ndarray)
and :meth:get_config (JSON-serialisable kwargs to rebuild the extractor).
set_intensity ¶
Rule mapping images with more than 8 bits to 8 bits: "auto", "minmax", "dicom",
"percentile:1,99" or "window:lo,hi" (see :mod:imgad.imageio).
full_config ¶
{"type": ..., **get_config()} plus the intensity rule when it is not the default; this is what
saved pipelines store and :func:build_extractor accepts.
extract ¶
extract(images: Sequence[ImageLike] | ImageLike, cache_dir: Optional[str | PathLike] = None, show_progress: bool = True) -> np.ndarray
Extract features. If cache_dir is set and all inputs are file paths,
features are cached per image (keyed by extractor id + path + mtime + size).
cached_count ¶
How many of images already have cached features (0 without a cache or for in-memory images).
imgad.features.huggingface.HFFeatureExtractor ¶
HFFeatureExtractor(model_name: str, pooling: str = 'auto', layer: Optional[Union[int, Sequence[int]]] = None, normalize: bool = True, device: Optional[str] = None, batch_size: int = 32, dtype: Optional[str] = None, revision: Optional[str] = None, trust_remote_code: bool = False, image_size: Optional[int] = None, hf_kwargs: Optional[Dict[str, Any]] = None, token: Union[None, bool, str] = None)
Bases: FeatureExtractor
imgad.features.open_clip.OpenCLIPFeatureExtractor ¶
OpenCLIPFeatureExtractor(model_name: str, pretrained: Optional[str] = None, normalize: bool = True, device: Optional[str] = None, batch_size: int = 32, token: Union[None, bool, str] = None)
Bases: FeatureExtractor
encode_text ¶
L2-normalised text embeddings in the same space as the image features.
imgad.features.timm_models.TimmFeatureExtractor ¶
TimmFeatureExtractor(model_name: str, pooling: str = 'auto', timm_kwargs: Optional[Dict[str, Any]] = None, pretrained: bool = True, normalize: bool = True, device: Optional[str] = None, batch_size: int = 32, token: Union[None, bool, str] = None, checkpoint: Optional[str] = None, checkpoint_key: Optional[str] = None, checkpoint_prefix: Optional[str] = None, data_config: Optional[Dict[str, Any]] = None)
Bases: FeatureExtractor
imgad.features.retina_clip.RetinaCLIPExtractor ¶
RetinaCLIPExtractor(model: str = 'flair', device: Optional[str] = None, batch_size: int = 16, checkpoint: Optional[str] = None)
Bases: FeatureExtractor
imgad.features.timm_models.load_checkpoint ¶
load_checkpoint(model, checkpoint: str, key: Optional[str] = None, prefix: Optional[str] = None, token: Union[None, bool, str] = None, state: Optional[Dict[str, Any]] = None) -> List[str]
Load checkpoint into a timm model (non-strict, see the module docs); returns the model keys that
stayed uninitialised. Raises if almost nothing matched (wrong architecture, key or prefix). state: the
already-read tensors of checkpoint (to avoid reading a large file twice).
imgad.features.timm_models.checkpoint_file ¶
Local path of a checkpoint given as hf-hub:org/repo/file, an https:// URL, gdrive:<id>/<name>
or a path (downloads are cached).
imgad.features.base.build_extractor ¶
Build an extractor from {"type": "hf", "model_name": ...}, a bare HF model id, or a curated model
"zoo:<name>" (see :mod:imgad.zoo). The optional
key "intensity" sets the rule for images with more than 8 bits (see :mod:imgad.imageio).
token: Hugging Face access token for gated/private models (passed only to extractors that
accept one; see :mod:imgad.hub).
Medical model zoo¶
imgad.zoo ¶
Curated feature extractors for medical imaging, with recommendations per modality.
from imgad import zoo
zoo.recommend("chest-xray") # ranked ModelCards (domain models, then general baselines)
zoo.recommend("histopathology", commercial=True, public_only=True)
build_extractor("zoo:rad-dino") # any card by name, also in benchmark YAML files
imgad models --modality histopathology --check --yaml
imgad models -m fundus --zero-shot # vision–language models for zero-shot detection
Every card records how to load the model (an extractor spec), its access (public / gated), license and intended use. These facts come from the model cards on the Hugging Face Hub; verify them on the model page before relying on them, especially for any use beyond research.
MODALITIES
module-attribute
¶
MODALITIES: Dict[str, str] = {'chest-xray': 'Chest radiographs', 'ct': 'Computed tomography (2-D slices)', 'mri': 'Magnetic resonance (2-D slices)', 'histopathology': 'H&E (and IHC) histology tiles', 'dermatology': 'Dermoscopic and clinical skin images', 'ophthalmology': 'Retinal colour fundus photographs (CFP)', 'oct': 'Retinal optical coherence tomography (B-scans)'}
ModelCard
dataclass
¶
ModelCard(name: str, spec: Dict[str, Any], modalities: Tuple[str, ...], hub_id: str, access: str, license: str, use: str, dim: Optional[int] = None, intensity: str = 'auto', notes: str = '', reference: str = '', general: bool = False, text: bool = False)
recommend ¶
recommend(modality_name: Optional[str] = None, include_general: bool = True, public_only: bool = False, commercial: bool = False, zero_shot: bool = False) -> List[ModelCard]
Models to benchmark for a modality, most specific first: domain models trained on that modality (those covering fewer modalities first), then general-purpose baselines.
public_only drops gated models; commercial drops models whose license or terms are
non-commercial / research-only; zero_shot keeps only vision–language models (card.text) for the
zero-shot detector. Without a modality, every card is returned (filters still apply).
A recommendation is a starting point to benchmark on your own data, not a ranking of quality.
benchmark_yaml ¶
extractors: block for a benchmark YAML with these models (intensity rules included).
Detectors¶
imgad.detectors.base.BaseDetector ¶
Bases: ABC
Base class for all detectors.
Parameters¶
threshold_quantile:
After fitting, the decision threshold is set to this quantile of the out-of-sample
training scores; predict flags samples above it.
E.g. 0.95 -> ~5% false-positive rate on normal data.
Calibration (class attributes, can be overridden per instance):
calibration = "cv" -> out-of-fold scores via K-fold (n_calibration_splits), unless the
detector provides unbiased leave-one-out scores (kNN, LOF).
calibration = "train"-> in-sample scores (fast, but optimistic for KDE/OCSVM/PCA...).
predict ¶
1 = anomaly, 0 = normal.
imgad.detectors.base.build_detector ¶
build_detector("knn", k=5) or build_detector({"name": "knn", "k": 5}).
imgad.detectors.zero_shot.ZeroShotDetector ¶
ZeroShotDetector(modality: Optional[str] = None, normal: Optional[Sequence[str]] = None, anomalous: Optional[Sequence[str]] = None, templates: Optional[Sequence[str]] = None, mode: str = 'auto', visual_weight: float = 0.5, k: int = 1, temperature: Optional[float] = None, threshold_quantile: float = 0.95)
Bases: BaseDetector
Prompt-based zero/few-shot detector (see the module docs).
modality: prompt set from :mod:imgad.prompts ("fundus", "oct", "cxr"...; None = generic).
normal / anomalous: states that replace the modality's defaults (["healthy"], ["glaucoma", ...]).
templates: replace the modality's templates (each with one {}).
mode, visual_weight, k: see the module docs. temperature: overrides the model's logit scale.
imgad.prompts.default_prompts ¶
{"templates": [...], "normal": [...], "anomalous": [...]} for a modality (zoo names and aliases,
e.g. "fundus", "oct", "cxr") or for natural/industrial images (None).
Data and image reading¶
imgad.data ¶
Dataset helpers. Datasets are represented as plain lists of image paths + labels (0 = normal, 1 = anomaly) so they plug directly into extractors and pipelines.
AnomalyDataset
dataclass
¶
AnomalyDataset(train_paths: List[str], test_paths: List[str], test_labels: ndarray, test_types: List[str] = list(), name: str = 'dataset', test_masks: List[Optional[str]] = list())
Train (normal only) / test (normal + anomalous) split.
mvtec_ad ¶
MVTec AD / MVTec LOCO / VisA-in-MVTec-format layout::
root/
from_folders ¶
from_folders(train_normal: PathLike, test_normal: PathLike, test_anomalous: Union[PathLike, Sequence[PathLike]], name: str = 'folders') -> AnomalyDataset
Generic layout: one folder of normal training images, one of normal test images and one or more folders of anomalous test images.
one_vs_rest ¶
one_vs_rest(root: PathLike, normal_class: str, train_fraction: float = 0.8, max_anomalies: Optional[int] = None, seed: int = 0) -> AnomalyDataset
Semantic AD from a classification folder (root/<class>/*.jpg): one class is normal,
all others are anomalies. Normal images are split train/test.
volume_slices ¶
volume_slices(volumes: Union[PathLike, Sequence[PathLike]], axis: str = 'z', step: int = 1, middle_only: bool = False, skip_empty: bool = True) -> List[str]
Slice references ("vol.nii.gz::z=42") for NIfTI volumes or multi-frame DICOM files; every imgad
function accepts them as image paths. volumes: files and/or folders (searched recursively).
axis x / y / z (NIfTI, RAS orientation; ignored for DICOM, which uses its frames), step keeps
every n-th slice, middle_only one slice per volume, skip_empty drops constant slices (e.g. the
background around a skull-stripped brain).
Scores are per slice: aggregate them per volume (e.g. the maximum) for a volume-level decision.
list_images ¶
list_images(root: PathLike, recursive: bool = True, extensions: Sequence[str] = IMG_EXTENSIONS) -> List[str]
imgad.imageio ¶
Image reading for 8-bit, high-bit-depth (16-bit PNG/TIFF, float) and medical images (DICOM, NIfTI).
Every model in imgad receives 8-bit RGB images. Images with more than 8 bits (X-ray, CT, MRI, 16-bit industrial cameras...) are mapped to 8 bits with an intensity rule:
======================== =========================================================================
"auto" (default) 8-bit images unchanged; DICOM with a window -> that window; anything else
with more than 8 bits -> per-image percentiles 0.5-99.5
"minmax" per-image minimum and maximum
"percentile:1,99" per-image percentiles
"window:-1000,400" fixed range (e.g. CT Hounsfield units, or window:0,4095 for a 12-bit
detector). The same mapping for every image: recommended for anomaly
detection, since per-image rules hide global intensity anomalies
"dicom" the DICOM WindowCenter/WindowWidth of each file (else "auto")
======================== =========================================================================
Volumes (NIfTI .nii / .nii.gz, multi-frame DICOM) are read slice by slice through references
"<path>::<axis>=<index>" (axis x/y/z for NIfTI, frame for DICOM), e.g. "ct.nii.gz::z=42";
see :func:imgad.data.volume_slices. Only pixel data is read: DICOM/NIfTI metadata (patient name, IDs,
dates...) is never stored or logged.
DICOM needs pydicom and NIfTI nibabel (pip install "imgad[medical]").
read_image ¶
RGB 8-bit PIL image from a path (raster, DICOM, NIfTI slice reference), a PIL image or an array.
to_uint8 ¶
to_uint8(arr: ndarray, spec: IntensityLike = 'auto', dicom_window: Optional[Tuple[float, float]] = None) -> np.ndarray
Map an image array (any dtype; HxW, HxWxC) to uint8 with the given intensity rule.
parse_intensity ¶
Normalise an intensity rule: "auto", "minmax", "dicom", "percentile:1,99",
"window:lo,hi" or the dict forms {"percentile": [1, 99]} / {"window": [lo, hi]}.
split_ref ¶
"ct.nii.gz::z=42" -> ("ct.nii.gz", "z", 42); plain paths -> (path, None, None).
Evaluation and benchmarks¶
imgad.evaluation ¶
Metrics and a benchmarking helper that reuses extracted features across detectors.
benchmark ¶
benchmark(extractors: Iterable[Union[FeatureExtractor, str, Dict[str, Any]]] = (), detectors: Iterable[Union[BaseDetector, str, Dict[str, Any]]] = (), datasets: Iterable = (), preprocessings: Iterable[Optional[Sequence]] = (None,), cache_dir: Optional[str] = None, verbose: bool = True, anomalib_models: Iterable[Union[str, Dict[str, Any]]] = (), seeds: Sequence[int] = (0,), train_subsample: Optional[Union[float, int]] = None, bootstrap: int = 0, ci_alpha: float = 0.05, keep_scores: bool = False, token=None, check_hub_access: bool = True, intensity: Optional[str] = None, n_jobs: int = 1, devices: Union[None, str, Sequence[str]] = None, gpu_jobs: Union[int, str] = 'auto', gpu_mem_gb: Optional[float] = None, k_shots: Optional[Sequence[int]] = None, on_error: str = 'continue') -> List[Dict[str, Any]]
Grid evaluation: extractor x preprocessing x detector x dataset, plus each anomalib model x
dataset (anomalib_models: names or {"name": "patchcore", "backbone": ...} specs).
Features are extracted once per (extractor, dataset) and reused for all detectors and seeds.
Anomalib rows also get pixel-level metrics (see :func:evaluate_maps) when the dataset has masks.
Repetitions and uncertainty:
* seeds: every configuration is run once per seed (one row each, column seed). The seed
sets the detectors' random_state (unless fixed in the spec), the calibration folds, the
anomalib seed and the training subset.
* train_subsample: fraction (float) or number (int) of normal training images drawn per
seed. Without it, deterministic detectors (kNN, Mahalanobis...) give the same result for
every seed; with it, the spread over seeds measures sensitivity to the training set.
* bootstrap: if > 0, adds <metric>_ci_low/high for AUROC and AUPR from that many
stratified bootstrap resamples of the test set (ci_alpha = 0.05 -> 95% CI).
* Progress (plan, current phase, one line per finished run with counter, elapsed time and ETA)
is logged under the imgad logger; verbose=True shows it on the console unless logging is
already configured (see :func:imgad.progress.setup_logging) and enables progress bars.
* keep_scores: store per-image scores and labels in each row (needed for paired tests
such as :func:imgad.stats.delong_vs_baseline).
* k_shots: few-shot protocol, e.g. [0, 1, 2, 4, 8, 16]. Every configuration is run with k
normal training images drawn per seed (column k_shot; the configuration name ends in
| k-shot), instead of the whole training set or train_subsample. k = 0 is zero-shot and runs
only detectors that need no training images (zero_shot); detectors that cannot fit on k
images are recorded as failed runs.
* intensity: rule mapping images with more than 8 bits (16-bit, DICOM, NIfTI) to 8 bits for every
extractor and Anomalib model that does not set its own (see :mod:imgad.imageio).
Gated / private Hub models: token is passed to the extractors (otherwise HF_TOKEN or the
stored login are used; see :mod:imgad.hub). With check_hub_access (default) access to every
Hub model is checked before any work starts, and a :class:imgad.hub.GatedModelError lists what
to do for each model that cannot be loaded.
Parallel execution (see :mod:imgad.parallel; results are identical to the sequential run):
* n_jobs: CPU worker processes for the detector runs (-1 = all cores). 1 together with
devices=None runs everything sequentially in this process.
* devices: where feature extraction and Anomalib models run: "auto" (every GPU, or the
CPU), "cpu", "cuda:0,cuda:1" or a list. Each extractor is loaded once in a worker
process and its detector runs start on the CPU pool as soon as its features are ready.
* gpu_jobs: maximum concurrent jobs per GPU ("auto": as many as fit in memory, up to 4).
* gpu_mem_gb: memory assumed for a job whose needs are unknown (default: estimated from the
checkpoint size for Hub models, 4 GiB otherwise); after the first run of a model its measured
peak is used. A job starts only when it fits in the free memory of a device.
Zero-shot detectors (:mod:imgad.detectors.zero_shot) run only with extractors that encode text
(vision–language models) and without preprocessing that changes the embeddings (None or l2);
other combinations are skipped. The prompts are encoded once per extractor.
Failures: with on_error="continue" (default) a run that fails (loading or extracting with a model,
fitting a detector, an Anomalib model...) is logged, recorded as a row with an error column and no
metrics, and the benchmark goes on with the rest; "raise" stops at the first failure.
:func:successful_rows drops failed rows; :mod:imgad.stats and the report ignore them.
Returns a list of result rows (pandas.DataFrame(rows) for a table); analyse them with
:mod:imgad.stats.
successful_rows ¶
Rows of runs that finished (without an error).
evaluate_scores ¶
Image-level metrics. y_true: 1 = anomaly. scores: higher = more anomalous.
evaluate_maps ¶
Pixel-level metrics. masks: ground-truth masks (paths, arrays or None for defect-free
images); maps: anomaly maps (higher = more anomalous). Masks are resized (nearest) to each
map's resolution, so evaluating at model resolution keeps memory low.
Returns pixel AUROC, pixel AUPR, pixel F1-max and AUPRO (normalised area under the
per-region-overlap curve up to max_fpr, the standard MVTec AD localisation metric).
aupro ¶
aupro(masks: Sequence[ndarray], maps: Sequence[ndarray], max_fpr: float = 0.3, n_thresholds: int = 500) -> float
Area under the PRO curve (mean overlap per connected defect region vs. pixel FPR) for
FPR in [0, max_fpr], normalised to [0, 1]. masks are boolean arrays shaped like maps.
per_type_auroc ¶
AUROC of each anomaly type vs. all normal test samples.
imgad.stats ¶
Statistics for comparing configurations: bootstrap confidence intervals, paired tests on one test set (DeLong, paired bootstrap) and tests across datasets (Friedman + Nemenyi, Wilcoxon-Holm).
Typical use, after rows = benchmark(..., seeds=[0, 1, 2], bootstrap=1000, keep_scores=True)::
from imgad import stats
stats.summarize_seeds(rows) # mean, std and 95% CI over seeds
res = stats.compare_configs(rows, metric="auroc") # ranks, Friedman, Nemenyi CD, Wilcoxon-Holm
print(stats.report_markdown(res))
stats.plot_cd_diagram(res, "cd.png")
stats.delong_vs_baseline(rows, baseline=res["configs"][0]) # per-dataset paired AUROC tests
Rows are the dicts returned by :func:imgad.evaluation.benchmark (or read back from its CSV with
:func:load_results). Configurations are identified by the config column
("extractor | preprocessing | detector") and datasets by dataset.
bootstrap_ci ¶
bootstrap_ci(y_true, scores, metrics: Sequence[str] = ('auroc', 'aupr'), n_boot: int = 1000, alpha: float = 0.05, seed: int = 0) -> Dict[str, Dict[str, float]]
Percentile bootstrap CI over test images (stratified by class).
Returns {metric: {"value", "low", "high", "se"}}. It reflects test-set sampling variability
only; variability due to training (seeds / training subsets) is measured by repeating runs.
delong_test ¶
DeLong test for two correlated AUROCs (same test images, two models).
Fast O(n log n) implementation (Sun & Xu, 2014). Returns both AUROCs, their difference
(a - b) with a 1 - alpha CI, the z statistic and the two-sided p-value.
paired_bootstrap_test ¶
paired_bootstrap_test(y_true, scores_a, scores_b, metric: str = 'aupr', n_boot: int = 2000, alpha: float = 0.05, seed: int = 0) -> Dict[str, float]
Paired bootstrap for any metric (e.g. AUPR, where DeLong does not apply): both models are
evaluated on the same resampled test images. Two-sided p-value for metric(a) == metric(b).
summarize_seeds ¶
summarize_seeds(rows: List[Dict[str, Any]], metrics: Sequence[str] = ('auroc', 'aupr'), alpha: float = 0.05) -> List[Dict[str, Any]]
Aggregate repeated runs: one row per (dataset, config) with mean, std, number of seeds and a
t-based 1 - alpha CI of the mean for each metric. Adds a dataset="mean" row per config
with the across-dataset mean of the seed means. Failed runs (rows with an error) are ignored.
compare_configs ¶
compare_configs(rows: List[Dict[str, Any]], metric: str = 'auroc', alpha: float = 0.05) -> Dict[str, Any]
Compare configurations across datasets (each dataset = one block), following Demšar (2006):
- average ranks (1 = best; ties get the average rank),
- Friedman test (+ Iman-Davenport F correction) of "all configs perform equally",
- Nemenyi post-hoc: critical difference (CD) and pairwise p-values,
- pairwise Wilcoxon signed-rank tests with Holm correction (more powerful with few configs).
Failed runs (rows with an error) are ignored.
Needs >= 2 datasets; Friedman/Nemenyi need >= 3 configs. Seeds are averaged per dataset first.
delong_vs_baseline ¶
delong_vs_baseline(rows: List[Dict[str, Any]], baseline: str, alpha: float = 0.05) -> List[Dict[str, Any]]
Per (dataset, seed): DeLong test of every config against baseline (needs rows produced
with keep_scores=True). p-values are Holm-adjusted within each dataset and seed.
plot_cd_diagram ¶
Critical-difference diagram (Demšar 2006): average ranks on an axis; configs joined by a bar are not significantly different (Nemenyi). Needs matplotlib. Returns the figure.
report_markdown ¶
Markdown summary of :func:compare_configs.
imgad.report.generate_report ¶
generate_report(rows: List[Dict[str, Any]], datasets: Sequence = (), out_dir='report', config: Optional[Dict[str, Any]] = None, title: Optional[str] = None, metric: str = 'auroc', alpha: float = 0.05, command: Optional[str] = None, figures: bool = True) -> Path
Write out_dir/report.md (+ figures, CSV and config). Returns the path of the markdown file.
rows: output of :func:imgad.evaluation.benchmark (ideally with keep_scores=True).
datasets: the :class:~imgad.data.AnomalyDataset objects used (for the dataset analysis,
per-defect-type results and example images); can be empty.
Hugging Face access¶
imgad.hub ¶
Access to gated / private models on the Hugging Face Hub.
Some models (e.g. DINOv3, some SigLIP2 or medical checkpoints) are gated: you must accept their conditions on the model page and authenticate with an access token. imgad looks for a token, in order:
- an explicit
token=argument (HFFeatureExtractor(..., token=...),load(..., token=...),--hf-token-filein the CLI), - the
HF_TOKENenvironment variable, - the token stored by
imgad hf-login/hf auth login(~/.cache/huggingface/token).
The token is never written to saved pipelines, configs, feature-cache keys, reports or logs.
imgad hf-login # stores a token (input hidden)
imgad hf-check facebook/dinov3-vitb16-pretrain-lvd1689m
imgad hf-check --config configs/mvtec_benchmark.yaml # every Hub model of a benchmark
AccessStatus
dataclass
¶
AccessStatus(repo_id: str, ok: bool, state: str, gated: Optional[str] = None, private: Optional[bool] = None, cached: bool = False, user: Optional[str] = None, token_source: str = 'none', detail: str = '', license: Optional[str] = None)
GatedModelError ¶
Bases: RuntimeError
A Hub model cannot be accessed (gated, private, wrong token...). The message says what to do.
check_access ¶
Can this model be downloaded/loaded with the current credentials (and under which license, when the model card declares it)? Never raises.
check_models ¶
Access status of every Hub model in names (non-Hub names are skipped).
login ¶
Store a token like hf auth login (asks for it with hidden input if not given) and return the
user name. Does not touch git credentials.
read_token_file ¶
Read a token from a file (first non-empty line). Warns if other users can read the file.
Devices and progress¶
imgad.devices ¶
Device discovery, memory checks and GPU memory admission for parallel benchmarks.
The parent process of a parallel benchmark never creates CUDA contexts: GPU memory is queried with
nvidia-smi (or NVML through pynvml when installed), and torch.cuda is only used as a last
resort. Jobs reserve an estimated amount of memory on a device and only start when it fits both in
the budget (free memory at start × safety, minus what running jobs reserved) and in the memory
that is free right now (other processes may have started meanwhile).
imgad devices # what the benchmark would see
ResourceManager ¶
ResourceManager(devices: Sequence[DeviceInfo], max_jobs: Union[int, str] = 'auto', safety: float = 0.9, margin_bytes: int = int(0.3 * GiB), auto_cap: int = 4, refresh_s: float = 1.0, query=None)
Decides on which device (if any) a job may start now.
A job asks for est_bytes. On a CUDA device it starts only if
est_bytes <= min(budget - reserved, free_now - margin), where budget is the free memory
when the benchmark started times safety, reserved what running jobs asked for, and
free_now a fresh query (refreshed at most every refresh_s seconds). Devices without
memory information (CPU, MPS) only limit the number of concurrent jobs.
list_devices ¶
All usable devices: CUDA GPUs (with total/free memory), Apple MPS, and the CPU.
resolve_devices ¶
"auto" -> every GPU (or the CPU if there is none); "cpu"; "cuda:0,cuda:1" or a list.
Raises if a requested device does not exist.
estimate_extractor_bytes ¶
GPU memory needed to run a feature extractor: weights (size of the checkpoint files on the Hub or
in the local HF cache) + activations for one batch + the CUDA context. Falls back to
default_bytes when the model is not a Hub model or the size cannot be found.
imgad.progress.setup_logging ¶
setup_logging(level: int = logging.INFO, log_file: Optional[str] = None, fmt: str = '%(asctime)s %(message)s') -> None
Send imgad logs to the console (level) and optionally to log_file (always DEBUG,
with the logger name). Calling it again replaces the previous handlers.