Anomaly detection, visually

Which reference evidence should a frozen model retain, route, and trust?

I keep the representation fixed and study the decisions that determine which normal evidence reaches the final anomaly score. Each stage below is one of those decisions. Scroll to watch it happen.

Interactive scenes are 2-D schematics of the 768-dimensional computation; numbers update live from the rule shown.

01 Fixed representation · ProCon

An image becomes 784 tokens

A frozen DINOv2 ViT-B/14 turns a 392 × 392 image into a 28 × 28 grid of 768-dimensional patch tokens at several depths. Nothing is trained; everything downstream is a decision about how normal tokens are collected, kept, retrieved and compared.

F_{ℓ} (x) = f_{ℓ} (x) \in R^{P \times C}, P = 28 \cdot 28, C = 768, ℓ \in {- 3, - 6, - 8, - 9}

The hero animation above is this step: pixels → patch grid → tokens in feature space. Red tokens are the cells overlapping the ground-truth crack.

784 tokens · 0 overlap the annotated cut

02 Candidate evidence · eligibility · CLEANCON

Which training images may contribute at all

Before anything is stored, the candidate pool itself is a decision. CLEANCON scores every training image by how well the other training images explain it, using out-of-bag soft-projection residuals from twenty support banks, and passes only the lower-scoring half to the unchanged memory builder. The detector, the builder and the final memory size never change.

a_{i} = TopMean_{0.5%} (median_{b} r_{i, p, b}), T = {i : a_{i} \leq median_{j} a_{j}}

Three contaminated images sort to the top and the median gate drops them. Raise retention and one re-enters, yet in the paper P-AP kept rising past the cleanest point: purity alone does not order useful memories.

Retention R = 50%

03 Candidate evidence · missing support · BoundarySupport

Clean images miss the normal patches beside defects

Even a perfectly clean pool can be incomplete. Most patches of a defective image are normal, but a normal patch right beside a defect is encoded under a context no clean image provides, so its token has no reference. BoundarySupport creates that context on purpose: insert a synthetic defect into a clean image, exclude every token the insertion or the blending touched, and keep only the pixel-preserved ring as new normal evidence.

C = P (A) \cup P (D), R = Dilate (P (A), 2) ∖ C

Insertion A, blended pixels D, the excluded set C, and the preserved ring R. At a fixed memory budget on MVTec, the same ring around real defects recovers 94.7% of the oracle gain.

Step

04 Retain · CLEANCON

Keep a few normal tokens as references

Storing every candidate token is too slow, so PatchCore-style pipelines keep a coreset: greedy farthest-first selection that covers feature space with K centers. Coverage loves rare points. If a few anomalous images slipped past the gate, their rare tokens are exactly what the selector reaches for first.

μ_{t} = ar g q max s < t min ∥ q - μ_{s} ∥^{2}, A_{K} = \frac{p _{mem}}{p _{pool}}

Move the budget and switch the selector. Random selection keeps contamination at its pool rate (A_K ≈ 1); global farthest-first pulls it into memory many times over, as measured in CLEANCON (16 to 40× at a 1% budget).

Memory budget 2% (K = 16)

Selector

05 Route · GCR

Pick the head by geometry, not by anomaly score

When categories arrive one after another and test images carry no label, the detector must choose a category head before it can compare anything. Comparing anomaly scores across heads is unstable: scales differ and a defect can flip the choice. GCR routes with the mean nearest-prototype distance of a few sampled patches, then scores only inside that head.

r_{c} (x) = \frac{1}{M _{r}} p \in P_{r} \sum k min ∥ q_{p} - μ_{c, k} ∥^{2}, \overset{c}{^} = ar g c min r_{c} (x)

Thirty-two patches are sampled from a head-2 image with four defect patches. The mean criterion routes correctly; a peak-sensitive top-q criterion is dragged toward the broad neighboring head.

06 Compare and trust · ProCon

One near anchor is not enough

Nearest-neighbor scoring asks whether some normal anchor is close. A defective patch can accidentally land beside one stray anchor and score as normal: a false-normal match. ProCon instead asks whether the patch can be reconstructed from its local normal neighborhood, weighting k anchors by distance and scoring the projection residual.

w_{j} \propto exp (- ∥ z - m_{j} ∥^{2} / τ), \overset{z}{^} = j \sum w_{j} m_{j}, r = ∥ z - \overset{z}{^} ∥

(k = 1 \Rightarrow r = s_{NN})

Hover to move the test token. Press the preset to park it next to the stray anchor: the dashed hard-NN distance stays small while the residual jumps. Set k = 1 and the two rules coincide.

Neighbors k = 5

07 Consensus · ProCon

Agree across memories before trusting a peak

A single coreset depends on one particular anchor selection, and an unlucky selection can create spurious peaks. ProCon builds B seed-perturbed banks per depth and takes the median residual across them, then averages aligned residual maps across depths. Fusion happens after every bank and layer has produced the same quantity.

S_{ℓ} (p) = median_{b} R_{ℓ, b} (p), S_{map} (p) = \frac{1}{∣ L ∣} ℓ \sum S_{ℓ} (p)

Bank 2 carries a false peak. Min keeps it out but also erodes the true defect; mean lets it leak; median removes it while the shared defect survives.

Bank rule

08 Decide · StructCore

The maximum throws information away

The image-level decision is usually the largest map value. Two maps with the same maximum can have completely different structure: an isolated spurious peak on a normal image versus distributed, tail-heavy evidence on a defect. StructCore describes each map with three numbers, standardizes them on train-good maps, and reports the strongest calibrated violation.

ϕ (S) = [σ_{S}, TopMean_{r} (S), TV (S)], z = (ϕ - μ) / σ

D_{struct} = ∥ z ∥_{\infty}, S_{hyb} = S_{base} + λ D_{struct}

Both maps are scaled to the same maximum, so max pooling cannot separate them. Grow the defect extent and watch the structural score diverge while the maximum stays fixed.

Defect extent 3

09 What changed downstream?

What changed downstream?

Every stage above is one controlled intervention: the representation, memory budget, detector and readout stay fixed while a single decision changes. These are the consequences each paper measured.

CLEANCON02 · 04
Gate candidate images by out-of-bag support; keep the builder and the memory size.
memory contamination
before6.5%
after0.0%
Contamination in the deployed memory falls to zero and category-macro P-AP rises in all 12 matched comparisons. Yet the cleanest memory is not the best: R80 beats R50 in 12 of 12.
BoundarySupport03
Same budget, same detector, same encoder; only the identity of candidate patches changes.
P-AP
before73.3
after77.0
MVTec P-AP moves from 73.34 to 76.95, and patches within two cells of the defect recover 94.7% of that gain.
GCR05
Route by shared prototype geometry instead of comparing anomaly scores across heads.
forgetting (I-FM)
before0.010
after0.0
Forgetting drops to 0.000 on MVTec AD and VisA under task-agnostic continual evaluation, with no representation learning at all.
ProCon06 · 07
Replace the single nearest anchor with local reconstruction, then agree across banks and depths.
P-AP
before67.4
after73.0
Along the ablation ladder MVTec P-AP climbs from 67.4 to 73.0 at a fixed 1% coreset, and image AUROC never regresses.
StructCore08
Same anomaly map, different image-level readout.
image AUROC
before98.7
after99.6
Image AUROC rises from 98.7 to 99.6 on MVTec AD and from 97.6 to 98.4 on VisA while every localization metric stays identical.

One example on real defects

The ladder of stage 06 and 07 on MVTec AD

Real ProCon outputs, not renderings: hard nearest-neighbor memory, soft projection memory, and the full method with layer consensus on the same image.

bottle: Soft projection — Soft projection

Anomaly maps: ProCon paper, Figure 5 (MVTec AD examples).

One program, five decisions

Which normal evidence is available, which of it is kept, which reference system is consulted, how a reference is compared and trusted, and how patch evidence becomes a decision.

2026
CLEANCON
Does coverage-driven memory selection amplify sparse contamination, and does purity fix it?
Global coverage over-represents contamination 16 to 40 times. An out-of-bag gate removes it and helps, yet the cleanest memory is not the best one.
paper code
2026
BoundarySupport
Do clean images contain every normal patch a detector needs?
No. Normal patches beside real defects are missing from clean pools. Altering context around a clean image and keeping only pixel-preserved neighbors recovers them.
paper code
2026
GCR
How does a frozen detector pick the right head when categories keep arriving?
Route by mean nearest-prototype distance in the shared space, then score only inside the routed head. Routing instability, not forgetting, was the failure.
paper code
2026
ProCon
Is a single nearby anchor enough evidence of normality?
No. Soft local projection turns memory retrieval into decoder-free reconstruction; the residual, stabilized by bank and depth consensus, is the anomaly evidence.
paper code
2026
StructCore
Is the largest patch response the right image-level decision?
Not alone. A three-number structural descriptor, standardized on train-good maps and read with a Chebyshev rule, adds evidence max pooling discards.
paper code

Images: MVTec AD (Bergmann et al., CVPR 2019; CC BY-NC-SA 4.0) and figures from the author's ProCon repository. Interactive scenes are schematic and use synthetic 2-D data.