Benchmark route

Score distributions

Every table cell is a mean over images. These are the distributions those means come from, so you can see whether a number describes a typical image or is an artefact of a lopsided tail.

Each row is one model on one route: a histogram of its per-image score in 20 bins, with the mean marked by a solid line and the median by a dotted one. The gap between them is the whole point. A mean below the median means a minority of bad images is dragging the average down and the model is typically better than its headline number. A mean above the median means the opposite: most images are worse than the average, which is being propped up by a minority that work.

The tails run in opposite directions, and it changes how three rows should be read

Detecture's mean understates it on every route. Mean minus median is negative everywhere — −0.086 on CSTD, where the median image scores 0.969 against a mean of 0.884. A small tail of genuinely bad images pulls the average down; the typical image is markedly better than the table suggests.

Sa2VA is the reverse, and severely so. On RWTD-COCO its mean is 0.240 but its median is exactly 0.000: it returns nothing at all on 63% of images, and the aggregate is carried entirely by the minority it does answer. Its score is not a typical performance, it is an average over a bimodal split. The same holds on TextureADE, where 42% score zero.

LISA's spread is the tightest of any model, and that is the finding. On RWTD-COCO its p10 is 0.442 and its p90 is 0.503 — almost every image lands in a band 0.06 wide. That is not robustness, it is the signature of returning one mask of roughly constant area regardless of the image. Nothing is biasing LISA; it is uniformly the same answer.

mean median bar height is images per bin, scaled per row
Metric

RWTD

Detecture
00.51.0
mean0.795 median0.866 mean−med-0.071 p100.505 p900.968 exactly 00%
LISA-7B
00.51.0
mean0.542 median0.506 mean−med+0.036 p100.403 p900.748 exactly 00%
GSVA-7B
00.51.0
mean0.423 median0.406 mean−med+0.016 p100.206 p900.671 exactly 00%
GLaMM
00.51.0
mean0.420 median0.485 mean−med-0.065 p100.000 p900.752 exactly 019%
Sa2VA-4B
00.51.0
mean0.399 median0.422 mean−med-0.023 p100.000 p900.862 exactly 017%

RWTD-COCO

Detecture
00.51.0
mean0.854 median0.911 mean−med-0.058 p100.615 p900.967 exactly 00%
LISA-7B
00.51.0
mean0.482 median0.480 mean−med+0.001 p100.442 p900.503 exactly 00%
GSVA-7B
00.51.0
mean0.425 median0.399 mean−med+0.026 p100.214 p900.742 exactly 00%
GLaMM
00.51.0
mean0.356 median0.444 mean−med-0.088 p100.000 p900.498 exactly 018%
Sa2VA-4B
00.51.0
mean0.240 median0.000 mean−med+0.240 p100.000 p900.896 exactly 063%

TextureADE

Detecture
00.51.0
mean0.748 median0.771 mean−med-0.023 p100.535 p900.934 exactly 00%
LISA-7B
00.51.0
mean0.316 median0.314 mean−med+0.002 p100.203 p900.424 exactly 00%
GSVA-7B
00.51.0
mean0.209 median0.204 mean−med+0.005 p100.078 p900.350 exactly 00%
GLaMM
00.51.0
mean0.167 median0.161 mean−med+0.006 p100.000 p900.382 exactly 027%
Sa2VA-4B
00.51.0
mean0.179 median0.061 mean−med+0.119 p100.000 p900.498 exactly 042%

CSTD

Detecture
00.51.0
mean0.883 median0.969 mean−med-0.086 p100.540 p900.980 exactly 00%
LISA-7B
00.51.0
mean0.509 median0.517 mean−med-0.007 p100.290 p900.717 exactly 01%
GSVA-7B
00.51.0
mean0.715 median0.733 mean−med-0.018 p100.420 p900.973 exactly 00%
GLaMM
00.51.0
mean0.484 median0.544 mean−med-0.060 p100.000 p900.737 exactly 013%
Sa2VA-4B
00.51.0
mean0.596 median0.625 mean−med-0.029 p100.108 p900.984 exactly 07%