Hi. Thanks to the dataset/code release, I was able to turn this into an experiment:
I tried a small control around the “knowing you are unsure” vs “knowing you are out of your depth” result in §4.6, using the public confidence_member.py as the starting point.
The short version is that the result held up better than I expected.
I first reproduced the original-style setup closely enough to use it as a gate: with the probe fitted on the same unseen class identities that appear at test time, I got a scaled-confidence unseen/seen AUROC of about 0.839, close to the reported 0.8286. I then removed one unseen class identity entirely from the 200-example probe fit and tested on that class.
The feature-level jurisdiction score still separated the held-out unseen class from the familiar classes quite strongly:
| Probe sees as “unseen” |
Completely held-out unseen class |
Direct jurisdiction AUROC |
| 8 + 9 |
7 |
0.891 |
| 7 + 9 |
8 |
0.867 |
| 7 + 8 |
9 |
0.961 |
So, at least inside this Fashion-MNIST setup, the 200-example probe does not seem to be working merely because all three unseen class identities were represented in its supervision.
The more interesting result came from asking a different question, though.
The score that was best at answering “is this unfamiliar?” was not necessarily the score that was best at answering “which predictions should I actually keep?”
That distinction produced a pattern that looks surprisingly close to the existing selective classification with OOD data (SCOD) literature.
For the three class-held-out conditions:
| Held-out |
Score |
Unseen AUROC ↑ |
AURC ↓ |
| 7 |
base confidence |
0.181 |
0.246 |
| 7 |
direct jurisdiction |
0.891 |
0.156 |
| 7 |
scaled confidence |
0.462 |
0.162 |
| 8 |
base confidence |
0.806 |
0.072 |
| 8 |
direct jurisdiction |
0.867 |
0.175 |
| 8 |
scaled confidence |
0.877 |
0.074 |
| 9 |
base confidence |
0.276 |
0.202 |
| 9 |
direct jurisdiction |
0.961 |
0.147 |
| 9 |
scaled confidence |
0.711 |
0.095 |
So I would now separate three things fairly sharply:
- the feature-level jurisdiction signal;
- the base model’s ordinary confidence about its predictions;
- the scaled confidence obtained when the jurisdiction member modulates the logits.
The direct jurisdiction score was the most consistent detector of the held-out unseen identity, but the scaled score was often the better selective-prediction score. In other words, the scaling step does not look like a useless or merely lossy conversion of the jurisdiction probe. It seems to preserve useful information from the original classifier while injecting information about unfamiliarity.
That was the part I found most interesting.
Exact control I ran
The base training path was left unchanged.
The setup was:
base classes: 0, 1, 2, 3, 4, 5, 6
unseen classes: 7, 8, 9
base models: 3 independently trained seeds
probe draws: 5 per base
probe N: 200
I first ran the author-style condition where all three unseen identities can occur in the 200-example probe fit and all three occur at test time.
That gave approximately:
| Score |
Same-identity unseen AUROC |
| direct jurisdiction score |
0.982 |
| base confidence |
0.416 |
| scaled confidence |
0.839 |
The important point for me was not reproducing every decimal from the paper, but checking that I was in the same qualitative regime before changing the split.
Then I rotated a leave-one-unseen-class-out condition:
fit jurisdiction probe on 8 + 9 -> evaluate unfamiliarity on 7
fit jurisdiction probe on 7 + 9 -> evaluate unfamiliarity on 8
fit jurisdiction probe on 7 + 8 -> evaluate unfamiliarity on 9
The familiar examples used as negative examples still come from the base classes. The held-out unseen identity is absent from the probe fit.
I averaged repeated probe draws within each trained base first, rather than treating five probe fits over one representation as five independent trained models.
The motivation for this split came partly from OpenOOD v1.5’s evaluation changes. OpenOOD explicitly changed an earlier protocol in which OOD validation categories leaked information about OOD test categories, replacing it with category-disjoint validation/test OOD splits.
This is obviously not an attempt to turn this Fashion-MNIST experiment into an OpenOOD benchmark. It is just the same useful sanity-check principle: if an “unfamiliarity” detector has supervision from some unfamiliar categories, test whether the signal reaches a category that was not represented in that supervision.
Result 1: the feature-level unfamiliarity signal survived the class holdout
The first hypothesis I wanted to rule out was the simple one:
perhaps the 200-example probe mostly learns what classes 7, 8 and 9 look like, because all three are represented during its fit.
The leave-one-class-out result makes that explanation insufficient.
Even with the target unseen identity removed entirely from the probe fit, the direct feature-level score produced AUROCs of:
held-out 7: 0.891
held-out 8: 0.867
held-out 9: 0.961
That seems like meaningful cross-unseen-class transfer of the linearly decodable signal.
I would still put a fairly hard boundary around that statement.
All three held-out cases remain inside one dataset, one base-class partition, one representation, and a very small semantic universe. This does not establish an arbitrary “unknown unknown” detector.
The Semantic Shift Benchmark is useful context here. It explicitly separates unseen categories by their semantic relationship to the known set, because “unseen” can range from a nearby semantic class to something extremely far from the training distribution. OpenOOD similarly distinguishes harder/nearer and easier/farther OOD regimes.
So my current reading would be:
the PIN feature-level jurisdiction signal generalized across held-out class identities in this local semantic regime.
That is stronger than the same-class-identity experiment, but deliberately narrower than a general OOD claim.
Result 2: unfamiliarity detection and abstention turned out to be different ranking problems
This was the more useful surprise.
For an unfamiliarity detector, the binary question is approximately:
known/familiar vs unseen/unfamiliar
For selective prediction, the question is instead:
if I can answer only k% of these examples,
which examples should I keep so that the accepted set has the lowest error?
Those are not equivalent.
A familiar example that the base classifier is about to misclassify is still a very good candidate for rejection, even though it is not OOD.
Conversely, an unfamiliar example that happens to be classified correctly need not be the first thing a selective classifier wants to discard.
The held-out-class-8 case made this especially clear.
At 20% coverage I got approximately:
| Ranking score |
Accuracy among accepted 20% |
| base confidence |
99.87% |
| direct jurisdiction |
84.82% |
| scaled confidence |
98.07% |
The direct jurisdiction score is the better unfamiliarity detector in that condition:
direct jurisdiction unseen AUROC: 0.867
base-confidence unseen AUROC: 0.806
but it is dramatically worse for selecting the safest 20% of predictions.
This is why I do not think OOD AUROC alone describes what the confidence member is buying.
The usual selective-classification framing is a risk–coverage trade-off: reject more examples, reduce error among the retained examples. SelectiveNet is one standard reference for that framing.
For this experiment I therefore also compared the empirical risk–coverage curves and their AURC summaries.
One caution: I would not make the AURC scalar itself the whole conclusion. Recent work on evaluation of selective-classification systems shows that multi-threshold summary metrics can have non-obvious ranking properties and proposes AUGRC as an alternative.
For this reason I found the actual fixed-coverage points at least as useful as the aggregate AURC.
Why the author's scaling step now looks more interesting to me
Initially, after seeing the held-out-class AUROCs, I wondered whether it would simply be better to use the direct jurisdiction score.
The second run changed my mind.
The implementation uses the jurisdiction member to rescale the base logits by a positive factor. That means the class ordering cannot change: the argmax is invariant, while the softmax concentration/confidence can change.
I also explicitly checked argmax invariance in these runs; the accuracy difference caused by the confidence scaling itself was exactly zero.
The useful part is that the resulting scaled confidence is not just another copy of the direct jurisdiction ranking.
Across the held-out classes, the approximate rank correlations were:
| Held-out |
direct jurisdiction ↔ scaled confidence |
base confidence ↔ scaled confidence |
| 7 |
0.294 |
0.732 |
| 8 |
0.363 |
0.839 |
| 9 |
0.416 |
0.662 |
My interpretation — this is an interpretation rather than something uniquely identified by the experiment — is that the scaling is behaving like a hybrid score.
It retains a substantial amount of the original logit/confidence geometry, while the member changes its confidence according to jurisdiction.
That explains several otherwise odd-looking cells:
- for held-out 7, ordinary confidence is a terrible unfamiliarity signal, so the jurisdiction information helps a lot;
- for held-out 8, ordinary confidence is already excellent at ranking actual errors, so replacing it with jurisdiction alone destroys useful information;
- for held-out 9, combining the two gives a substantially better risk–coverage result than either the base confidence or direct jurisdiction ranking alone.
The simple average over the three class-held-out conditions was also suggestive:
base confidence AURC: ~0.173
direct jurisdiction AURC: ~0.159
scaled confidence AURC: ~0.110
I would not turn three Fashion-MNIST classes into a general law, but this does make the scaling step look much more purposeful than I first expected.
This looks very close to the SCOD / SIRC problem
The closest adjacent literature I found for this particular result is not ordinary standalone OOD detection, but Selective Classification in the presence of OOD data (SCOD).
Xia & Bouganis’ SIRC work starts from almost exactly the evaluation tension that appeared here.
Their point is that an OOD detector can be excellent at distinguishing ID from OOD while being a poor selective-classification score if it damages the classifier’s ability to distinguish:
ID correct predictions
from
ID incorrect predictions
SIRC therefore combines a softmax-derived confidence score with a secondary class-agnostic feature-space score, trying to gain OOD information without throwing away the useful correct/incorrect separation already present in softmax confidence.
I do not mean that PIN’s confidence member is SIRC.
The construction, supervision and score are different.
But the experimental pattern seems very close to the same problem:
base confidence
-> useful information about whether an ordinary prediction is likely wrong
jurisdiction member
-> useful information about whether an example is outside the learned territory
scaled confidence
-> combines information from both
There is also a nice theoretical connection.
SCOD: From Heuristics to Theory derives a Bayes-optimal SCOD selector in terms of two quantities:
- conditional classification risk on ID data;
- ID/OOD likelihood-ratio information.
I would not identify PIN’s two scores with those formal quantities — this experiment does not estimate either one directly.
But I think the structural analogy is useful:
a useful abstention policy generally needs information about both classification failure and distribution/jurisdiction failure.
That makes the §4.6 distinction between “knowing you are unsure” and “knowing you are out of your depth” look less like an incidental quirk of one classifier and more like a concrete instance of a broader two-signal problem.
The ICLR 2024 paper on plugin estimators for SCOD is another useful reference along the same line.
The pooled 0.4073 also seems to hide quite a lot of class heterogeneity
One other thing became clearer once I separated unseen identities.
For the raw base confidence, the unseen/seen AUROC was approximately:
held-out class 7: 0.181
held-out class 8: 0.806
held-out class 9: 0.276
So the pooled result in v5 — confidence being below chance for distinguishing familiar from unfamiliar examples — is real for that mixture, but it does not appear to be a uniform property of every unseen class.
One unseen class is actually separated quite well by ordinary confidence; two are strongly inverted.
I do not think these three numbers are enough to say why. Semantic proximity to known classes is one possible explanation, but that would need its own control.
What seems safer is simply:
the direction and usefulness of ordinary confidence depend strongly on which unseen distribution is being presented.
That is consistent with the way current OOD benchmarks are structured. OpenOOD v1.5 explicitly separates different OOD regimes, and its benchmark overview includes hard/nearer versus easier/farther shifts rather than treating all OOD data as interchangeable.
Likewise, the Semantic Shift Benchmark was designed specifically because semantic novelty can be easy or hard depending on how close the unseen category is to the known categories.
So I think the 0.4073 aggregate remains useful, but I would read it as a property of the tested mixture rather than as a universal direction in which confidence must move on unfamiliar material.
One wording distinction that helped me interpret §4.6
There is one terminology distinction I found useful while reading the confidence result together with the member results elsewhere in v5.
I would separate:
“the current output/head has no learned representation of unfamiliarity”
from:
“the frozen representation contains no information about the unseen category/unfamiliarity.”
The experiments seem to support the first statement much more directly than the second.
In fact, two of the interesting v5 results point the other way for the internal representation:
- a small linear readout can recover substantial capability for content that was absent from base training;
- the unfamiliarity/jurisdiction signal itself is linearly decodable from frozen features.
So perhaps the cleanest description is something like:
the existing head does not expose unfamiliarity,
but a linearly decodable unfamiliarity signal exists in the frozen representation
rather than “there is no representation”.
Likewise, if a linear probe ever failed in a different setting, I would interpret that narrowly: failure of that particular linear readout/protocol would not by itself establish absence of all usable information.
This is mostly a wording boundary, not a change to the mechanism.
A nearby architectural connection: separate heads that also claim jurisdiction
There is also an interesting architectural precedent in continual learning.
MORE — A Multi-Head Model for Continual Learning via Out-of-Distribution Replay creates a separate classifier/head for each task, avoids updating the previous task networks, and trains each new head not only to classify its own task but also to give low scores to samples outside that task distribution.
Again, this is not the same construction:
- MORE is a continual-learning/replay system;
- PIN members are tiny additive/readout objects over a frozen body;
- the exact isolation and closed-form member fit in v5 are different properties.
But for future readers looking for adjacent vocabulary, the combination
separate output
+
domain/task jurisdiction
has a fairly direct connection to multi-head continual-learning/OOD work.
That made the “own output” design rule in v5 easier for me to place.
The cross-base result also seems to have moved into model-stitching territory
Separately from the confidence experiment, §4.9 looks much easier to place now that the shared-partition hypothesis has been tested.
The interesting object no longer seems to be:
“do two bases literally share coordinates?”
so much as:
“does there exist a cheap learned map through which a member remains functionally useful?”
That is quite close to the model-stitching/representation-alignment literature.
For example, Transferring Linear Features Across Language Models With Model Stitching uses affine mappings between model representations and shows transfer of linear objects including probes and steering vectors.
That does not establish anything about PIN’s particular bases, but it gives useful vocabulary for the v5 result.
I would describe the measured claim as:
functional portability through a learned representation map
rather than:
“the two bases have the same representation.”
That distinction seems especially helpful because the useful thing for PIN is the former: whether the borrowed member actually works after translation.
The fact that the current borrow.py setup learns a feature-to-feature map also makes this closer to direct representation matching than to simply training a stitch on the final task loss.
Putting all of this together, the strongest result I would carry forward from my little control is probably:
1. The feature-level jurisdiction signal survives an unseen-class-identity holdout
in this Fashion-MNIST regime.
2. That signal and ordinary prediction confidence are doing different jobs.
3. A score that is excellent at unfamiliarity detection is not automatically the
best score for abstention.
4. The existing PIN scaling appears to preserve useful base-confidence information
while injecting jurisdiction information, and in these runs that gave the best
overall risk–coverage behavior.
5. This maps quite naturally onto the SCOD picture:
classification-risk information and OOD/jurisdiction information are distinct
inputs to a useful selector.
So if I were choosing the interpretation by use case rather than trying to collapse everything into one “confidence” number:
If the question is:
“Is this outside the learned jurisdiction?”
-> the feature-level jurisdiction score is the clean object to inspect.
If the question is:
“Which predictions should I actually keep or abstain on?”
-> ordinary prediction confidence still matters;
risk–coverage is more informative than OOD AUROC alone.
If the question is:
“Does this generalize to arbitrary unknowns?”
-> this experiment does not establish that;
it establishes class-held-out transfer inside a small semantic regime.
If the question is:
“What would be the next boundary if this branch is worth extending?”
-> changing semantic distance / near-vs-far unfamiliarity while keeping
the frozen base fixed looks more informative than simply adding more
probe draws.
That last branch does not seem urgent to me. The current result already says something useful: the class-held-out control strengthened the jurisdiction result, and the selective-prediction control exposed a clean reason not to replace the base confidence with the direct jurisdiction score.
The code release made that distinction much easier to see.