Kinda weird to generalize "LLM". Every lab, every model is different. Has its own biases, reward functions etc.
Centigonal 27 minutes ago [-]
[dead]
troupo 5 minutes ago [-]
Without reading the article (doesn't matter if it's pro or contra): no, of course not.
It shouldn't even be a debatable question.
novaapi 34 minutes ago [-]
[dead]
asamoahf 1 hours ago [-]
The unsupervised framing is the part I'd push on. If true labels are latent and you infer them jointly with judge parameters, then a blind spot every judge shares isn't a correlated error the model can down-weight. It's indistinguishable from the ground truth, and the likelihood has no reason to prefer the correct answer over the consensus one.
So this fixes dependence between judges and leaves dependence between all the judges and the truth untouched, which is the failure people are actually worried about when they say eight models agreed. You still want a small human-labelled anchor set to break it. The number I'd find interesting is how much smaller that anchor set gets once you model the dependence, since that's the real saving.
Same shape as offline policy evaluation. Correlated logging errors survive any amount of re-weighting, and one real experiment would be probably what pins them.
Forgeties79 13 minutes ago [-]
the LLM-speak is unbearable
Rendered at 18:09:32 GMT+0000 (Coordinated Universal Time) with Vercel.
It shouldn't even be a debatable question.
So this fixes dependence between judges and leaves dependence between all the judges and the truth untouched, which is the failure people are actually worried about when they say eight models agreed. You still want a small human-labelled anchor set to break it. The number I'd find interesting is how much smaller that anchor set gets once you model the dependence, since that's the real saving.
Same shape as offline policy evaluation. Correlated logging errors survive any amount of re-weighting, and one real experiment would be probably what pins them.