Why is mode collapse in RLHF such a serious problem that almost nobody talks about?

The spicy take is that I believe Yann LeCun is among the closest to reality in his criticisms of large language models. He has this famous slide about how LLMs are doomed as sequence length increases because the probability of an error grows monotonically. Mathematically obvious, yet empirically it does not hold. The reason is mode collapse. When you train with RLHF, the model stops covering the full distribution of possible answers and drops the minority cases to concentrate on what gets rewarded. This is exactly what GANs did when they started making blurry images instead of diverse ones. The model becomes hyperconfident. It needs to be miscalibrated in order to not go off the rails because errors are very easy to detect and subtle wrongs are very hard to spot. That miscalibration then warps the entire probability space of the model.