Local AI models destroyed by further Distillation

About the epistemic token count (ETC), the situation is the same as before, so the distillation worked well, reproducing an inefficiency of the model (DeepSeek). Could it be an explanation?

I am not certain this is such a great discovery. The way these are trained is I believe to keep changing so it improves. It is then increasingly the case that a change makes no difference or makes it worse. They're normally shortened already as much as can be. You could perhaps create one with faster reaction time that is specialised by pruning certain paths if they are overly general. The reasoning though is not real. That's a mistake. It is reproducing the illusion of reasoning but it's not actually thinking or understanding what it is saying like a person would. If you want to improve a model you often need to bloat it which is fiddly, then compact again. If you over train on something specific then a model can like a student just memorise the answer.

so...basically, they sussed that the model was shortcutting, and they tried to overcome it. They did so but with limited success - but even limited success might point at complete solutions. As stated, the limited success suggests the solution is incomplete. Without question, the team found a rather good way to preserve the signal...but did they really have enough data from which to draw a strong enough signal in the first place? As a non academic, I'll have to scurry off to an AI for better understanding of this paper and the points made in this vid, but these are my immediate impressions.

Always count on chinese to provide the rigorous mathematical explanation of intuition.🎉

Simple fix, hook up real intelligence into the loop. Ethical options include human neurons, mycelium, slime mold, ant hive, virtual c eligans(misspelled), etc. Probably unethical solution is a human sacrifice with bci. Only ethical if the owner/CEO of the AI is the willing sacrifice. (Note, sacrifice doesn't mean death here, and they can stop being a sacrifice later, just gotta live with the irreversible changes to their body)

Simple fix. 3:44 You have your valid solution. Teacher generates the synthetic CoT or reasoninv traces to simulate the prompt to valid solution trace. Chunk the CoT or reasoning into components steps. Mask or remove one step of reasoning, have the students learn by recreating the masked out steps.

So self improving harnesses are obsolete?

I'll take an attempt at explaining the epistemic count and marker distribution graphs. First I'd like to point out, that these are the reasoning tokens they are looking at. When it says, "Wait, maybe I'm overthinking this" or, "Perhaps, we need to think differently." a) Shows the total token count of ["wait", "maybe" "check" and Others] over the training steps. With Qwen, the token count of "wait", "maybe" and "check" drop dramatically. Conversely with Deepseek-R1, token count goes up, nearly doubling "wait" more than doubles. I think what the paper is trying to indicate that this is not an indication of the model learning the result, but instead memorizing through overfitting, because if it was learning, it would make smaller movements from the base/frozen model. This is what we can see in the OPSD-PMI (Ours) bars, smaller movements from the base.

Do one on concolic testing

Good paper , thanks for sharing!