Why Fine-Tuned LLMs (SFT & LoRA) Fail to Reason

so we are going back to KANs with trained activations! What a time to be alive!

Knowledge modules should simply be standard procedure going forward. Llms should look at the data they are being trained on and optimize in buckets accordingly

Amazing video as usual.

Super interesting. Thank you. I wonder whether, and to what degree, the same thing applies to in context learning, and also conversely whether some de-minimis reference in the context, to a complex set of facts that the model has been SFT'd on would serve to close the knowledge -> use gap.

Weights are like employees, or layers are like employees?

The knowing-using gap is the part that should worry people, and it's because of HOW it fails: silently. The fact is in the weights. The model passes your recall eval. And then it just doesn't use it downstream - and you need activation patching to even find out. Compare that to leaving the fact outside the model and putting it into context at inference. That has its own failure mode: if the right fact isn't retrieved, the model never sees it at all. But that failure is OBSERVABLE. It's in the retrieval logs. You can measure it, alert on it, fix it. An invisible failure and a visible one are not the same class of problem, even when the end behaviour looks identical to a user. To be clear, that's not a solution to reasoning - an external store only fixes availability, not the model's ability to reason over what it's given. But it does mean you stop debugging layer 12.

fascinating. i wonder if there's detectable patterns in the activations that differ between memorised facts that have and haven't been bound to the reasoning yet. also, that idea about no gradient for generalisation after loss becomes low, i wonder if/how RL changes that.