Will JEPA Fail & World Models are just Tools?

Jepa is embedding ++, it encode parameters as point in the dimensional feature space, physics is just vector between point in which the parameter change. So you get a flow map of state changes. Being embedding, we can just strap a transformer on jepa to parse and recover the traces and correlate with languages. Which solve the convexity problem inherent to embedding. Current architecture research are implicitly converging to an hybrid model, with mtp and loop transformer enforcing more of a jepa like reasoning by keeping latent momentum. The main issue is the landscape navigation have edge sensitivity, it's like it learn a building layout, but occasionally go through the walls because they are too thin. And the issue is that we are trying to not let him do that without entering nor knowing the layout of the building. Harness kinda are trying to solve the wall phasing problem. I mean it's a comment, it's over simplified, before i got answer that implicitly tell me i wasn't as precise as a 40mn video 😂 it's meant that you can fill the blank.

Well damn... That worldlab marble model is exactly something I've been wanting... I was thinking of creating a a VR game for my mother who has a crippling fear of heights, some games exist but they are usually extreme to the point any normal person would freeze... So I thought if I could get images from places she would actually visit and have issues with - turn it into 3d and let her experience those places in a safe place she could get used to the heights she has issues with in real life. Talking crossing bridges, stairs with a open side, glass elevators or balconies. BUT, I could never find anything that could export actual usable assets from limited images, best was those 'video' worlds and those were useless... This on the other hand looks perfect. Thanks for the shout out to such a handy hidden gem ^_^

I'm sure we do need to go beyond LLMs or even multi-model models, but I suspect the continuous learning process would be key as well. The reason is that beyond token sequences or tensor sequences, the complexity explodes and generality decreases. A 3D game engine is essentially numerical integration (plus constraints), eg if not collision, position += velocity * delta t. 3D game, jet engine simulation and satellite orbit modelling (jet requires chemistry, orbits general relativity) require very different approaches to the numerical analysis. So given that LLM can write code, including game engines, finite element models and algebraic PDE solvers, it might have a better chance of physical modelling. ie learn the model, use the model. My personal view is that prompt completion can't measure 'real intelligence'. Take your elevator logic test for example. An intelligent system might well get it wrong first time, but after some feedback and a few different scenarios, it should start to get it right. Intelligence is not a large knowledge base or even a reasoning capability, surely it is a learning capability?

29:40 A separate decoder would need to be trained and, even then, the decoded latent vectors would be an approximation.

Completely agreed at 35:26. The 'truth' of the world model will not be perfect for the observer (robot, agent) to learn or imagine best in all situations. The latent representation for the world model (e.g. JEPA) may be better at compression, but not efficient in speed / time step, hiding granular details that can then not be rendered in the world model. 1) The timestep granularity of each of these three parts of the "world system" and 2) which interactions between them take place at which timestep or interval size is probably something to resolve (e.g. The agent could imagine in milliseconds to seconds, world model could render in milliseconds, and the representation model, model latent space next step / trajectory in micro seconds - so my question is basically: what is the micro / macro scale and overlap in time across the parts of the "system", as states and actions progress?)

It does not matter whether the modeling of the world occurs before or after language production; what matters is that the heuristics learned from the world are used for prediction, and the generated thoughts are fed back in to improve planning strategies.

World models make the system dumb (in a virtual world - while trying to be smart in the physical world), Specialist models are where I would be going for me its language ideas persuassion + some image undertanding - where the $$ is - what people will pay for

Isn’t the latest JEPA TC-JEPA though?

So puts on AMI Labs (if it goes public?)

26:40 Well, yes. If you already know the kinematic equations of motion, then you should use them directly instead of simply throwing everything into a ML model. That was always the case.