Well, it feels like architecturally there's a lot of scope to do better for coding tasks specifically. Like, if you had FAIR level resources and wanted to train a really great Java coding model for example it would make sense to train the model to predict ASTs rather than tokens. You'd still need some kind of joint normal LLM for predicting comments, identifier names and so on, but you wouldn't model the program itself as a stream of tokens. Instead it would predict things like "add an if block", "add a method call block with 4 parameters" and so on.

You could also train the model to expect certain context window positions to be reserved for things like "type members at the current cursor" and then integrate the inferencing loop with IDE/LSP-style static analysis. This would allow the model to see more information than is actually contained in the text.

I think the reason we're not seeing models like this right now is the cost of doing such research combined with the fact that AI people are all Python-heads, and Python doesn't benefit from much IDEs.

That sounds right. My vague idea of a "second dimension" could well be some sort of structure - be it ASTs for programming languages or for natural language.

Another possibility would be some sort of fixed knowledge base, which could be program language documentation or "common sense" like CYC wants to provide.