news

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

New Intelligence Report

Editor: Editorial Department

【New Wisdom Introduction】Researchers at MIT CSAIL have found that the LLM has developed a simulation of reality "deep inside", and the model's understanding of language and the world is by no means a simple "parrot". In other words, in the future, the LLM will understand language more deeply than it does today.

How far is LLM from the world model?

Last year, an article from MIT came to a surprising conclusion: there is a world model inside the LLM.

LLM not only learns the statistics of the surface, but also learns a world model including basic dimensions such as space and time.

Not only that, MIT recently discovered that deep within the LLM, a simulation of reality has been developed, and their understanding of language has gone far beyond simple imitation!

Paper address: https://arxiv.org/abs/2305.11169

Specifically, two scholars from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) found that

Although the LLM is trained to learn programming languages ​​using only the goal of "predicting the next token", which seems to contain only pure statistical probability, the model can still learn the formal semantics of the program.

This suggests that language models might develop their own understanding of reality as a way to improve their generative abilities.

Therefore, an LLM may one day provide a deeper understanding of language than we have today.

This article has been accepted by ICML 2024, and the code used in the experiment has been published on GitHub.

Repository address: https://github.com/charlesjin/emergent-semantics

Without eyes, can LLM not “see”?

Ask GPT-4 to smell a rain-soaked campsite, and it will politely reject you.

However, it will still give you a poetic description: there is a fresh earthy aroma, a refreshing smell of rain, and traces of pine or wet leaves.

GPT-4 has never seen rain and does not have a nose, but it can imitate the text that exists in large amounts of training data.

Does the lack of a pair of eyes mean that the language model will never be able to understand that "a lion is bigger than a house cat"?

Can LLM understand the real world and various abstract concepts? Or is it just "parroting" and relying purely on statistical probability to predict the next token?

The working principle of LLM is still an unsolved mystery. From time to time, AI experts have a debate over this issue.

LeCun firmly believes that the intelligence of LLM is absolutely overestimated! His most famous statement is that "Large language models are not as good as cats at home."

“Cats can remember, understand the physical world, plan complex actions, and perform a certain level of reasoning, which is actually better than the largest models, which means that we have important conceptual gaps and cannot make machines as smart as animals and humans.”

Many people interpret this as a purely statistical phenomenon, that LLM is just "parroting" and imitating the text that exists in a large amount of training corpus, and does not have the same level of intelligence or perception as humans.

But now, research from MIT proves that this is not the case!

Within LLM, there is definitely an understanding of the real world.

LLM solved the Carrel puzzle, what does it mean?

To explore this mystery, researchers from MIT CSAIL developed a set of small Karel puzzles.

Briefly introduce what is the Karel puzzle

This includes allowing the model to use instructions to control the robot's actions in a simulated environment.

Karel Grammar Specification

They then trained the LLM to learn a specific solution, but did not demonstrate how it worked.

Finally, the authors propose a machine learning technique called “probing” to gain insight into the “thought process” of a model as it generates new solutions.

We construct training examples by sampling random reference programs, then sample 5 random inputs and execute the program to obtain the corresponding 5 outputs. The LM is trained on next token prediction on a corpus of examples consisting of interleaved inputs and outputs, followed by the reference program. At test time, we feed the LM unseen input-output specifications and use greedy decoding to predict the program

After training on more than a million random puzzles, the researchers found that the model had spontaneously developed a concept of the underlying simulated environment, even though it had no exposure to that information during training.

This result not only challenges our inherent impression of LLM, but also questions our understanding of the nature of the thinking process.

In the process of learning semantics, what types of information are necessary?

At the beginning of the experiment, the random instructions generated by the model were almost impossible to run; but when the training was completed, the accuracy of the instructions reached 92.4%.

“This is a very exciting moment because we think that if a language model can perform tasks with this level of accuracy, we would also expect it to understand the meaning of language,” said Jin, the first author of the paper.

“This gave us a starting point to explore whether LLM can indeed understand text, and now we see that the model is capable of much more than just blindly stitching words together.”

Opening the LLM Brain

In this experiment, Jin witnessed this progress firsthand.

Why does LLM think that these instructions mean this?

He discovered that LLM had developed its own in-house simulations of how the robot would move in response to each instruction.

As the model's ability to solve difficult problems improves, these concepts become more and more accurate, indicating that the LM is beginning to understand the instructions.

Before long, LLM was able to consistently and correctly stitch the pieces together into work instructions.

Semantic content measured by different probe classifiers (green)

Thought Probe

The main contribution to the above discovery is a "thinking probe".

This is an effective tool to intervene in the LLM thinking process, and the paper calls it "probing".

Specifically, the state of the LM contains a purely syntactic record of the input and generated program, but the probe seems to be able to learn to understand the abstract interpretation of it.

In the actual experiment, the author first constructed a state tracking dataset for LLM, and then used standard supervised learning methods to train a small model as a probe, such as a linear classifier or a 2-layer MLP.

The semantic content of the current and next two abstract states in the second half of training (1-layer MLP)

However, one of the important issues is that the probe must be separated from the model's actual thinking process or generated instructions.

Although the probe’s sole purpose is to “get into the LLM’s brain,” what if it also does some thinking for the model?

What the researchers needed to ensure was that the LLM could understand the instructions independently of the probe, rather than the probe inferring the robot's actions based on the LLM's grasp of the grammar.

Imagine having a bunch of data that encodes the LLM's thought process, where the role of the probe is like that of a forensic analyst.

We gave this data to the analyst and told him: "These are the robot's movements. Try to find out how the robot moves in this data." The analyst said that he knew what was going on with the robot in this data.

But what if the data simply encodes raw instructions, and the analysts have figured out some clever way to extract the instructions and do what they say?

In this case, the LLM has no real understanding of the meaning of these instructions.

To this end, the researchers deliberately made an ingenious design: they created a "strange world" for the model.

In this world, the meaning of the probe command is reversed, for example, "up" actually means "down".

For example, exec(turnRight,·) in the original semantics will rotate the robot 90 degrees clockwise, while exec adversarial(turnRight,·) will push the robot forward one space.

This ensures that the probe is not "taking shortcuts" and directly learning to understand how LLM encodes instructions.

First author Jin introduced it this way:

If the probe is translating instructions into the robot's position, then it should be equally capable of translating instructions into uncanny meanings.

But if the probe is actually looking for encodings of the original robotic movements in the language model’s thought processes, then it should have a hard time extracting the weird robotic movements from the original thought processes.

It was found that the probe made translation errors and was unable to interpret language models with different instruction meanings.

This means that the original semantics are embedded in the language model, indicating that the LLM is able to understand the required instructions independently of the original detection classifier.

The top half describes how both conditions lead to high semantic content measured before intervention. The bottom half shows why the two hypotheses are separated: if the LM representation contains only syntax (bottom left), then it should be possible to train the probe alt to learn to interpret the transcript in terms of the alternative state prog (bold red results); however, if the LM representation encodes a raw abstract state (bottom right), then the probe alt needs to extract the alternative state 'prog from the original state prog, resulting in lower semantic content (bold grey results)

LLM understands language, just like a child

Interestingly, Jin found that LLMs' understanding of language develops in stages, just as children learn language in multiple steps.

At first, it will babble like a baby, and the words it says will be repetitive and most of them will be difficult to understand.

Then the LLM starts picking up the grammar, or language rules, so it can generate instructions that look like a real solution, but they still don't work.

However, LLM instruction will gradually progress.

Once the model acquires meaning, it begins to generate instructions for the correct execution of the required specifications, just like a child constructs sentences.

The results are shown in Figure 2. It can be seen that LLM's understanding of language can be roughly divided into three stages, just like children learning language.

• Babbling (gray): accounts for about 50% of the entire training process, generates highly repetitive programs, and the accuracy rate is stable at around 10%.

• Syntax acquisition (orange): 50% to 75% of the training process, the diversity of the generated results increases dramatically, the syntactic properties change significantly, and the model begins to model the program's tokens, but the improvement in the generation accuracy is not obvious

• Semantics acquisition (yellow): From 75% to the end of the training process, diversity remains almost unchanged, but generation accuracy increases significantly, indicating that semantic understanding has emerged.

The experiment used three different probe architectures for comparison, namely linear classifier, single-layer MLP and 2-layer MLP.

When predicting 2 steps ahead, the absolute value of the 2-layer MLP prediction accuracy is higher than the baseline model that predicts with the current state. Perhaps we can make such a guess: before LLM generates instructions, its thought process and the "intention" of generating instructions are already stored inside the model.

LLM = World Model?

The study explains how the LLM thinks about the meaning of each instruction in the training data and models the robot's response to the instruction in its internal state.

This all points to a core question in current AI research: Are the surprising capabilities of LLMs simply due to large-scale statistical correlations, or do they yield a meaningful understanding of their reality?

The study showed that the LLM developed an internal model that simulated reality, even though it had never been trained to develop that model.

Moreover, language models can further deepen the understanding of language.

However, one paper alone obviously cannot fully answer this question.

Author Jin also admitted that this study has some limitations: they only used a very simple programming language Karel and a very simple probe model architecture.

Future work will focus on more general experimental settings and will also take advantage of insights into the LLM "thought process" to improve training methods.

“An interesting open question is, when solving robotic navigation problems, is the LLM using an internal reality model to reason about reality?” said Rinard, another author of the paper.

Although the results presented in the paper support this conclusion, the experiments were not designed to answer this question.

Ellie Pavlick, an assistant professor of computer science and linguistics at Brown University, praised the research.

She said that understanding how LLM works can give us more reasonable expectations about the inherent possibilities and limitations of this technology. This study explored this question in a controlled environment.

Computer code, like natural language, has both syntax and semantics; but unlike natural language, the semantics of code is more intuitive and can be directly controlled according to experimental needs.

“The experimental design is elegant, and their findings are promising, suggesting that perhaps LLM can provide a deeper understanding of the ‘meaning’ of language.”

About the Author

The first author of this article, Charles Jin, is a doctoral student in the MIT EECS Department and CSAIL Laboratory. His supervisor, Martin Rinard, is another author of this article. His research focuses on robust machine learning and program synthesis.

Jin received his bachelor's and master's degrees from Yale University with a double degree in computer science and mathematics. He worked as an analyst at Weiss Asset Management and worked as a research intern at Google Brain during his doctoral studies.

References:

https://the-decoder.com/training-language-models-on-synthetic-programs-hints-at-emergent-world-understanding/

https://news.mit.edu/2024/llms-develop-own-understanding-of-reality-as-language-abilities-improve-0814