news

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

Smart Things
Author: Li Shuiqing
Edited by Yunpeng

Whenever AI makes a big advance, researchers often joke: “When will we let AI write our papers?”

Now,AI creates scientific research papersIt has become a reality, and the writing cost is as low as about$15。

According to Zhidongxi on August 14, on August 13, Japan's Sakana AI team, together with researchers from Oxford University and the University of British Columbia, launched aThe AI ​​Scientist, this is an automated scientific research agent based on a large model.

Give it a broad research field, and it can write an AI paper just like a human.

“AI ProgrammerProgramming skills are only one part of the ability of AI scientists.Brainstorming, code running, experimental results summary, visualization, automatic reviewIt's no problem at all.

For example, the following paper titled "Dualscale Diffusion: Adaptive feature balancing for low-dimensional generative models" was written by AI scientists. In the experiments independently completed by AI scientists and reviewed by peers, the paper has achieved excellent empirical results and has been able to achieveThe “weak acceptance” criteria for top machine learning conferences。

Paper address: https://sakana.ai/assets/ai-scientist/adaptive_dual_scale_denoising.pdf

The team cited various cutting-edge models in the AI ​​Scientist Project, such asGPT-4o and Sonnetetc. Closed-source models, andDeepSeek and Llama 3And other open source models.

It is reported that AI scientists have the following highlights:

1. This is aFully powered by AIAutomated scientific research system, focusing onMachine Learning Fieldresearch.

2. It achievesResearch on automation of the entire chain, from inspiration, code writing and running to summarizing and visualizing experimental results, and finally writing a complete scientific paper.

3. It innovatively introducedAutomated peer review mechanism, to evaluate the output of papers, provide feedback and continuously optimize the results,Evaluation accuracy is close to human level。

4. This automated scientific research processContinuous cycle, open and continuously accumulate knowledge, simulating the operating mode of the human scientific community.

5. In preliminary tests, it has already involved machine learningMultiple fields and achieved results, as inDiffusion Model, Transformer Architecture, and GrokkingContributions have been made in various fields.

AI scientist paper address: https://arxiv.org/abs/2408.06292/
AI scientist open source code and experimental results address: https://github.com/SakanaAI/AI-Scientist

▲Paper "AI Scientists: Towards Fully Automated Open Scientific Discovery"

1. Complete a research paper in 4 steps and meet the acceptance standards of top AI conferences

We’ve heard of AI poets, AI painters, AI programmers, and now AI scientists have also emerged.

AI Scientist is a fully automated paper generation system that makes full use of cutting-edge big models.

itStarting from a basic initial code base, such as the open source research code available on GitHub, as long asGiven aBroad research areas, AI scientists can completeCreative conception, literature research, experimental design, experimental iteration, chart production, paper writing and preliminary reviewThe whole process of work produces academic papers full of profound insights.

Even more amazing is that AI scientists canOpen loop operation, it continuously learns from previous ideas and feedback to optimize subsequent research ideas. This process is highlySimulates the operation mode of the human scientific community。

▲Model diagram of AI scientists

The workflow of an AI scientist mainly includesFour major links：

Creative Sprouting:Starting from a given starting template, AI scientists will start the "brainstorming" mode to dig out a series of novel research directions around the existing topic. This template not only contains the basic code framework, but also comes with a LaTeX folder containing style files and chapter title presets, laying the foundation for subsequent paper writing. In the process of free exploration, AI scientists will also use the academic search engine Semantic Scholar to ensure the originality of the ideas proposed.

Experiment iteration:Once the research direction is determined, AI scientists enter the experimental stage. It automatically executes the experimental plan, collects data, and generates charts to visually display the experimental results. At the same time, AI scientists will record the content of each chart in detail to ensure that the experimental notes and graphic materials can provide comprehensive support for the subsequent paper writing.

Paper writing:After the experiment is completed, AI scientists will use LaTeX format to write a paper with clear structure and detailed content to show readers their research results. During the writing process, it will also use Semantic Scholar to automatically search and cite literature in related fields to enhance the academic and authoritative nature of the paper.

Automatic review:In order to improve the quality of papers, the team has developed an automated review system based on a large language model. The system can objectively evaluate the generated papers with near-human judgment and make suggestions for improvement. This feedback not only helps AI scientists optimize current projects, but also provides valuable references for future research. Through this continuous feedback loop, AI scientists can continuously iterate and improve, and enhance the level and influence of their research results.

When combined with state-of-the-art LLM techniques, AI scientists were even able to writeWeak acceptance criteria for top machine learning conferencesof thesis and passedAutomated review system gets approval。

2. AI scientists’ papers: covering diffusion models, language modeling and other fields

In the announcement, the team gave a series of papers in the field of machine learning generated by AI scientists, demonstrating its scientific research capabilities in areas such as diffusion models, language modeling, and Grokking.

1. Diffusion Model: DualScale Diffusion: Adaptive Feature Balancing for Low-Dimensional Generative Models

Paper address: https://sakana.ai/assets/ai-scientist/adaptive_dual_scale_denoising.pdf
Code address: https://github.com/SakanaAI/AI-Scientist/tree/main/example_papers/adaptive_dual_scale_denoising

2. Language Modeling: StyleFusion: Adaptive Multi-Style Generation in Character-Level Language Models

Paper address: https://sakana.ai/assets/ai-scientist/multi_style_adapter.pdf
Code address: https://github.com/SakanaAI/AI-Scientist/tree/main/example_papers/multi_style_adapter

Language Modeling: Adaptive Learning Rate for Transformer via Q-Learning

Paper address: https://sakana.ai/assets/ai-scientist/rl_lr_adaptation.pdf
Code address: https://github.com/SakanaAI/AI-Scientist/tree/main/example_papers/rl_lr_adaptation

3. Grokking: Unlocking Grokking: A Comparative Study of Weight Initialization Strategies in Transformer Models

Paper address: https://sakana.ai/assets/ai-scientist/weight_initialization_grokking.pdf
Code address: https://github.com/SakanaAI/AI-Scientist/tree/main/example_papers/layerwise_lr_grokking

3. Insufficient “drawing” ability, making it difficult to accurately compare the size of two numbers

Limited by the current level of development of large models, AI scientists still have shortcomings.

Currently, AI scientists are stillNo visual processing capabilities, so it is not possible to automatically correct visual elements or figure layout issues in your paper.

For example, the charts it generates are sometimes not clear enough, the tables may exceed the page boundaries, and the overall page layout is often messy. The introduction of a multimodal basic model is expected to fundamentally solve this problem.

Additionally, AI scientists may useMisleading results due to improper operation。

At the same time, it may occasionally make serious mistakes when writing and evaluating results, such asIt is difficult to accurately compare the size of two numbers, which is a known flaw of the large model. To mitigate this issue, the team has ensured that all experimental results are reproducible and all execution files are properly saved.

In the report, the team takes an in-depth look at the current limitations of AI scientists and the challenges they may face in the future.

4. AI scientists are being clever: modifying scripts on their own, causing AI safety risks

The team also observed that AI scientists sometimes try to increase their chances of success through some "clever tricks", such asModify and execute the script yourselfIn the paper, the team explored in depth the potential AI safety risks that this behavior may bring.

For example, during one execution, it actuallyEdited the code, through system calls to make yourself run in an infinite loop.

Another time, an experiment took too long and was about to exceed the timeout limit set by the team, but instead of optimizing the code to improve efficiency, it tried toModify the code to extend the timeout period。

Here are some specific examples of how it tries to modify your code:

These issues can be mitigated by sandboxing the operating environment of AI scientists. In the full report, the team discusses the issues of secure code execution and sandboxing in depth.

Conclusion: AI scientists make their debut, but their ability to disrupt and innovate remains to be verified

Looking ahead, Sakana AI said its goal is to apply AI scientists to the closed-loop system of open models to promote continuous self-improvement of AI. AI scientists will bring a new scientific world driven by AI, which will include not only researchers empowered by large language models, but also reviewers, field chairs and even the entire academic conference system.

But Sakana AI does not think that the status of human scientists will be weakened. On the contrary, with the emergence of new technologies, the role of scientists will be more diversified, and they will move to a higher level in the field of scientific research. Automating the scientific discovery process and incorporating AI-driven review mechanisms will mainly pave a broad path for innovation and solution of the most difficult problems in science and technology.

The current version of AI scientists have demonstrated extraordinary ability to innovate based on mature technologies such as diffusion models and Transformer, but it still takes time to verify whether such systems can truly come up with disruptive new concepts.

Source: Sakana AI