news

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

CodexGraph team contributions
Quantum Bit | Public Account QbitAI

Can’t finish the code generation and completion tasks? !

Why not try to let Large Language Models (LLMs) help?

However, if it isLLMs can struggle with large, multi-file code bases.At this time, the Retrieval-Augmented Code Generation (RACG) system is often needed to retrieve relevant code information.

Existing solutions, such as semantic similarity-based retrieval and manually designed tools and APIs, although effective in specific tasks, have limitations in generalization and universality.

In order to alleviate the versatility defects of existing RACG methods, researchers from the National University of Singapore, Alibaba, and Xi'an Jiaotong University proposedCodexGraph，A system that connects language models and code bases using a code graph database as a medium.

The research team evaluated CodexGraph on three representative academic benchmarks, and also implemented CodexGraph in five real-code application scenarios based on ModelScope-Agent.

Experiments and real-world practices demonstrate the effectiveness and versatility of CodexGraph.

Research Methods

CodexGraph innovatively connects language models with code bases through the code graph database interface, in order to overcome the shortcomings of existing methods.

The working process of this system mainly includes the following steps:

First, build a code graph database。

Using static analysis technology, according to the predefined schema, the code symbols (such as modules, classes, functions, etc.) in the code base and the relationships between them are mapped to nodes and edges in the graph database.

Second, code structure-aware query.

By taking advantage of the flexibility of graph query language and combining it with the structural characteristics of graph databases, we can achieve complex queries and deep retrieval of code graphs.

Third, the “Write then translate” strategy.

The Primary LM agent first analyzes code issues and generates natural language queries.

Then, a dedicated translation LM agent translates these queries into formatted graph query language.

Fourth, iterative pipeline.

In an iterative manner, the LM agent gradually optimizes the query and retrieval results based on user questions and the information obtained.

experiment

In the experimental part, CodexGraph was evaluated on three representative codebase-level benchmarks: CrossCodeEval, EvoCodeBench, and SWE-bench.

These testsCovers cross-file code completion, code generation, and automated GitHub issue resolutionAnd many other aspects.

Experimental results show that CodexGraph demonstrates excellent and stable performance in all benchmarks, verifying the effectiveness of the code graph database as an interface between the language model and the code base.

The analysis in the original paper further points out thatWhen CodexGraph handles reasoning tasks of different difficulty levels, the effects of single query and multiple query strategies differ.

In CrossCodeEval, the multiple query strategy is more effective, while in SWE-bench, the single query strategy performs better.

In addition, removing the Cypher LM agent will significantly increase the reasoning burden of the primary LM agent, resulting in performance degradation.

This shows that the Cypher LM agent plays a key role in alleviating the reasoning pressure of the primary LM agent.

Real application scenario examples

Based on the ModelScope-Agent framework, the practical application value of CodexGraph has been reflected in multiple scenarios, such as:

• Code Chat Assistant（Code Chat）
• Code debugging tools（Code Debugger）
• Unit test generator（Code Unit tester）
• Code Generator（Code Generator）
• Code commenting tools（Code Commenter）

These applications demonstrate how CodexGraph can help developers solve practical problems in a production environment and improve the efficiency and quality of code development.

Summarize

Using graph database as a medium, CodexGraph provides a new way of interaction between language models and code bases.

It not only improves the language model's ability to understand complex code bases, but also enhances the accuracy of overall code retrieval and navigation.

Its emergence has opened up the field of automated software engineering.Especially when facing RACG challenges, bringing new solutions and demonstrating broad potential and practical value in diverse coding tasks and real-world applications.

With the continuous development of technology, CodexGraph is expected to support more programming languages ​​in the future and further optimize its workflow and index building efficiency.

Code link:

https://github.com/modelscope/modelscope-agent/tree/master/apps/codexgraph_agent

arXiv link:

https://arxiv.org/pdf/2408.03910