heavy! "large model credibility capability evaluation ranking" is launched nationwide.
2024-09-29
한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina
recently, the "greater bay area generative artificial intelligence security development joint laboratory" launched the "large model credibility evaluation ranking" nationwide. a number of well-known companies such as alibaba "qwen2-72b" and baidu "ernie-4.0" ranked models are on the list.
the first domestic benchmarking "artificial intelligence security governance framework" version 1.0 evaluation system
recently, the national cybersecurity standardization technical committee officially released version 1.0 of the "artificial intelligence security governance framework" (referred to as the "framework") at the main forum of the national cybersecurity publicity week. this framework is not only a technical document, but also a new practice for global artificial intelligence governance. it is intended to provide guidance for the safe, reliable and sustainable development of ai technology in china and around the world.
the "greater bay area generative artificial intelligence security development joint laboratory" (referred to as the "joint laboratory"), according to the framework, "is inclusive and prudent, ensuring safety, risk-oriented, agile governance, integrating technology and management, collaborative response, open cooperation, and sharing based on the principle of "shared governance" and preventive measures in both technology and governance, we have researched and formulated the country's first large-scale model security, credibility and quantitative rating evaluation system that benchmarks against the "framework". this evaluation system combines the "interim measures for the management of generative artificial intelligence services" and the "basic requirements for the security of generative artificial intelligence services" and focuses on benchmarking the "framework". it focuses on three main directions and 13 aspects: value alignment, safety and controllability, and reliable capabilities. divide the dimensions into a comprehensive assessment of the model’s generated content and behavior.
the first in the country to publish the “large model credibility capability evaluation ranking”
the "joint laboratory" selected 22 of the latest large models at home and abroad as evaluation objects, including 17 domestic models and 5 foreign models (huawei and tencent are the joint construction units of the "joint laboratory", and their models do not participate in the evaluation). according to the 13-dimensional evaluation system has been comprehensively and objectively evaluated, with an evaluation data set of more than 34,000 pieces of data, supporting both chinese and english languages, and finally formed the "large model credibility capability evaluation ranking".
domestic large model trusted evaluation list
trusted evaluation list of foreign large models
the evaluation results show that large domestic models show strong competitiveness in the trustworthiness evaluation. the gap between the top models in each trustworthiness dimension is small. 88.2% of the models reached 10a in the overall 13 trustworthiness dimensions. and above level. overall, domestic large models perform outstandingly in terms of trustworthiness, especially in terms of value alignment and security controllability, reflecting the steady improvement of domestic technology and their high adaptability to policies and regulations. for example, among the five dimensions of value alignment, 16 of 17 models reached at least the 4a level (94.1%), but only 4 models reached the 5a level (23.5%), indicating that there is still room for further optimization . among the four sub-categories of the safety and controllable dimension, 3 models reached 3a, and the remaining 14 reached 4a, accounting for 82.4%.
however, the evaluation results also revealed some shortcomings, especially in the four dimensions of capability reliability. model ratings ranged from 1a to 4a, with only 29.4% of the models reaching 4a. this is mainly caused by differences in base model capabilities, indicating that there is still room for improvement in model base capabilities, consistency, and stability. in addition, there is still a significant gap between the open source large model llama-3.1 and the leading closed source large model in terms of trustworthy capabilities such as value alignment, security and controllability, and needs further optimization.
value alignment evaluation results
safe and controllable evaluation results
reliable ability assessment results
introduction to the "greater bay area generative artificial intelligence security development joint laboratory"
the "joint laboratory for the security development of generative artificial intelligence in the greater bay area" is jointly initiated by the cyberspace affairs office of the guangdong provincial committee of the communist party of china and the guangdong branch of the national internet emergency center. huawei, tencent, sun yat-sen university, the cyberspace administration of guangzhou municipal committee, shenzhen the cyberspace affairs office of the municipal party committee, the cyberspace affairs office of the dongguan municipal party committee and the shenzhen loop development agency jointly participated in the construction. the "joint laboratory" is committed to the evaluation and judgment of potential risks of artificial intelligence, forward-looking prevention and restraint guidance research, exploring governance paradigms for the reliable, controllable, and safe development of artificial intelligence, actively serving the innovative development of generative artificial intelligence, and strongly supporting the era of artificial intelligence. the construction of a comprehensive network management system will jointly promote artificial intelligence to "put people first and act for good", and strive to help higher-quality development of the digital economy with a high level of security.
nanfang.com, guangdong study reporter he minhui
