news

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

Text｜Deng Yongyi and Zhou Xinyu

Editor｜Su Jianxun

Before GPT-4o could even sit on the throne, Zuckerberg led the open source army to rush in.

As previously rumored, Meta officially released Llama 3.1 on July 23, Pacific Time, which is the most widely used and top-performing large model series in the open source field today.

Coincidentally, the day before its release, Llama 3.1 was "tragically leaked" in the developer community. In addition to model information, it also included a magnetic link to the 405B model, and developers were already playing with it enthusiastically.

The official release information of Llama 3.1 is also the same as the leaked information: there are three sizes: 8B, 70B and 405B, and the context length has been increased to 128K.

According to the benchmark data provided by Meta, the most popular 405B (405 billion parameters) is comparable to GPT-4 and Claude 3.5 in performance.

△Comparison with GPT-4 and Claude 3.5

In front of the current top models, Llama 3.1 is not afraid:

△Comparison with closed source model

△Comparison with open source models

It can be said that the release of Llama 3.1 has written a milestone in the recent heated debate between open source and closed source: the top open source model has truly joined forces with the top closed source model.

"Until now, open source large-scale language models have mostly lagged behind closed models in terms of capabilities and performance. Now, we are ushering in a new era led by open source," Meta said.

△A picture that has been popular in the AI ​​circle recently, open source models catch up with closed source models

Meta also included a 92-page paper in its official release, revealing the training details:

Llama 3.1 was trained on data from more than 15 trillion tokens, using 16,000 H100s. The pre-training data used is up to December 2023. To ensure training stability, only the Transformer model architecture was used for adjustment, rather than the popular Mixed Experts (MoE) architecture.

This also means that even when Llama 3.1 is expanded to 128K context length, it can still maintain high-quality output for short context windows. It is no longer "special for long texts" but "free to use long and short texts".

This is the only open source large model in the world that can achieve such a large training scale.

For more details, we have also introduced them in detail in the push a few hours ago, so we will not repeat them here. Please click here.

In 2024, as the scale of model training grows, developers are also wondering: Will large companies, which have paid huge training costs, continue to open source?

After all, OpenAI is a lesson for us - it once upheld the spirit of open source in the early days, but since GPT3.5 became popular and began to be commercialized, OpenAI has no longer been open source and has been ridiculed as Closed AI.

But at the release of Llama 3.1, Zuckerberg reiterated:

Carry out open source to the end!

In addition to releasing the model, Zuckerberg also published a heartfelt and idealistic open source manifesto, explaining why Meta should be open source and why open source is beneficial to developers.

It is worth noting that he believes that although the United States and China face fierce competition in AI, choosing the open source route still has more advantages than disadvantages.

Okay, Llama 3.1 is already like this, and the question is raised again: OpenAI, when will GPT-5 come out?

The following is Zuckerberg’s open letter, compiled by Intelligence Emergence:

Open Source AI Is the Path Forward

In the early days of high-performance computing, when major technology companies invested heavily in developing their own closed-source versions of Unix, it was hard to imagine any other way to develop such advanced software.

Still, open source Linux eventually caught on — initially because it allowed developers to modify its code any way they wanted, and was cheaper; over time it became more advanced, more secure, and had a broader ecosystem supporting more features than any closed-source Unix. Today, Linux is the industry-standard foundation for cloud computing and the operating systems that run most mobile devices — and we all benefit from its superior products.

I believe AI will evolve in a similar way. Today, several tech companies are developing leading closed-source models. But open source is rapidly closing the gap. Last year, Llama 2 was only comparable to older generation models that lagged behind the cutting edge. This year, Llama 3 is competitive with state-of-the-art models and ahead in some areas. Starting next year, we expect future Llamas to be the most advanced in the industry. But until then, Llama is already leading in open source, modifiability, and cost efficiency.

Today, we are taking the next step towards open source AI becoming the industry standard. We are releasing Llama 3.1 405B, the first open source industry-leading AI model, as well as new and improved Llama 3.1 70B and 8B models. In addition to having better cost/performance relative to closed source models, the fact that the 405B model is open source will make it the best choice for fine-tuning and extracting smaller models.

In addition to releasing these models, we are working with a range of companies to grow the broader ecosystem. Amazon, Databricks, and Nvidia are launching full suites of services to support developers fine-tuning and refining their own models. Innovators such as Groq (an AI chip startup) have built low-latency, low-cost inference services for all new models.

These models will be available on all major clouds, including AWS, Azure, Google, Oracle, and more. Scale.AI, Dell, Deloitte, and others are ready to help enterprises deploy Llama and train custom models with their own data. As the community grows and more companies develop new services, we can together make Llama an industry standard and bring the benefits of AI to everyone.

Meta is committed to open source AI. I will outline why I think open source is the best development stack for people, why open source Llama is good for Meta, why open source AI is good for the world, and because of this, the open source community will be around for the long haul.

Why open source AI is good for developers

When I talk to developers, CEOs, and government officials around the world, I typically hear the following themes:

We need to train, fine-tune, and refine our own models.

Every organization has different needs that are best met by models of different sizes, which are trained or fine-tuned using specific data. On-device tasks and classification tasks require smaller models, while more complex tasks require larger models.

Now you’ll be able to take state-of-the-art Llama models, continue training them using your own data, and then refine them into your optimally sized models — without us or anyone else ever seeing your data.

We need to control our own destiny and not be tied to a closed source vendor.

Many organizations don't want to depend on models they can't run and control. They don't want closed source model providers to be able to change their models, change their terms of use, or even stop providing services to them entirely. They also don't want to be locked into a single cloud that has exclusive rights to a certain model. Open source provides a compatible toolchain across an ecosystem of many companies, and you can easily switch between them.

We need to protect our data.

Many organizations handle sensitive data that needs to be protected and cannot be transferred to closed-source models via cloud APIs. Other organizations simply don't trust closed-source model providers with their data. Open source solves these problems by enabling you to run models anywhere you want. It is widely accepted that open source software is more secure because it is developed more transparently.

We need an operating model that is efficient and affordable.

Developers can run inference on Llama 3.1 405B on their own infrastructure at about 50% of the cost of using closed-source models such as GPT-4o, for both user-facing and offline inference tasks.

We are betting on an ecosystem that can become a long-term standard.

Many people see that open source is growing faster than closed source models, and they want to build their systems in an architecture that gives them the greatest long-term advantage.

Why open source AI is good for Meta

Meta's business model is to build the best experiences and services for people. To do this, we must ensure that we always have access to the best technology and not be locked into a competitor's closed source ecosystem that would limit what we can build.

One of my formative experiences was that our services were limited by what Apple allowed us to build on their platform. The way they tax developers, the arbitrary rules they apply, and all the product innovation they prevent from being released, it became clear that if we could build the best version of our products and competitors couldn't limit what we could build, then Meta and many other companies would be free to build better services for people. On a philosophical level, this is the main reason why I believe so strongly in building open source ecosystems for the next generation of computing in AI and AR/VR.

People often ask me if I’m worried about open sourcing Llama and giving up technical advantages, but I think that overlooks some important reasons:

First, to ensure we have access to the best technology and not be locked into a closed source ecosystem for the long term, Llama needs to grow into a full ecosystem of tools, efficiency improvements, silicon optimizations, and other integrations. If we are the only company using Llama, this ecosystem will not grow and we will not be any better than closed source Unix variants.

Second, I expect that as intelligence develops, competition will increase, which means that at that point in time, by open sourcing any particular model, people will not give up on the next model that has a greater advantage. Llama's path to becoming an industry standard is through consistent competition, efficiency, and open sourcing generation after generation of models.

Third, a key difference between Meta and closed-source model providers is that selling access to AI models is not our business model. This means that releasing Llama publicly does not undermine our revenue, sustainability, or ability to invest in research, as it does for closed-source providers. (This is one reason several closed-source providers have been lobbying against open source.)

Finally, Meta has a long history of open source projects and success. We have saved billions of dollars by publishing our server, network, and data center designs through open source compute projects and standardizing our supply chain on our designs. We benefit from ecosystem innovation, open source leading tools like PyTorch, React, and more. This approach has always worked for us when we stick with it for the long haul.

Why open source AI is good for the world

I believe open source is necessary for a positive future for AI. AI has more potential than any other modern technology to increase human productivity, creativity, and quality of life, and to advance medical and scientific research while accelerating economic growth.

Open source will ensure that more people around the world have access to the benefits and opportunities of AI, that power is not concentrated in the hands of a few companies, and that technology can be deployed more evenly and safely across society.

There has been debate about the safety of open source AI models, and my view is that open source AI will be safer than the alternatives. I think governments will conclude that it is in their interest to support open source because it will make the world more prosperous and safer.

The way I frame safety is that we need to protect against two types of harm: unintentional harm and intentional harm. Unintentional harm is when an AI system could cause harm even if the people running it don’t intend to do so.

For example, modern AI models could inadvertently give poor health recommendations. Or, in more futuristic scenarios, some worry that models could inadvertently replicate themselves or over-optimize for goals that harm humans. Intentional harm occurs when bad actors use AI models with the goal of causing harm.

It’s worth noting that unintentional harm encompasses most of the concerns people have about AI—from the effects AI systems will have on the billions of people who will use them to the most truly catastrophic science fiction scenarios for humanity. In this regard, open source should be much safer because the systems are more transparent and can be widely inspected.

Historically, open source software has been safer for this reason. Similarly, using Llama and its security systems (such as Llama Guard) is likely to be safer than a closed source model. As a result, most discussions about open source AI safety focus on intentional harm.

Our security process includes rigorous testing and red teaming to assess whether our models are capable of causing meaningful harm, with the goal of reducing risk before release. Because the models are open source, anyone can conduct their own testing.

We have to remember that these models are trained from information that already exists on the internet, so when considering harm, our starting point should be whether a model is more likely to cause harm than information that can be quickly retrieved from Google or other search results.

When reasoning about intentional harm, it is helpful to distinguish between what an individual or small-scale actor might do and what a large-scale actor, such as a nation-state with vast resources, might do.

At some point in the future, individual bad actors may exploit the intelligence of AI models to create entirely new harms from information available on the internet. At this point, the balance of power will be critical to AI safety.

I think it would be better to live in a world where AI is widely deployed so that the big players can check the power of the small bad guys. This is how we manage security on social networks - our more powerful AI systems identify and block threats from less sophisticated attackers who often use smaller AI systems.

More broadly, large institutions that deploy AI at scale will promote safety and stability across society. As long as everyone can use similar models — something open source facilitates — then governments and institutions with more computing resources will be able to check bad actors with less computation.

The next question is how the United States and democratic countries should respond to the threat of a country like China that has vast resources. The United States’ advantage lies in decentralization and open source innovation.

Some people believe that we must block our models to prevent China from obtaining them, but my view is that this will not work and will only put the United States and its allies at a disadvantage. Our adversaries are very good at espionage, it is relatively easy to steal models on a USB stick, and the way most technology companies operate does not make this anywhere near as difficult.

A world with only closed-source models seems most likely to result in a world where a handful of large companies plus our geopolitical adversaries have access to leading models, while startups, universities, and small businesses miss out.

Moreover, limiting American innovation to closed-source development increases the likelihood that we will not lead at all. Instead, I believe our best strategy is to build a strong open source ecosystem and have our leading companies work closely with our government and allies to ensure they can best take advantage of the latest advances and achieve a long-term sustainable first-mover advantage.

As you think about future opportunities, remember that most of today’s leading technology companies and scientific research are built on open source software. If we invest together, the next generation of companies and research will use open source AI. This includes startups that are just getting started, as well as people in universities and countries that may not have the resources to develop their own cutting-edge AI from scratch.

Most importantly, open source AI represents the world’s best opportunity to use this technology to maximize economic opportunity and security for everyone.

Let's build together

With past Llama models, Meta developed them for itself and then released them, but didn’t focus too much on building a broader ecosystem.

We’re taking a different approach with this launch. We’re building teams internally to make Llama available to as many developers and partners as possible, and we’re also actively building partnerships so that more companies in the ecosystem can also provide unique capabilities to their customers.

I believe the release of Llama 3.1 will be a turning point for the industry, where most developers start using primarily open source, and I expect this approach will only grow from here. I hope you'll join us on our journey to bring the benefits of AI to everyone in the world.

The link to obtain Llama 3.1 is: https://llama.meta.com/

MZ (Mark Zuckerberg)

Welcome to communicate

Welcome to communicate