2024-08-19
한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina
Text/ Tencent Technology Zhou Xiaoyan
Let me tell you something new - in the skits you watch in the future, from the front stage to behind the scenes, there may be no "people" in them.
In the traditional model, the production chain of a short drama, from script development, production to distribution, is full of "people" in every link. But now with the emergence of AI, perhaps only a director who can move the mouse in front of the screen is needed to complete the entire process.
On August 19, Kunlun Wanwei released SkyReels, the world's first AI short drama platform that integrates video and 3D models. The SkyReels platform integrates script generation, character customization, storyboards, plot, dialogue/BGM and film synthesis, allowing creators to "make a drama with one click" and easily produce high-quality AI videos. The launch of Kunlun Wanwei's AI short drama platform SkyReels marks a major technological leap in the creation of global AI short dramas.
In other words, there is no need to hire actors or find a venue. You can use the power of AI, open SkyReels, and operate the keyboard to make a short play.
AI is making short plays and is beginning to enter the "no man's land".
Going overseas to make short plays is not a new thing.
In 2022, Chinese online short drama product ReelShort was launched, and one year later it ranked first on the US iOS entertainment list. Soon after, rising stars such as ShortTV and DramaBox also continued to produce hit short dramas. Their themes are mainly domineering, love, and werewolves. ReelShort's hit short drama "The Double Life of My Billionaire Husband" includes segments such as "heroes saving beauties" and "property competition".
These short dramas are usually original local creations, which means that the overseas team needs to combine local culture and user interests to create scripts that conform to the aesthetic tastes of local users. They can accurately capture the exciting points of the plot that make the locals "addicted".
The core problem with these short plays is that they require sufficient understanding and research of local culture, have a long production cycle, and require a certain amount of cost in finding local actors.
In fact, the initial model of short dramas going overseas was very simple and crude. They usually directly translated the subtitles of Chinese local dramas and then broadcast them locally. In order to reduce the sense of disharmony, some overseas teams would use AI to optimize the lip movements of the actors in the short dramas, making it look like "actors with Chinese faces speaking lines in English." However, some viewers still found it awkward, just like when we watch European and American TV dramas in China, we see blue-eyed and blue-haired Europeans and Americans speaking Chinese in a strange tone, which makes it difficult for the audience to have a real sense of immersion.
Therefore, some overseas teams will use AI to "change the faces" of the actors in the short dramas, so that they can be more acceptable to the locals.
It can be seen that in the past, in order to make a popular short drama, the overseas teams would come up with various "tricks" and burn a lot of brain CPU.
So, in the era of big models, is there a short drama product that can break cultural differences, is not restricted by language, and use AI to solve all links?
In the same year when ReelShort launched its overseas business, Kunlun Wanwei sawAIGCBy reviewing the successful models in the mobile Internet era, it is found that the key lies in the "free + To C" model. By observing the "giant" companies that are still doing well today, it is found that many of them have achieved the extreme in a certain vertical category. Based on this, Fang Han, Chairman and CEO of Kunlun Wanwei, judged that this logic can be followed to attack the AIGC market.
Kunlun Wanwei thought of "AI UGC"Strategy, starting with high-frequency scenarios on the C-end, such as exploring unicorn portraits in segmented tracks such as online literature, short dramas, animation, and music. Among them, short dramas have also become an important option for Kunlun Wanwei's senior management. They hope to create a new species that subverts the original short drama production model.
The AI short drama platform SkyReels was also born under this logic. It is a one-stop short drama video creation platform driven by AI. From script generation, character generation, shot fusion to the final short drama generation, everything is generated by AI from beginning to end.
Overall, SkyReels acts like a director’s AI agent that can turn a simple idea into a short film.
bySoraAfter the emergence of AI video generation models represented by , many players at home and abroad have also launched large video generation models, such as Pixverse,Quick Hand, Zhiying Qingyuan, and a group of emerging directors who use these large models to produce film and television works have also emerged. Although using these models can achieve good video generation effects, if you want to make a mature film, you still have certain requirements for the professional ability of the user himself. They need to have high screenwriting skills and artistic aesthetic ability.
For example, users need to prepare the composition in advance, imagine the desired shot, and then use prompts containing professional terms toMidjourneyAfter debugging a satisfactory image on the product, import it into the video generation product to make lens adjustments, and finally export the work.
Although these tools have greatly shortened the production chain of film and television works, they are more oriented towardsPUGCSkyReels has a lower threshold for use and the entire operation process is simple and intelligent. Just input an idea and leave the rest to AI.In comparison, SkyReels, as the world's first AI short drama platform, is a UGC product with a larger audience.
(SkyReels product demo)
SkyReels is the world's first hybrid 3D engine and generative AI video creation platform. The use process is also very simple. After choosing to create a short film, enter the creative idea and select the desired video style, such as anime style, realism, Ghibli style, etc., and you can generate the script with one click. The system will automatically generate characters according to the needs of the script story. Then the user can view the automatically generated storyboard script, and then convert it into a video with one click, and can automatically convert the content into1080P 60fpsHigh-definition video;The length of a video that can be generated at one time is up to 180 secondsThis is a significant breakthrough compared to Sora, which can generate 60 seconds of video at a time, and Keling, which can generate 10 seconds of video at a time.
Overall, the complete production chain of a short drama covers script generation, character customization, storyboard generation, episode shooting and post-production, and SkyReels can improve the efficiency of this production chain with three self-developed large models.
In the content development process, the traditional short drama production method,It is to select some chapters from existing online novels and then adapt them according to the characteristics of the short drama. The screenwriter needs to be able to combine the interests of the audience and cleverly set the turning points of the plot, so as to increase the user's willingness to pay for the next chapter.
SkyReels uses its own script model in this stageSkyScript, replacing the traditional manual content development of short plays. Kunlun Wanwei first built a high-quality short play structured dataset SkyScript-100M with hundreds of millions of short plays. This dataset provides high-quality annotations for the plot rhythm, excitement points, and emotional changes of a large number of excellent short plays. In addition, the algorithm team designed a progressive script generation framework, which enables the model to output highly structured excellent scripts. Both novice and experienced screenwriters can use this model to create attractive short play scripts.
After having a moving story, the next step is to convert it into a shot-by-shot script. In the traditional production model, this step requires inviting actors, setting up scenes offline, and the director needs to communicate with the actors about the essentials before filming, which involves manpower and set costs. However, all of these can be replaced by AI big models. SkyReels uses Kunlun Wanwei in this step.Self-developed storyboard model StoryboardGen,The model reverse-processes film and television animation data to obtain high-quality industrial-grade storyboard data sets, and designs a storyboard multi-agent framework to achieve high-quality generation of scenes, shots, characters, and actions. In terms of image generation effects, Kunlun Wanwei uses the self-developed DiT-Moe model to improve image quality.
Just like in the formal shooting process, the storyboards need to be organically integrated together to obtain a complete story system.SkyReels uses WorldEngine, the industry's first innovative platform that deeply integrates AI 3D engine and video large models, realizing 3D generation, video generation, and 3D and video hybrid creation.。
The WorldEngine innovation platform consists of multiple sub-links, each of which uses a different model.
In the 3D generation module,The self-developed Sky3DGen large model is used, which can generate a variety of 3D elements and scenes, from natural landscapes to future cities, from static objects to dynamic characters, and provides real-time 3D scene interaction capabilities, allowing users to instantly see the effects of adjustments and editing, speeding up the creation process. With the support of WorldEngine, users can build a 3D virtual studio and realize a virtual shooting experience.
(Intelligent generation of 3D scenes)
In the 3D and video hybrid creation module, Kunlun Wanwei uses layer fusion technology as the core to create a 3D and video hybrid creation mode. Any element in the video can be freely generated using the 3D AI engine or the video model. In addition, Kunlun Wanwei has trained a fusion generation model for the details of lighting and style differences encountered in the mix, so that the video after layer fusion presents better consistency.
(Mixed generation video)
Thanks to the use of the engine, WorldEngine has achieved cost reduction compared to traditional video generation, while the generation speed has also increased by several orders of magnitude.
Although a short video has basically been generated here, people who often watch TV series will also feel that good dialogue skills and BGM can achieve twice the result with half the effort in presenting the plot.
In terms of dialogue, SkyReels' AI text understanding technology, after deep learning from tens of thousands of novels, can automatically identify character dialogues and emotional expressions from texts, and predict reasonable pauses between sentences. In terms of narration, SkyReels uses emotional speech synthesis technology to help narration use ups and downs in rhythm to make character dialogues full of rich emotions.
Kunlun Wanwei revealed that SkyReels's intelligent agent can not only understand text, but also intelligently analyze the atmosphere of the scenes in the script. "Based on advanced multimodal retrieval and background music (BGM) generation capabilities, we can intelligently match the video with reasonable background music, so that every frame of the picture is perfectly integrated with the music."
SkyReels, Kunlun Wanwei's AI short drama platform, is a revolutionary tool that is expected to bring about explosive growth in AI short drama user-generated content (UGC) and professional user-generated content (PUGC), injecting unprecedented vitality into the short drama content ecosystem and promoting further rapid growth in the short drama content creation and consumption market. An era of short drama creation and consumption driven by AI technology is coming.
Guohai Securities Research Report mentioned that AI is expected to increase the production capacity of short dramas and reduce costs. The overseas market space for short dramas is conservatively estimated to be about US$36 billion.
SkyReels is a microcosm of Kunlun Wanwei's overseas expansion strategy, and behind it is their strategic thinking of All in AIGC.
In 2020, Zhou Yahui, the founder of Kunlun Wanwei, saw the potential of AIGC. They predicted that the AI system would usher in a golden age of new content.
Fang Han, Chairman and CEO of Kunlun Wanwei, reviewed the opportunities in the mobile Internet era and summarized the patterns of popular content products. He found that most of them are "free + To C" models. Because the group base is large enough, as long as the product can hit the user's needs, it can achieve exponential user growth. Building an "AI UGC" platform has become Kunlun Wanwei's core strategy, that is, to provide services to 1% of content creators and let 99% of users consume content.
So, what is the product portrait that has become popular in the AIGC era?
Fang Han believes that when new hardware and new mobile phones that are compatible with AI are born, that may be the time when AIGC will achieve disruptive results, but this process will still take 3 to 5 years, and the current core task is to find the right opportunity, make predictions and occupy the cusp of AIGC.
According to the rules of the mobile Internet era, each country may have its own super app, or a super app may be divided into multiple countries at the same time. After some thought, Fang Han believes that it should first occupy countries with high cost-effectiveness and user acceptance, such as Europe, the United States, Japan, South Korea and China, and then gradually spread to the Middle East, Southeast Asia, South America and other regions.
After narrowing down the countries, Kunlun Wanwei’s management began to think about the vertical categories that could give birth to super apps. They kept asking themselves, “How can we become a giant?” The content giants that are still alive from the last era have basically been deeply involved in a certain vertical entertainment category, such as online literature, short dramas, and animations. The same logic will be repeated on AIGC.
At present, Kunlun Wanwei has carried out a systematic layout for the "AI UGC" strategy in China, and has built a product matrix in large models, AI search, AI music and other sectors.
Among them, Kunlun Wanwei's large model has gone through many iterations:
New version released in FebruaryMoEThe large language model "Tiangong 2.0" and the new version of the "Tiangong AI Smart Assistant" APP;
In April, the “Tiangong 3.0” pedestal model and the “Tiangong SkyMusic” music model were officially launched for public testing;
In June, Kunlun Wanwei open-sourced Skywork-MoE, a 200 billion sparse model belonging to "Tiangong 3.0".OpenAIWhen announcing the blocking of API traffic from non-supported countries and regions, Tiangong Open Platform launched a "one-click migration plan" for OpenAl API users and provided 150 million tokens of API call resources, including Tiangong 2.0 and Tiangong 3.0 large models.
While iterating the big model, AI search has also become Fang Han's key focus. Kunlun Wanwei started to deploy AI search one year earlier than OpenAI's SearchGPT, and upgraded its search product this year, supporting searches for text, images, videos and other sections. Users can present the results in the form of mind maps, tables, outlines, etc., and can save outlines and mind maps to directly generate PPTs.
In addition to the product level, Kunlun Wanweiye has collaborated with international universities to open source the digital agent development toolkit AgentStudio, providing researchers and developers with a comprehensive platform covering the entire agent development process.
Whether at home or abroad, each language system needs to have its own set of products. Kunlun Wanwei believes that the global dividend period of the AIGC platform is about to come.
Collect and organize the latest news information and make it available for free for everyone to read
