news

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

Remember Segment Anything (SAM) released by Meta last year?

It is used forSegmenting an imagePreviously, for different types of objects, there were walls between the segmentation models. For example, a model that can segment pigs and dogs is unlikely to be able to segment apples and pears. . .

As soon as SAM was launched, it was known as "dividing everything". Many people called itThe field of computer vision is about to be disrupted。

It is indeed the same as its name. It can separate an object from a picture, no matter how familiar it is, and can also cut any object in the picture.

When it first came out, the commotion it caused was quite rare in the industry. Within a few days, many videos were released on B Station, teaching netizens how to deploy SAM.

Moreover, netizens have been using SAM to do many things one after another, and changing hair color is considered entry-level.

You can also give Zappa a blonde hairstyle or some dreadlocks.

She is also very good at fashion. Not only can she change her outerwear, but she can also switch her inner wear at will.

Even Jim Fan, an artificial intelligence scientist at Nvidia, uses " GPT-3 moment ” to describe it.

And just in the past few days, Meta announced that the second generation of SAM is released.

This time, not only can you edit pictures, but you can also edit videos, and its capabilities are even stronger and more flexible than the previous generation.

More importantly, this little Zuckerberg is still a good guy to the end. SAM2 is also open source and free to use. In addition, a demo has been released, and you can play it on the official website. Shichao also tried it and found that the effect is really good.

In terms of the level of detail in video editing, SAM2 can already be calledClick where you want to pick.

We first uploaded a video of "a group of fish swimming in the water", selected the target we wanted to cut out, and then we could do it directly.

And if you feel that there are too many targets to be removed, you can click the "Remove" buttonRemove the excessFor example, Shichao can accurately select the stripes of one of the fish.

Click Next to see the effect. It not only accurately identifies the markings on the specified fish, but even if it runs out of the frame, the selected markings will not run onto other fish.

If you want to isolate the person in a skateboarding video, SAM2 can also do it with ease.

Not only that, even if there isMany similar objects, SAM2 will not get confused when picking. For example, in this cup-changing game, no matter how you switch, it is within the range of SAM2.

Also, among this bunch of small balls that look similar, it can follow the ball we choose.

In this case, let's make SAM2 a little more difficult. There is the Paris Olympics recently, and everyone should be paying attention to our "national sport" table tennis competition.

Then upload a video of the battle between Xiaopang and Zhang Benzhihe to see if SAM2 can pick out the "high-speed running" ping-pong ball.

I didn't expect it to be so good.Cut the screen midway, when the ping-pong ball comes back again, SAM2 will be able to capture it accurately.

In addition, in this demo released by Meta, we can alsoDo some simple editingIn addition to changing the selected object, you can also adjust the background.

For example, in the above game, the World Super LeagueErase the backgroundIt took less than five minutes from uploading the video to exporting it.

If you think the ping pong ball is not conspicuous enough, we can alsoChange to emoticons, or give it a separate emphasis.

If it's background, we can also give itAdd text, filteror something.

However, SAM2 still has its downfalls, such as in this video, Shichao chose the front wheel of the skateboard.

But when the skateboard was out of sight and came back, SAM2 somehow scratched the rear wheel.

Also, when the speed is too fast, the video cut by SAM2 will lose details.A bit rough。

But these are not big problems. SAM2 has also taken these into consideration, so it also provides us with the opportunity to correct them and find the mistakes.Manual Adjustment。

Shichao also asked his friends who were doing post-production of the video "Chapingjun". When they cut out the video, they usually use the roto brush that comes with AE. Although it can also automatically recognize it, the first frame still needs to be cut out manually.

In addition to the rendering process, it usually takes more than ten to twenty minutes to cut out a one-minute video. If the object to be cut out is out of the picture in the middle, when you come back, you have to cut out that frame again.

In comparison, at least in terms of video editing efficiency,SAM2 It was handled very well.

Anyway, in just a few days, UPs on B Station have already started to publishCombining SAM2 with other AITutorials, videos on changing outfits, changing backgrounds, everything.

Meta previously used the first generation of SAM functions on its own social media platform INS to remove backgrounds from images. It is possible that the functions of SAM2 will be incorporated into INS in the future.

But don’t think that SAM can only be used in the fields of photo editing and video editing. It also has many uses in other fields.

For example, the SAM generation relies onIdentify imagesThe ability of artificial intelligence has already made inroads into fields such as ocean, environment and medicine.

I'm not making this up, these are what Meta said themselves. In the past year, SAM has not only been able to analyze coral reefs through images, but also use satellite images for disaster relief. Even in the medical field, it can help detect skin cancer by segmenting cell images.

Now the upgraded SAM2 can not only do all these tasks, but also come up with some new uses.

For example, it can help some video AI, autonomous drivingTraining DataMake annotations.

Let's go a step further and take a look at the video footage taken by the drone.Tracking endangered animals, during diagnosis and treatment, helpPositioning the laparoscopic cameraareas, etc., etc., are also tasks that SAM2 can perform.

Some netizens with great imagination said on X that SAM2 might make the XR device more accurate. . .

In short, at a time when generative models are developing rapidly, Meta can still draw everyone's attention to SAM, which can be regarded as an indirect proof of its strength. Shichao is also looking forward to using this function on social platforms one day.

Oh, by the way, when SAM2 was released a few days ago, Zuckerberg also came out to explain the “open source” thing.

This is not just out of kindness, but after opening it to the public, with everyone's help, it can also helpBuild your own ecosystem。

But in any case, other manufacturers should learn from this pattern.

Written by:squirrel

edit:Jiang Jiang

Art editor: Xuanxuan

Image, source：

Meta, Paris Olympics