news

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

Contributed by BlazeBVD Team

Quantum Bit | Public Account QbitAI

Families, eliminate"Video flickering"(For example, the screen suddenly turns white) There is a new trick!

Think back to when you were watching an old movie or a video shot on your phone and the scene occasionallyFlickering or inconsistent colorsAnd other phenomena.

To eliminate these problems, researchers from Meitu Image Research Institute, University of Chinese Academy of Sciences, and Sichuan University proposedA new algorithm。

New Algorithm“BlazeBVD”It can automatically remove flicker in videos, and the processing speed is very fast, which is said to be faster than existing methods.10 times faster。

Even better! This method doesn’t even require prior knowledge of the specific type or degree of flicker in the video.

In other words, it is"blind"It can be applied to various videos.

Now, even if the light in the shooting environment changes or the camera hardware cannot keep up, there is no need to worry. [doge]

Currently, relevant papers have beenComputer Vision ConferenceAccepted by ECCV 2024.

If you are interested, let's continue.

How to eliminate video flickering with BlazeBVD?

First, inspired by the classic STE (Scale Time Equalization for Flicker Removal), BlazeBVD introducesHistogramAuxiliary solutions.

Image histogram is defined as the distribution of pixel values ​​and is widely used in image processing to adjust the brightness or contrast of an image.

For example, the image histogram is like a statistical table, which tells us how many pixels of different brightness there are in a photo.

△ Image generated by Claude 3.5 Sonnet

STE analyzes the videoHistogram of each frameThen use a technique calledGaussian filteringThe method is used to smooth these histograms. The image frames with sudden changes in histogram distribution are preliminarily corrected to make the picture look more stable and reduce flicker.

Although STE is only effective for some minor flickers,But it verifies：

• Histograms are more concise than raw pixel data and can more effectively capture brightness changes and flicker in videos.
• By smoothing the histogram, you can reduce flickering in the video and make the video look more stable.

Therefore, using STE and histogram tips to improveBlind Video De-FlickeringThe quality and speed are feasible.

Specifically, BlazeBVD consists of three stages.

BlazeBVD three-stage detailed explanation

Just like a doctor treating a disease, BlazeBVD first examines each frame of the video.

It introducesSTECorrect the histogram sequence of video frames in the illumination space.

Then extract important information from the processed frames, such as which frames flicker most obviously(Strange Frame Set), Where the light needs to be adjustedFiltered light map, and where areas are overexposed or underexposed(Exposure picture)。

Next, BlazeBVD beginsrepair。

On the one hand, BlazeBVD uses aGlobal Flicker Removal ModuleThe GFRM tool uses the previously extracted light map to adjust the lighting of the entire video, ensuring that the brightness and color of each frame looks natural.

On the other hand, for some local areas that require special attention, such as over- or under-exposure, BlazeBVD will useLocal flicker removal module(LFRM). This module uses optical flow information (like tracking the movement of objects in a video) to restore details in these areas.

After completing this step, BlazeBVD finally performsPerfect work。

It introduces aLightweight timing network(TCM), this network is like a "beautician" of the video, ensuring that each frame has a smooth transition visually without abrupt changes.

To further improve the consistency of the videos, BlazeBVD has designed a special scoring system(Adaptive Mask Weighted Loss)The system scores each frame to ensure they are visually consistent, making the entire video look smoother and more natural.

At this point, BlazeBVD has completed the entire "diagnosis and treatment" process.

Experimental Results

So, how effective is BlazeBVD?

Directly look at existing methods and BlazeBVDIn the Blind VideoComparison of results on the de-flickering task:

Deflicker is an existing method, GT (Ground Truth) represents an ideal flicker-free video, and KL divergence represents the difference between the processed video and the ideal flicker-free video. The larger the KL value, the greater the difference.

It can be seen that BlazeBVD can restore the lighting histogram well while avoiding color artifacts and color distortion.(e.g. the arms of the men in the second column)。

Further andBaseline MethodFor quantitative comparison:

BlazeBVD scored high on PSNR (peak signal-to-noise ratio, higher values ​​indicate better video quality) and SSIM (structural similarity index, values ​​closer to 1 indicate better video quality), and low on Ewarp (lower values ​​indicate more coherent and consistent video).

In a word,BlazeBVD outperforms existing baseline methods.

To visualize this difference, BlazeBVD and the baseline methodVisual Comparisonas follows:

Ablation experimentIt also verifies the effectiveness of the modules designed by BlazeBVD:

In summaryThrough comprehensive experiments on synthetic, real, and generated videos, BlazeBVD demonstrates superior qualitative and quantitative results and is 10 times faster than the state-of-the-art model inference speed.

The relevant papers have been made public and you can learn more if you are interested.

paper:
https://arxiv.org/html/2403.06243v1