STB-VMM: Swin Transformer Based Video Motion Magnification

Authors: Ricard Lado-Roigé, Marco A. Pérez

Published: 2023-02-20 14:21:56+00:00

Comment: Code available at: https://github.com/RLado/STB-VMM

Journal Ref: Knowl.-Based Syst. 269 (2023) 110493

AI Summary

This paper introduces STB-VMM, a novel video motion magnification model based on the Swin Transformer, designed to overcome the common issues of noisy and blurry outputs in existing techniques. The proposed architecture achieves state-of-the-art performance by offering better tolerance to noisy inputs and producing higher-quality, clearer magnified video sequences with fewer artifacts than prior art. This advancement enables more precise measurements for applications relying on magnified video.

Abstract

The goal of video motion magnification techniques is to magnify small motions in a video to reveal previously invisible or unseen movement. Its uses extend from bio-medical applications and deepfake detection to structural modal analysis and predictive maintenance. However, discerning small motion from noise is a complex task, especially when attempting to magnify very subtle, often sub-pixel movement. As a result, motion magnification techniques generally suffer from noisy and blurry outputs. This work presents a new state-of-the-art model based on the Swin Transformer, which offers better tolerance to noisy inputs as well as higher-quality outputs that exhibit less noise, blurriness, and artifacts than prior-art. Improvements in output image quality will enable more precise measurements for any application reliant on magnified video sequences, and may enable further development of video motion magnification techniques in new technical fields.

Key findings

The STB-VMM model significantly outperforms previous state-of-the-art learning-based methods, achieving 9.63% higher average MUSIQ scores. It produces clearer, less blurry, and less noisy magnified video outputs with superior edge stability, demonstrating improved tolerance to noisy inputs. However, the model requires more computational resources, nearly doubling the compute time compared to prior methods.

Approach

The authors propose STB-VMM, a learning-based Eulerian approach composed of a feature extractor, a manipulator, and a reconstructor. It utilizes Residual Swin Transformer Blocks (RSTB) in its shallow and deep feature extractors to process two input frames. The manipulator then computes and magnifies the difference between the feature spaces, which is finally reconstructed into a magnified frame, notably abandoning temporal filtering used in previous methods.

Datasets

Synthetic dataset by Oh et al. (derived from PASCAL VOC and MS COCO), custom benchmark sequences (e.g., AC00, Car00, Building00), and the 'baby' sequence from Wu et al. [7] for evaluation.

Model(s)

STB-VMM (Swin Transformer Based Video Motion Magnification), which incorporates Residual Swin Transformer Blocks (RSTB) and Swin Transformer Layers (STL).

Author countries

Spain

← Previous