Bringing Rolling Shutter Images Alive with Dual Reversed Distortion

Zhihang Zhong1,3, Mingdeng Cao1, Xiao Sun2, Zhirong Wu2, Zhongyi Zhou1, Yinqiang Zheng1, Stephen Lin2, Imari Sato1,3
1The University of Tokyo, 2Microsoft Research Asia, 3National Institute of Informatics
Paper arXiv Code

Accepted by ECCV'2022 Oral
teaser


Dual-Reversed-RS exploits a pair of images captured by dual rolling shutter (RS) cameras with reversed RS directions to extract undistorted global shutter (GS) video clips.

Synthetic Demos

demo1
demo2
demo3
demo4
demo5
demo6

Real-world Demos

demo1
demo2
demo3
demo4
demo5
demo6

Abstract

Rolling shutter (RS) distortion can be interpreted as the result of picking a row of pixels from instant global shutter (GS) frames over time during the exposure of the RS camera. This means that the information of each instant GS frame is partially, yet sequentially, embedded into the row-dependent distortion. Inspired by this fact, we address the challenging task of reversing this process, i.e., extracting undistorted GS frames from images suffering from RS distortion. However, since RS distortion is coupled with other factors such as readout settings and the relative velocity of scene elements to the camera, models that only exploit the geometric correlation between temporally adjacent images suffer from poor generality in processing data with different readout settings and dynamic scenes with both camera motion and object motion. In this paper, instead of two consecutive frames, we propose to exploit a pair of images captured by dual RS cameras with reversed RS directions for this highly challenging task. Grounded on the symmetric and complementary nature of dual reversed distortion, we develop a novel end-to-end model, IFED, to generate dual optical flow sequence through iterative learning of the velocity field during the RS time. Extensive experimental results demonstrate that IFED is superior to naive cascade schemes, as well as the state-of-the-art which utilizes adjacent RS images. Most importantly, although it is trained on a synthetic dataset, IFED is shown to be effective at retrieving GS frame sequences from real-world RS distorted images of dynamic scenes.


Video

Rolling Shutter Correction Ambiguity

Correction ambiguity of consecutive frames setup: Suppose there are two similar cylinders, one of them is tilted, as shown in GS view. Then, two RS cameras moving horizontally at the same speed but with different readout time setups can produce the same RS view. Models do not know how much correction is correct facing data beyond their training dataset.

Instead of two consecutive frames, we introduce another constraint setting that utilizes intra-frame spatial constraints of dual images taken simultaneously but with reversed distortion. Dual-RS setup can avoid ambiguity because the correct correction pose can be estimated based on the symmetry




Methodology

We propose a dual time cube as an RS prior, regress the dual velocity cube to indirectly estimate the dual optical flow for backward warping, and then through an encoder-decoder branch, efficiently merge the symmetric information to extract the potential GS frames.



Visual Results

Previous method cannot generalize to either the case of camera-only motion (the left example) or the case of object-only motion (the right example), while ours is robust to different motion patterns.



Grounded on the symmetric and complementary nature of dual reversed distortion, our method can successfully generalize to different readout settings without artifacts and undesired distortions.



Related Links

We have another interesting work that uses another kind of motion artifact, i.e., motion blur, to realize image2video: Animation from Blur: Multi-modal Blur Decomposition with Motion Guidance (Website, Code).

BibTeX

@inproceedings{zhong2022bringing,
  title={Bringing rolling shutter images alive with dual reversed distortion},
  author={Zhong, Zhihang and Cao, Mingdeng and Sun, Xiao and Wu, Zhirong and Zhou, Zhongyi and Zheng, Yinqiang and Lin, Stephen and Sato, Imari},
  booktitle={Computer Vision--ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part VII},
  pages={233--249},
  year={2022},
  organization={Springer}
}