CrossNet++: Cross-Scale Large-Parallax Warping for Reference-Based Super-Resolution

Yang Tan, Haitian Zheng, Yinheng Zhu, Xiaoyun Yuan, Xing Lin, David Brady, Lu Fang

Research output: Contribution to journalArticlepeer-review

19 Scopus citations

Abstract

The ability of camera arrays to efficiently capture higher space-bandwidth product than single cameras has led to various multiscale and hybrid systems. These systems play vital roles in computational photography, including light field imaging, 360 VR camera, gigapixel videography, etc. One of the critical tasks in multiscale hybrid imaging is matching and fusing cross-resolution images from different cameras under perspective parallax. In this paper, we investigate the reference-based super-resolution (RefSR) problem associated with dual-camera or multi-camera systems. RefSR consists of super-resolving a low-resolution (LR) image given an external high-resolution (HR) reference image, where they suffer both a significant resolution gap ($8times$8×) and large parallax ($sim 10%$∼10% pixel displacement). We present CrossNet++, an end-to-end network containing novel two-stage cross-scale warping modules, image encoder and fusion decoder. The stage I learns to narrow down the parallax distinctively with the strong guidance of landmarks and intensity distribution consensus. Then the stage II operates more fine-grained alignment and aggregation in feature domain to synthesize the final super-resolved image. To further address the large parallax, new hybrid loss functions comprising warping loss, landmark loss and super-resolution loss are proposed to regularize training and enable better convergence. CrossNet++ significantly outperforms the state-of-art on light field datasets as well as real dual-camera data. We further demonstrate the generalization of our framework by transferring it to video super-resolution and video denoising.

Original languageEnglish (US)
Pages (from-to)4291-4305
Number of pages15
JournalIEEE Transactions on Pattern Analysis and Machine Intelligence
Volume43
Issue number12
DOIs
StatePublished - Dec 1 2021
Externally publishedYes

Keywords

  • Reference-based super-resolution
  • camera array
  • image synthesis
  • image warping
  • light field imaging
  • optical flow

ASJC Scopus subject areas

  • Software
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics
  • Artificial Intelligence
  • Applied Mathematics

Fingerprint

Dive into the research topics of 'CrossNet++: Cross-Scale Large-Parallax Warping for Reference-Based Super-Resolution'. Together they form a unique fingerprint.

Cite this