From Sim-to-Real: Toward General Event-based Low-light Frame Interpolation with Per-scene Optimization

Ziran Zhang1,2,*, Yongrui Ma3,2,*, Yueting Chen1, Feng Zhang2, Jinwei Gu3, Tianfan Xue3,†, Shi Guo2,†
1Zhejiang University, 2Shanghai AI Laboratory, 3The Chinese University of Hong Kong

*Authors contributed equally. †Corresponding authors.

Paper arXiv Video Code Data
Teaser Image

the interpolated result from the real-captured rgb-event sequence under low light conditions. our proposed per-scene optimization method can successfully correct the impact of event latency, accurately interpolate the correct positions, and produce visually pleasing interpolation results.

Abstract

Video Frame Interpolation (VFI) is important for video enhancement, frame rate up-conversion, and slow-motion generation. The introduction of event cameras, which capture per-pixel brightness changes asynchronously, has significantly enhanced VFI capabilities, particularly for high-speed, nonlinear motions. However, these event-based methods encounter challenges in low-light conditions, notably trailing artifacts and signal latency, which hinder their direct applicability and generalization. Addressing these issues, we propose a novel per-scene optimization strategy tailored for low-light conditions. This approach utilizes the internal statistics of a sequence to handle degraded event data under low-light conditions, improving the generalizability to different lighting and camera settings. To evaluate its robustness in low-light condition, we further introduce EVFI-LL, a unique RGB+Event dataset captured under low-light conditions. Our results demonstrate state-of-the-art performance in low-light environments. Both the dataset and the source code will be made publicly available upon publication.

Video

BibTeX

@article{zhang2024sim,
  title={From Sim-to-Real: Toward General Event-based Low-light Frame Interpolation with Per-scene Optimization},
  author={Zhang, Ziran and Ma, Yongrui and Chen, Yueting and Zhang, Feng and Gu, Jinwei and Xue, Tianfan and Guo, Shi},
  journal={arXiv preprint arXiv:2406.08090},
  year={2024}
}