SiamTrans: Zero-Shot Multi-Frame Image Restoration with Pre-trained Siamese Transformers

We propose a novel zero-shot multi-frame image restoration method for removing unwanted obstruction elements (such as rains, snow, and moire patterns) that vary in successive frames. It has three stages: transformer pre-training, restoration, hard patch refinement. Using the pre-trained transformers, our model is able to tell motion difference between true information obstructing elements. For we design model, termed SiamTrans, which constructed by Siamese encoders, decoders. Each temporal attention layer several self-attention layers, capture both spatial of multiple Only self-supervisedly on denoising task, SiamTrans tested different low-level vision tasks (deraining, demoireing, desnowing). Compared with related methods, achieves best performances, even outperforming those supervised learning.