OVST: online video stabilization with two-stage training transformer

Video stabilization aims to mitigate or eliminate the shake presented within video frames. Existing online video stabilization technologies rely on information from future frames, which may introduce a lag during real-time video stabilization. To surmount this hurdle, an online video stabilization model called OVST is proposed, which leverages solely historical video frames and enhanced by an attention mechanism. To simplify the complexity of model training and enhance robustness, a two-stage training strategy is proposed to decouple the fitting of real poses and the stabilization of virtual poses, and a hybrid stabilization loss with interframe soft constraints is designed, which effectively regulates the changes in camera poses between adjacent frames through interframe displacement, angular distortion, and cropping rate, thereby suppressing the distortion effects caused by excessive pose smoothing while balancing stability and cropping rate. Experiments demonstrate the superiority of the proposed OVST method over existing state-of-the-art techniques, achieving a stability metric of 0.8878 and a distortion metric of 0.9870.