Video summarisation by classification with deep reinforcement learning

Kaiyang Zhou, Tao Xiang, Andrea Cavallaro

Research output: Contribution to conferenceConference paperpeer-review

6 Citations (Scopus)


Most existing video summarisation methods are based on either supervised or unsupervised learning. In this paper, we propose a reinforcement learning-based weakly supervised method that exploits easy-to-obtain, video-level category labels and encourages summaries to contain category-related information and maintain category recognisability. Specifically, We formulate video summarisation as a sequential decision-making process and train a summarisation network with deep Q-learning (DQSN). A companion classification network is also trained to provide rewards for training the DQSN. With the classification network, we develop a global recognisability reward based on the classification result. Critically, a novel dense ranking-based reward is also proposed in order to cope with the temporally delayed and sparse reward problems for long sequence reinforcement learning. Extensive experiments on two benchmark datasets show that the proposed approach achieves state-of-the-art performance.

Original languageEnglish
Publication statusPublished - Sept 2018
Externally publishedYes
Event29th British Machine Vision Conference, BMVC 2018 - Newcastle, United Kingdom
Duration: 3 Sept 20186 Sept 2018


Conference29th British Machine Vision Conference, BMVC 2018
Country/TerritoryUnited Kingdom
Internet address

Scopus Subject Areas

  • Computer Vision and Pattern Recognition


Dive into the research topics of 'Video summarisation by classification with deep reinforcement learning'. Together they form a unique fingerprint.

Cite this