Learning Deep Video Manifolds from Multiple Cameras for Abnormal Crowd Behavior Recognition

With a world increasing more divided and protesting becoming the norm for expressing differences in opinion, the growing concern in public security has sparked heated interest in understanding crowd behaviours from the computer science perspective. Specifically, technologies to automatically recognize abnormal crowd behaviour could provide valuable information on individual and crowd dynamics, thus opening up for wide range of public security applications such as abnormality detection, vandalism deterrence, and traffic flow management.

The key issue in abnormal crowd behaviour recognition has been to capture the dynamics of human activities in video sequences, where the smooth, nonlinear dynamics are natural characteristics in applying manifold learning methods that simulate human visual perception. However, manifold learning’s lack of explicit nonlinear mapping function limits its capacity in projecting the incoming data on the learned subspace, thus plagues further processing of the data. The limitation can be combat by integrating deep learning into the solution, with it providing natural nonlinear mapping function through simulating the physical structure and propagation of information in human visual cortex. The integrated model mimics both the way of perceiving and the architecture of the human visual system, thus is capable of providing human-like judgment in video understanding tasks.

In many real-world applications, multiple cameras from different views are deployed to monitor crowd behaviours in a certain place. In this case, data generally lie on multiple video manifolds. Even in a single video clip, multiple human activities, which are the components of the same crowd behaviour, may occur simultaneously. Moreover, the rareness of abnormal crowd behaviours makes the datasets highly imbalanced. To date, there has been neither manifold learning nor deep learning scheme to model the abnormal crowd behaviours observed from multiple cameras.

To discover compact and meaningful features from multi-camera videos for abnormal crowd behaviour recognition, this project works towards developing novel manifold learning mechanisms under the deep architecture. For the video from a single camera, we investigate a multi-label deep manifold learning method to characterize multiple human activities within a single manifold. For videos taken from different cameras, we explore the multi-label multi-view deep manifold learning strategy to extract the common semantic features hidden behind different visual appearances. In order to identify abnormal crowd behaviours, we further exploit the imbalanced multi-label multi-view deep manifold learning model. We will perform evaluations on both synthetic and public standard datasets and distribute our software freely for verification and comparison from fellow researchers