Integrating Spatial and Temporal Information for Violent Activity Detection from Video Using Deep Spiking Neural Networks

Wang, Xiang
Yang, Jie
Kasabov, Nikola K
Item type
Journal Article
Degree name
Journal Title
Journal ISSN
Volume Title

Increasing violence in workplaces such as hospitals seriously challenges public safety. However, it is time- and labor-consuming to visually monitor masses of video data in real time. Therefore, automatic and timely violent activity detection from videos is vital, especially for small monitoring systems. This paper proposes a two-stream deep learning architecture for video violent activity detection named SpikeConvFlowNet. First, RGB frames and their optical flow data are used as inputs for each stream to extract the spatiotemporal features of videos. After that, the spatiotemporal features from the two streams are concatenated and fed to the classifier for the final decision. Each stream utilizes a supervised neural network consisting of multiple convolutional spiking and pooling layers. Convolutional layers are used to extract high-quality spatial features within frames, and spiking neurons can efficiently extract temporal features across frames by remembering historical information. The spiking neuron-based optical flow can strengthen the capability of extracting critical motion information. This method combines their advantages to enhance the performance and efficiency for recognizing violent actions. The experimental results on public datasets demonstrate that, compared with the latest methods, this approach greatly reduces parameters and achieves higher inference efficiency with limited accuracy loss. It is a potential solution for applications in embedded devices that provide low computing power but require fast processing speeds.

deep learning , optical flow , spatial and temporal analysis , spiking neural network , violence detection , deep learning , optical flow , spatial and temporal analysis , spiking neural network , violence detection , 4611 Machine Learning , 46 Information and Computing Sciences , 4605 Data Management and Data Science , 4603 Computer Vision and Multimedia Computation , Bioengineering , 0301 Analytical Chemistry , 0502 Environmental Science and Management , 0602 Ecology , 0805 Distributed Computing , 0906 Electrical and Electronic Engineering , Analytical Chemistry , 3103 Ecology , 4008 Electrical engineering , 4009 Electronics, sensors and digital hardware , 4104 Environmental management , 4606 Distributed computing and systems software
Sensors (Basel), ISSN: 1424-8220 (Print); 1424-8220 (Online), MDPI AG, 23(9), 4532-4532. doi: 10.3390/s23094532
Rights statement