Fight Detection in Video Sequences Based on Multi-Stream Convolutional Neural Networks
Resumo
Surveillance has been gradually correlating itself to forensic computer technologies. The use of machine learning techniques made possible the better interpretation of human actions, as well as faster identification of anomalous event outbursts. There are many studies regarding this field of expertise. The best results reported in the literature are from works related to deep learning approaches. Therefore, this study aimed to use a deep learning model based on a multi-stream and high level hand-crafted descriptors to be able to address the issue of fight detection in videos. In this work, we focused on the use of a multi-stream of VGG-16 networks and the investigation of conceivable feature descriptors of a video's spatial, temporal, rhythmic and depth information. We validated our method in two commonly used datasets, aimed at fight detection, throughout the literature. Experimentation has demonstrated that the association of correlated information with a multi-stream strategy increased the classification of our deep learning approach, hence, the use of complementary features can yield interesting outputs that are superior than other previous studies.
Palavras-chave:
Fight detection, convolutional neural networks, video analysis
Referências
X. Sun H. Yao R. Ji X. Liu P. Xu "Unsupervised Fast Anomaly Detection in Crowds" 19th ACM International Conference on Multimedia pp. 1469-12011.
J. K. Dhillon A. K. S. Kushwaha "A Recent Survey for Human Activity Recognition based on Deep Learning Approach" Fourth International Conference on Image Information Processing pp. 1-6 2017.
V. Chandola A. Banerjee V. Kumar "Anomaly Detection: A Survey" ACM Computing Surveys vol. 41 no. 3 pp. 15 2009.
E. B. Nievas O. D. Suarez G. B. García R. Sukthankar "Violence Detection in Video using Computer Vision Techniques" International Conference on Computer Analysis of Images and Patterns pp. 332-2011.
W. Sultani C. Chen M. Shah Real-World Anomaly Detection in Surveillance Videos 2018.
J. Li X. Mao L. Chen L. Wang "Human Interaction Recognition Fusing Multiple Features of Depth Sequences" IET Computer Vision vol. 11 no. 7 pp. 560-2017.
A. Keçeli A. Kaya "Violent Activity Detection with Transfer Learning Method" Electronics Letters vol. 53 no. 15 pp. 1047-1048 2017.
W. Lejmi A. B. Khalifa M. A. Mahjoub "Fusion Strategies for Recognition of Violence Actions" IEEE/ACS 14th International Conference on Computer Systems and Applications pp. 178-2017.
T. Hassner Y. Itcher O. Kliper-Gross "Violent Flows: Real-Time Detection of Violent Crowd Behavior" IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops pp. 1-6 2012.
B. Antić B. Ommer "Video Parsing for Abnormality Detection" IEEE International Conference on Computer Vision pp. 2415-22011.
K. Stephens A. G. Bors "Group Activity Recognition on Outdoor Scenes" 13th IEEE International Conference on Advanced Video and Signal Based Surveillance pp. 59-65 2016.
N. Naikal P. Lajevardi S. S. Sastry "Joint Detection and Recognition of Human Actions in Wireless Surveillance Camera Networks" IEEE International Conference on Robotics and Automation pp. 4747-42014.
D. Du H. Qi Q. Huang W. Zeng C. Zhang "Abnormal Event Detection in Crowded Scenes based on Structural Multi-Scale Motion Interrelated Patterns" IEEE International Conference on Multimedia and Expo pp. 1-6 2013.
B. Wang M. Ye X. Li F. Zhao J. Ding "Abnormal Crowd Behavior Detection using High-Frequency and Spatio-Temporal Features" Machine Vision and Applications vol. 23 no. 3 pp. 501-2012.
O. Deniz I. Serrano G. Bueno T.-K. Kim "Fast Violence Detection in Video" International Conference on Computer Vision Theory and Applications vol. 2 pp. 478-2014.
I. S. Gracia O. D. Suarez G. B. Garcia T.-K. Kim "Fast Fight Detection" PloS One vol. 10 no. 4 pp. e01202015.
Y. Gao H. Liu X. Sun C. Wang Y. Liu "Violence Detection using Oriented Violent Flows" Image and Vision Computing vol. 48 pp. 37-41 2016.
S. Mukherjee R. Saini P. Kumar P. P. Roy D. P. Dogra B.-G. Kim "Fight Detection in Hockey Videos using Deep Network" The Journal of Multimedia Information System vol. 4 no. 4 pp. 225-2017.
E. Y. Fu M. X. Huang H. V. Leong G. Ngai "Cross-Species Learning: A Low-Cost Approach to Learning Human Fight from Animal Fight" ACM Multimedia Conference on Multimedia Conference pp. 320-2018.
E. Y. Fu H. V. Leong G. Ngai S. C. Chan "Automatic Fight Detection in Surveillance Videos" International Journal of Pervasive Computing and Communications vol. 13 no. 2 pp. 130-2017.
I. Serrano O. Deniz J. L. Espinosa-Aranda G. Bueno "Fight Recognition in Video Using Hough Forests and 2D Convolutional Neural Network" IEEE Transactions on Image Processing vol. 27 no. 10 pp. 4787-42018.
I. Serrano O. Deniz G. Bueno G. Garcia-Hernando T.-K. Kim "Spatio-Temporal Elastic Cuboid Trajectories for Efficient Fight Recognition using Hough Forests" Machine Vision and Applications vol. 29 no. 2 pp. 207-2018.
Q. Xia P. Zhang J. Wang M. Tian C. Fei "Real Time Violence Detection Based on Deep Spatio-Temporal Features" Chinese Conference on Biometric Recognition pp. 157-2018.
F. U. M. Ullah A. Ullah K. Muhammad I. U. Haq S. W. Baik "Violence Detection Using Spatiotemporal Features with 3D Convolutional Neural Network" Sensors vol. 19 no. 11 pp. 1-15 2019.
I. Febin K. Jayasree P. T. Joy "Violence Detection in Videos for an Intelligent Surveillance System using MoBSIFT and Movement Filtering Algorithm" Pattern Analysis and Applications pp. 1-13 2019.
C. Jain D. Gautam "Abnormal Behaviour Detection at Traffic Junctions using Lucas Kanade and Harris Corner Detector" 4th International Conference on Recent Advances in Information Technology pp. 1-5 2018.
A. Lowhur M. C. Chuah "Dense Optical Flow based Emotion Recognition Classifier" IEEE 12th International Conference on Mobile Ad Hoc and Sensor Systems pp. 573-2015.
C. Godard O. Mac Aodha G. J. Brostow "Unsupervised Monocular Depth Estimation with Left-Right Consistency" Conference on Computer Vision and Pattern Recognition 2017.
B. S. Torres H. Pedrini "Detection of Complex Video Events through Visual Rhythm" The Visual Computer vol. 34 no. 2 pp. 145-2018.
T. Moreira D. Menotti H. Pedrini "First-Person Action Recognition Through Visual Rhythm Texture Description" IEEE International Conference on Acoustics Speech and Signal Processing pp. 1-4 2017.
F. B. Valio H. Pedrini N. J. Leite "Fast Rotation-Invariant Video Caption Detection based on Visual Rhythm" in Iberoamerican Congress on Pattern Recognition Springer pp. 157-2011.
S. J. F. Guimarães M. Couprie A. A. Araújo N. J. Leite "Video Segmentation based on 2D Image Analysis" Pattern Recognition Letters vol. 24 no. 7 pp. 947-957 2003.
S. J. F. Guimaraes A. A. Araújo M. Couprie N. J. Leite "Video Fade Detection by Discrete Line Identification" Object Recognition Supported by User Interaction for Service Robots. IEEE pp. 1013-1016 2002.
S. J. Pan Q. Yang "A Survey on Transfer Learning" IEEE Transactions on Knowledge and Data Engineering vol. 22 no. 10 pp. 1345-12010.
J. Deng W. Dong R. Socher L.-J. Li K. Li L. Fei-Fei "ImageNet: A Large-Scale Hierarchical Image Database" IEEE Conference on Computer Vision and Pattern Recognition pp. 248-2009.
T. G. Dietterich "Ensemble Methods in Machine Learning" International Workshop on Multiple Classifier Systems pp. 1-15 2000.
K. Simonyan A. Zisserman Very Deep Convolutional Networks for Large-Scale Image Recognition 2014.
K. Soomro A. R. Zamir M. Shah UCF101: A Dataset of Human Actions Classes from Videos in the Wild 2012.
J. K. Dhillon A. K. S. Kushwaha "A Recent Survey for Human Activity Recognition based on Deep Learning Approach" Fourth International Conference on Image Information Processing pp. 1-6 2017.
V. Chandola A. Banerjee V. Kumar "Anomaly Detection: A Survey" ACM Computing Surveys vol. 41 no. 3 pp. 15 2009.
E. B. Nievas O. D. Suarez G. B. García R. Sukthankar "Violence Detection in Video using Computer Vision Techniques" International Conference on Computer Analysis of Images and Patterns pp. 332-2011.
W. Sultani C. Chen M. Shah Real-World Anomaly Detection in Surveillance Videos 2018.
J. Li X. Mao L. Chen L. Wang "Human Interaction Recognition Fusing Multiple Features of Depth Sequences" IET Computer Vision vol. 11 no. 7 pp. 560-2017.
A. Keçeli A. Kaya "Violent Activity Detection with Transfer Learning Method" Electronics Letters vol. 53 no. 15 pp. 1047-1048 2017.
W. Lejmi A. B. Khalifa M. A. Mahjoub "Fusion Strategies for Recognition of Violence Actions" IEEE/ACS 14th International Conference on Computer Systems and Applications pp. 178-2017.
T. Hassner Y. Itcher O. Kliper-Gross "Violent Flows: Real-Time Detection of Violent Crowd Behavior" IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops pp. 1-6 2012.
B. Antić B. Ommer "Video Parsing for Abnormality Detection" IEEE International Conference on Computer Vision pp. 2415-22011.
K. Stephens A. G. Bors "Group Activity Recognition on Outdoor Scenes" 13th IEEE International Conference on Advanced Video and Signal Based Surveillance pp. 59-65 2016.
N. Naikal P. Lajevardi S. S. Sastry "Joint Detection and Recognition of Human Actions in Wireless Surveillance Camera Networks" IEEE International Conference on Robotics and Automation pp. 4747-42014.
D. Du H. Qi Q. Huang W. Zeng C. Zhang "Abnormal Event Detection in Crowded Scenes based on Structural Multi-Scale Motion Interrelated Patterns" IEEE International Conference on Multimedia and Expo pp. 1-6 2013.
B. Wang M. Ye X. Li F. Zhao J. Ding "Abnormal Crowd Behavior Detection using High-Frequency and Spatio-Temporal Features" Machine Vision and Applications vol. 23 no. 3 pp. 501-2012.
O. Deniz I. Serrano G. Bueno T.-K. Kim "Fast Violence Detection in Video" International Conference on Computer Vision Theory and Applications vol. 2 pp. 478-2014.
I. S. Gracia O. D. Suarez G. B. Garcia T.-K. Kim "Fast Fight Detection" PloS One vol. 10 no. 4 pp. e01202015.
Y. Gao H. Liu X. Sun C. Wang Y. Liu "Violence Detection using Oriented Violent Flows" Image and Vision Computing vol. 48 pp. 37-41 2016.
S. Mukherjee R. Saini P. Kumar P. P. Roy D. P. Dogra B.-G. Kim "Fight Detection in Hockey Videos using Deep Network" The Journal of Multimedia Information System vol. 4 no. 4 pp. 225-2017.
E. Y. Fu M. X. Huang H. V. Leong G. Ngai "Cross-Species Learning: A Low-Cost Approach to Learning Human Fight from Animal Fight" ACM Multimedia Conference on Multimedia Conference pp. 320-2018.
E. Y. Fu H. V. Leong G. Ngai S. C. Chan "Automatic Fight Detection in Surveillance Videos" International Journal of Pervasive Computing and Communications vol. 13 no. 2 pp. 130-2017.
I. Serrano O. Deniz J. L. Espinosa-Aranda G. Bueno "Fight Recognition in Video Using Hough Forests and 2D Convolutional Neural Network" IEEE Transactions on Image Processing vol. 27 no. 10 pp. 4787-42018.
I. Serrano O. Deniz G. Bueno G. Garcia-Hernando T.-K. Kim "Spatio-Temporal Elastic Cuboid Trajectories for Efficient Fight Recognition using Hough Forests" Machine Vision and Applications vol. 29 no. 2 pp. 207-2018.
Q. Xia P. Zhang J. Wang M. Tian C. Fei "Real Time Violence Detection Based on Deep Spatio-Temporal Features" Chinese Conference on Biometric Recognition pp. 157-2018.
F. U. M. Ullah A. Ullah K. Muhammad I. U. Haq S. W. Baik "Violence Detection Using Spatiotemporal Features with 3D Convolutional Neural Network" Sensors vol. 19 no. 11 pp. 1-15 2019.
I. Febin K. Jayasree P. T. Joy "Violence Detection in Videos for an Intelligent Surveillance System using MoBSIFT and Movement Filtering Algorithm" Pattern Analysis and Applications pp. 1-13 2019.
C. Jain D. Gautam "Abnormal Behaviour Detection at Traffic Junctions using Lucas Kanade and Harris Corner Detector" 4th International Conference on Recent Advances in Information Technology pp. 1-5 2018.
A. Lowhur M. C. Chuah "Dense Optical Flow based Emotion Recognition Classifier" IEEE 12th International Conference on Mobile Ad Hoc and Sensor Systems pp. 573-2015.
C. Godard O. Mac Aodha G. J. Brostow "Unsupervised Monocular Depth Estimation with Left-Right Consistency" Conference on Computer Vision and Pattern Recognition 2017.
B. S. Torres H. Pedrini "Detection of Complex Video Events through Visual Rhythm" The Visual Computer vol. 34 no. 2 pp. 145-2018.
T. Moreira D. Menotti H. Pedrini "First-Person Action Recognition Through Visual Rhythm Texture Description" IEEE International Conference on Acoustics Speech and Signal Processing pp. 1-4 2017.
F. B. Valio H. Pedrini N. J. Leite "Fast Rotation-Invariant Video Caption Detection based on Visual Rhythm" in Iberoamerican Congress on Pattern Recognition Springer pp. 157-2011.
S. J. F. Guimarães M. Couprie A. A. Araújo N. J. Leite "Video Segmentation based on 2D Image Analysis" Pattern Recognition Letters vol. 24 no. 7 pp. 947-957 2003.
S. J. F. Guimaraes A. A. Araújo M. Couprie N. J. Leite "Video Fade Detection by Discrete Line Identification" Object Recognition Supported by User Interaction for Service Robots. IEEE pp. 1013-1016 2002.
S. J. Pan Q. Yang "A Survey on Transfer Learning" IEEE Transactions on Knowledge and Data Engineering vol. 22 no. 10 pp. 1345-12010.
J. Deng W. Dong R. Socher L.-J. Li K. Li L. Fei-Fei "ImageNet: A Large-Scale Hierarchical Image Database" IEEE Conference on Computer Vision and Pattern Recognition pp. 248-2009.
T. G. Dietterich "Ensemble Methods in Machine Learning" International Workshop on Multiple Classifier Systems pp. 1-15 2000.
K. Simonyan A. Zisserman Very Deep Convolutional Networks for Large-Scale Image Recognition 2014.
K. Soomro A. R. Zamir M. Shah UCF101: A Dataset of Human Actions Classes from Videos in the Wild 2012.
Publicado
28/10/2019
Como Citar
CARNEIRO, Sarah; SILVA, Gabriel; GUIMARAES, Silvio; PEDRINI, Helio.
Fight Detection in Video Sequences Based on Multi-Stream Convolutional Neural Networks. In: CONFERENCE ON GRAPHICS, PATTERNS AND IMAGES (SIBGRAPI), 32. , 2019, Rio de Janeiro.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2019
.
DOI: https://doi.org/10.5753/sibgrapi.2019.9806.
