With the decreased costs of security monitoring equipment such as cameras, video surveillance has been broadly applied to our communities and public places. However, at present most of the surveillance systems acquire anomalies and visual evidences only through video playback. Hence, it is necessary to develop the methods of real-time human behavior recognition so as to reduce security staff’s workload and improve work efficiency. The existing work needs feature extraction from the video frames to detect human body and achieve human behavior recognition. In this thesis, our focus is on the state-of-the-art methods for human behavior recognition based on deep learning. Since deep learning methods have been well investigated in the past decades, as an end-to-end computational method, it simplifies feature extraction as the operations in a black box. In this thesis, we explore and exploit the state-of-the-art methods, which are utilized for human behavior recognition. More importantly, in order to attain our goal, spatiotemporal information was collected and employed to the implementation of our research project. We firstly adopted ensemble learning with deep learning methods. We proposed Selective Kernel Network (SKNet) and ResNeXt with attention mechanism, which generate positive results to recognize human behaviours. The contributions of this thesis are: (1) The ResNeXt and SKNet with attention mechanism make the best accuracy of overall human behavior recognition at the rate up to 98.7% based on public datasets; (2) The YOLOv3 + LSTM network to reply on both spatiotemporal information with class score fusion is able to achieve 97.58% accuracy based on our dataset for sign language processing.