In this thesis, we detect and track anomalies on the sidewalk using deep learning. The proposed network consists of two parts: The first part is an object detection network, namely, SSD(Single Shot MultiBox Detector) is employed to detect and classify objects, then we get the abnormal targets. The second one is to find data association of objects. The proposed model is based on the single-target tracking network SiamRPN, which assists multi-target tracking through a cyclic structure. We follow Hungarian algorithm for getting the final matching results. Both the networks are trained offline, their performance is well. The contributions of this thesis are: (1) We implement the proposed model for object recognition, classification, and tracking for multiple types of anomalies. (2) We achieve multi-target tracking by combining our object detection algorithm and single-target tracking algorithm. (3) The proposed model is a novel type of deep neural networks to achieve anomaly detection, which has not been found in previous work.