Human gait recognition is one of the most promising biometric technologies, especially for unobtrusive video surveillance and human identification from a distance. Aiming at improving recognition rate, in this paper we study gait recognition using deep learning and propose a novel method based on convolutional Long Short-Term Memory (Conv-LSTM). Firstly, we present a variation of Gait Energy Images, i.e., frame-by-frame GEI (ff-GEI), to expand the volume of available GEI data and relax the constraints of gait cycle segmentation required by existing gait recognition methods. Secondly, we demonstrate the effectiveness of ff-GEI by analyzing the cross-covariance of one person’s gait data. Then, taking use of the temporality of our human gait, we design a novel gait recognition model using Conv-LSTM. Finally, the proposed method is evaluated extensively based on the CASIA Dataset B for cross-view gait recognition, furthermore the OU-ISIR Large Population Dataset is employed to verify its generalization ability. Our experimental results show that the proposed method outperforms other algorithms based on these two datasets. The results indicate that the proposed ff-GEI model using Conv-LSTM, coupled with the new gait representation, can effectively solve the problems related to cross-view gait recognition.