This thesis presents a comprehensive analysis of a variety of fruit images for freshness grading using deep learning. A number of algorithms have been reviewed in this project, including YOLO for detecting region of interest with considerations of digital images, ResNet, VGG, Google Net, and AlexNet as the base networks for freshness grading feature extraction. Fruit decaying occurs in a gradual manner, this characteristic is included for freshness grading by interpreting chronologically-related fruit decaying information. The contribution of this thesis is to propose a novel neural network structure, i.e., YOLO + Regression CNNs for fruit object locating, classification, and freshness grading. Fruits as an object, its images are fed into YOLO for segmentation and regression, then for freshness grading. The results reveal that our approach outperforms linear predictive model and demonstrate its special merit.