Fruit freshness grading is an innate ability of humans. However, there was not much work focusing on creating a fruit grading system based on digital images in deep learning. The algorithm proposed in this article has the potentiality to be employed so as to avoid wasting fruits or save fruits from throwing away. In this article, we present a comprehensive analysis of freshness grading scheme using computer vision and deep learning. Our scheme for grading is based on visual analysis of digital images. Numerous deep learning methods are exploited in this project, including ResNet, VGG, and GoogLeNet. AlexNet is selected as the base network, and YOLO is employed for extracting the region of interest (ROI) from digital images. Therefore, we construct a novel neural network model for fruit detection and freshness grading regarding multiclass fruit classification. The fruit images are fed into our model for training, AlexNet took the leading position; meanwhile, VGG scheme performed the best in the validation.