Skin cancer classification is a complex and time-consuming task. Existing approaches use segmentation to improve accuracy and efficiency, but due to different sizes and shapes of lesions, segmentation is not a suitable approach. In this research study, we proposed an improved automated system based on hybrid and optimal feature selections. Firstly, we balanced our dataset by applying three different transformation techniques, which include brightness, sharpening, and contrast enhancement. Secondly, we retrained two CNNs, Darknet53 and Inception V3, using transfer learning. Thirdly, the retrained models were used to extract deep features from the dataset. Lastly, optimal features were selected using moth flame optimization (MFO) to overcome the curse of dimensionality. This helped us in improving accuracy and efficiency of our model. We achieved 95.9%, 95.0%, and 95.8% on cubic SVM, quadratic SVM, and ensemble subspace discriminants, respectively. We compared our technique with state-of-the-art approach.