In this thesis, we provide an innovative approach to Facial Emotion Recognition (FER), which is use of ensemble learning with a lightweight model, mini-Xception. Compared to single lightweight models, it made a significant improvement. For a solution, we proposed an ensemble of mini-Xception models, where each expert is trained for a specific emotion and let confidence score for voting. Therefore, the expert model will transform the original multiclass task into binary tasks. We target the model to differentiate between a specific emotion and all others, facilitating the learning process. The principal innovation lies in our confidence-based voting mechanism, in which the experts “vote” based on their confidence scores rather than binary decisions. The confidence scores can be adjusted based on the strengths and weaknesses of the dataset to achieve maximum optimization based on the model.We found that these adjustments to confidence scores can be effectively applied to datasets, and we applied this method to FER2013 dataset, resulting in 72.8% accuracy. Furthermore, while we found the imbalance between emotion datasets, we introduced data augmentation methods, through oversampling positive samples to improve training effectiveness. Contrasted with the conventional mini-Xception model, our ensemble learning method showcased superior robustness, especially in ambiguous scenarios. This research project not only contributes a novel methodology to the FER domain but also devotes to promising avenues for real-time applications and devices with limited computational resources.