In this study, we present an empirical comparison of statistical models and machine learning models for daily electricity price forecasting in the New Zealand electricity market. We demonstrate the effectiveness of GARCH and SV models and their t-distribution variants when paired with feature selection techniques, including LASSO, mutual information, and recursive feature elimination. A key aspect of our study is the inclusion of a diverse set of explanatory variables in all models. We compare these models against a range of popular machine learning models, including LSTM, GRU, XGBoost, LEAR, and a four-layer DNN, where the latter two are considered benchmarks. Our results reveal that GARCH and SV models, particularly their t variants, perform exceptionally well when paired with feature selection techniques and explanatory variables. In most scenarios considered, these models outperform machine learning models when coupled with LASSO feature selection. This contribution provides a comprehensive evaluation of the performance of different models and feature selection techniques for electricity price forecasting in the New Zealand electricity market. Our best-performing model improves the symmetric mean absolute percentage error (sMAPE) and mean absolute scaled error (MASE) by 2% to 3% over the LEAR benchmark model, highlighting the practical relevance of our findings.