[2023.12.21] An Efficient Selective Ensemble Learning with Rejection Approach for Classification

Abstract

Recent studies found that selective ensemble learning (e.g., dynamic ensemble selection) shows better predictive performance for classi- fication tasks, compared to traditional static ensemble. However, there are some limitations of the available methods, such as high computational cost and multiple restrictions in base model ranking and aggregation (especially for class-imbalanced data modeling). Besides, the current methods make predictions for all data without measuring the credibility regarding different data patterns. This paper proposes a selective ensemble learning with rejection ap- proach that aggregates base models from a different perspective. The approach introduces rejection measures to quantify base model credibility, and learns how to use the models according to their credibility on different sample patterns. It avoids the complexity in base model ranking and therefore is computationally more effi- cient than current methods. Any common evaluation metrics can be adopted in the selective ensemble strategy, which allows the developed model to handle class-imbalanced data properly. Also, a global rejection region is developed which indicates whether the ensemble model can provide reliable predictions for the targets. We implement the approach in the modeling of 12 datasets, in- cluding both class-imbalanced and class-balanced cases. Results show that the approach significantly reduces the inference time while showing promising performance, compared to 8 dynamic ensemble selection methods in the literature. Feature contributions and impacts of different rejection ratios on performance are also investigated to better demonstrate the approach.