A semi-parallel framework for greedy information-theoretic feature selection

Feature selection (FS) is a well-studied area that avoids issues related the curse of dimensionality and overfitting. FS is a preprocessing procedure that identifies the feature subset that is both relevant and non-redundant. Although FS has been driven by the exploration of “big data” and the development of high-performance computing, the implementation of scalable information-theoretic FS remains an under-explored topic. In this contribution, we revisit the greedy optimization procedure of information-theoretic filter FS and propose a semi-parallel optimizing paradigm that can provide an equivalent feature set as the greedy FS algorithms in a fraction of the time. We focus on greedy selection algorithms due to their larger computational complexity associated with a rapidly growing number of features. Our framework is benchmarked against twelve datasets, including one extremely large dataset that has more than a million features, and we show our framework can significantly speed up the process of FS while selecting nearly the same features as the state-of-the-art information-theoretic FS methods.