Abstract
This article presents a study of three validation metrics used for the selection of optimal parameters of a support vector machine (SVM) classifier in the case of non-separable and unbalanced datasets. This situation is often encountered when the data is obtained experimentally or clinically. The three metrics selected in this work are the area under the ROC curve (AUC), accuracy, and balanced accuracy. These validation metrics are tested using computational data only, which enables the creation of fully separable sets of data. This way, non-separable datasets, representative of a real-world problem, can be created by projection onto a lower dimensional sub-space. The knowledge of the separable dataset, unknown in real-world problems, provides a reference to compare the three validation metrics using a quantity referred to as the “weighted likelihood”. As an application example, the study investigates a classification model for hip fracture prediction. The data is obtained from a parameterized finite element model of a femur. The performance of the various validation metrics is studied for several levels of separability, ratios of unbalance, and training set sizes.
Original language | English (US) |
---|---|
Pages (from-to) | 523-535 |
Number of pages | 13 |
Journal | Structural and Multidisciplinary Optimization |
Volume | 50 |
Issue number | 4 |
DOIs | |
State | Published - Oct 2014 |
Keywords
- Cross validation
- Non-separable and unbalanced datasets
- Support vector machines
- Validation metrics
ASJC Scopus subject areas
- Software
- Control and Systems Engineering
- Computer Science Applications
- Computer Graphics and Computer-Aided Design
- Control and Optimization