Assessing reproducibility and veracity across machine learning techniques in biomedicine: A case study using TCGA data

Ahyoung Amy Kim, Samir Rachid Zaim, Vignesh Subbian

Research output: Contribution to journalArticlepeer-review

4 Scopus citations


Background: Many studies that aim to identify gene biomarkers using statistical methods and translate them into FDA-approved drugs have faced challenges due to lack of clinical validity and methodological reproducibility. Since genomic data analysis relies heavily on these statistical learning tools more than before, it is vital to address the limitations of these computational techniques. Methods: Our study demonstrates these methodological gaps among most common statistical learning techniques used in gene expression analysis. To assess the classification ability and reproducibility of statistical learning tools for gene biomarker detection, six state-of-the-art machine learning models were trained on four different cancer data retrieved from The Cancer Genome Atlas (TCGA). Standard performance metrics including specificity, sensitivity, precision, and F1 score were evaluated to investigate the classification ability. For analysis of reproducibility, the identifiability of gene classifiers was examined by quantifying the consistency of the chosen classifier genes. Results: Among the six state-of-the-art machine learning methods, the random forest had the best classification ability overall. Very few genes were selected by multiple methods, which suggests poor identifiability and reproducibility of statistical learning methods for gene expression data. Our results demonstrated the challenges of reproducing discoveries from gene expression analysis due to the inherent differences that exist in statistical machine learning methods. Conclusion: Since statistical machine learning models can have large variations in high-dimensional settings such as analysis of gene expression data, transparent analysis procedures including data preprocessing, model parameterization, and evaluation and choice of interpretable models are required for clinical validity and utility.

Original languageEnglish (US)
Article number104148
JournalInternational Journal of Medical Informatics
StatePublished - Sep 2020
Externally publishedYes


  • Classification
  • Machine learning
  • Neoplasm
  • Reproducibility
  • TCGA

ASJC Scopus subject areas

  • Health Informatics


Dive into the research topics of 'Assessing reproducibility and veracity across machine learning techniques in biomedicine: A case study using TCGA data'. Together they form a unique fingerprint.

Cite this