Statistical inference for time course RNA-Seq data using a negative binomial mixed-effect model

Xiaoxiao Sun, David Dalpiaz, Di Wu, Jun S. Liu, Wenxuan Zhong, Ping Ma

Research output: Contribution to journalArticlepeer-review

15 Scopus citations


Background: Accurate identification of differentially expressed (DE) genes in time course RNA-Seq data is crucial for understanding the dynamics of transcriptional regulatory network. However, most of the available methods treat gene expressions at different time points as replicates and test the significance of the mean expression difference between treatments or conditions irrespective of time. They thus fail to identify many DE genes with different profiles across time. In this article, we propose a negative binomial mixed-effect model (NBMM) to identify DE genes in time course RNA-Seq data. In the NBMM, mean gene expression is characterized by a fixed effect, and time dependency is described by random effects. The NBMM is very flexible and can be fitted to both unreplicated and replicated time course RNA-Seq data via a penalized likelihood method. By comparing gene expression profiles over time, we further classify the DE genes into two subtypes to enhance the understanding of expression dynamics. A significance test for detecting DE genes is derived using a Kullback-Leibler distance ratio. Additionally, a significance test for gene sets is developed using a gene set score. Results: Simulation analysis shows that the NBMM outperforms currently available methods for detecting DE genes and gene sets. Moreover, our real data analysis of fruit fly developmental time course RNA-Seq data demonstrates the NBMM identifies biologically relevant genes which are well justified by gene ontology analysis. Conclusions: The proposed method is powerful and efficient to detect biologically relevant DE genes and gene sets in time course RNA-Seq data.

Original languageEnglish (US)
Article number324
JournalBMC bioinformatics
Issue number1
StatePublished - Aug 26 2016
Externally publishedYes


  • Analysis of variance
  • Differentially expressed gene
  • Gene set enrichment
  • Penalized likelihood
  • Smoothing spline

ASJC Scopus subject areas

  • Structural Biology
  • Biochemistry
  • Molecular Biology
  • Computer Science Applications
  • Applied Mathematics


Dive into the research topics of 'Statistical inference for time course RNA-Seq data using a negative binomial mixed-effect model'. Together they form a unique fingerprint.

Cite this