Large-scale benchmarking of circRNA detection tools reveals large differences in sensitivity but not in precision

Marieke Vromman; Jasper Anckaert; Stefania Bortoluzzi; Alessia Buratin; Chia Ying Chen; Qinjie Chu; Trees Juen Chuang; Roozbeh Dehghannasiri; Christoph Dieterich; Xin Dong; Paul Flicek; Enrico Gaffo; Wanjun Gu; Chunjiang He; Steve Hoffmann; Osagie Izuogu; Michael S. Jackson; Tobias Jakobi; Eric C. Lai; Justine Nuytens; Julia Salzman; Mauro Santibanez-Koref; Peter Stadler; Olivier Thas; Eveline Vanden Eynde; Kimberly Verniers; Guoxia Wen; Jakub Westholm; Li Yang; Chu Yu Ye; Nurten Yigit; Guo Hua Yuan; Jinyang Zhang; Fangqing Zhao; Jo Vandesompele; Pieter Jan Volders

doi:10.1038/s41592-023-01944-6

Large-scale benchmarking of circRNA detection tools reveals large differences in sensitivity but not in precision

Marieke Vromman, Jasper Anckaert, Stefania Bortoluzzi, Alessia Buratin, Chia Ying Chen, Qinjie Chu, Trees Juen Chuang, Roozbeh Dehghannasiri, Christoph Dieterich, Xin Dong, Paul Flicek, Enrico Gaffo, Wanjun Gu, Chunjiang He, Steve Hoffmann, Osagie Izuogu, Michael S. Jackson, Tobias Jakobi, Eric C. Lai, Justine NuytensJulia Salzman, Mauro Santibanez-Koref, Peter Stadler, Olivier Thas, Eveline Vanden Eynde, Kimberly Verniers, Guoxia Wen, Jakub Westholm, Li Yang, Chu Yu Ye, Nurten Yigit, Guo Hua Yuan, Jinyang Zhang, Fangqing Zhao, Jo Vandesompele, Pieter Jan Volders

Campus Administration, Phoenix

Research output: Contribution to journal › Article › peer-review

18 Scopus citations

Abstract

The detection of circular RNA molecules (circRNAs) is typically based on short-read RNA sequencing data processed using computational tools. Numerous such tools have been developed, but a systematic comparison with orthogonal validation is missing. Here, we set up a circRNA detection tool benchmarking study, in which 16 tools detected more than 315,000 unique circRNAs in three deeply sequenced human cell types. Next, 1,516 predicted circRNAs were validated using three orthogonal methods. Generally, tool-specific precision is high and similar (median of 98.8%, 96.3% and 95.5% for qPCR, RNase R and amplicon sequencing, respectively) whereas the sensitivity and number of predicted circRNAs (ranging from 1,372 to 58,032) are the most significant differentiators. Of note, precision values are lower when evaluating low-abundance circRNAs. We also show that the tools can be used complementarily to increase detection sensitivity. Finally, we offer recommendations for future circRNA detection and validation.

Original language	English (US)
Pages (from-to)	1159-1169
Number of pages	11
Journal	Nature Methods
Volume	20
Issue number	8
DOIs	https://doi.org/10.1038/s41592-023-01944-6
State	Published - Aug 2023

ASJC Scopus subject areas

Biotechnology
Biochemistry
Molecular Biology
Cell Biology

Access to Document

10.1038/s41592-023-01944-6

Cite this

Vromman, M., Anckaert, J., Bortoluzzi, S., Buratin, A., Chen, C. Y., Chu, Q., Chuang, T. J., Dehghannasiri, R., Dieterich, C., Dong, X., Flicek, P., Gaffo, E., Gu, W., He, C., Hoffmann, S., Izuogu, O., Jackson, M. S., Jakobi, T., Lai, E. C., ... Volders, P. J. (2023). Large-scale benchmarking of circRNA detection tools reveals large differences in sensitivity but not in precision. Nature Methods, 20(8), 1159-1169. https://doi.org/10.1038/s41592-023-01944-6

Vromman, M, Anckaert, J, Bortoluzzi, S, Buratin, A, Chen, CY, Chu, Q, Chuang, TJ, Dehghannasiri, R, Dieterich, C, Dong, X, Flicek, P, Gaffo, E, Gu, W, He, C, Hoffmann, S, Izuogu, O, Jackson, MS, Jakobi, T, Lai, EC, Nuytens, J, Salzman, J, Santibanez-Koref, M, Stadler, P, Thas, O, Vanden Eynde, E, Verniers, K, Wen, G, Westholm, J, Yang, L, Ye, CY, Yigit, N, Yuan, GH, Zhang, J, Zhao, F, Vandesompele, J & Volders, PJ 2023, 'Large-scale benchmarking of circRNA detection tools reveals large differences in sensitivity but not in precision', Nature Methods, vol. 20, no. 8, pp. 1159-1169. https://doi.org/10.1038/s41592-023-01944-6

@article{7851c1ca8d9644ab9c6e683c6b130c30,

title = "Large-scale benchmarking of circRNA detection tools reveals large differences in sensitivity but not in precision",

abstract = "The detection of circular RNA molecules (circRNAs) is typically based on short-read RNA sequencing data processed using computational tools. Numerous such tools have been developed, but a systematic comparison with orthogonal validation is missing. Here, we set up a circRNA detection tool benchmarking study, in which 16 tools detected more than 315,000 unique circRNAs in three deeply sequenced human cell types. Next, 1,516 predicted circRNAs were validated using three orthogonal methods. Generally, tool-specific precision is high and similar (median of 98.8%, 96.3% and 95.5% for qPCR, RNase R and amplicon sequencing, respectively) whereas the sensitivity and number of predicted circRNAs (ranging from 1,372 to 58,032) are the most significant differentiators. Of note, precision values are lower when evaluating low-abundance circRNAs. We also show that the tools can be used complementarily to increase detection sensitivity. Finally, we offer recommendations for future circRNA detection and validation.",

author = "Marieke Vromman and Jasper Anckaert and Stefania Bortoluzzi and Alessia Buratin and Chen, {Chia Ying} and Qinjie Chu and Chuang, {Trees Juen} and Roozbeh Dehghannasiri and Christoph Dieterich and Xin Dong and Paul Flicek and Enrico Gaffo and Wanjun Gu and Chunjiang He and Steve Hoffmann and Osagie Izuogu and Jackson, {Michael S.} and Tobias Jakobi and Lai, {Eric C.} and Justine Nuytens and Julia Salzman and Mauro Santibanez-Koref and Peter Stadler and Olivier Thas and {Vanden Eynde}, Eveline and Kimberly Verniers and Guoxia Wen and Jakub Westholm and Li Yang and Ye, {Chu Yu} and Nurten Yigit and Yuan, {Guo Hua} and Jinyang Zhang and Fangqing Zhao and Jo Vandesompele and Volders, {Pieter Jan}",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive licence to Springer Nature America, Inc.",

year = "2023",

month = aug,

doi = "10.1038/s41592-023-01944-6",

language = "English (US)",

volume = "20",

pages = "1159--1169",

journal = "Nature Methods",

issn = "1548-7091",

publisher = "Public Library of Science",

number = "8",

}

TY - JOUR

T1 - Large-scale benchmarking of circRNA detection tools reveals large differences in sensitivity but not in precision

AU - Vromman, Marieke

AU - Anckaert, Jasper

AU - Bortoluzzi, Stefania

AU - Buratin, Alessia

AU - Chen, Chia Ying

AU - Chu, Qinjie

AU - Chuang, Trees Juen

AU - Dehghannasiri, Roozbeh

AU - Dieterich, Christoph

AU - Dong, Xin

AU - Flicek, Paul

AU - Gaffo, Enrico

AU - Gu, Wanjun

AU - He, Chunjiang

AU - Hoffmann, Steve

AU - Izuogu, Osagie

AU - Jackson, Michael S.

AU - Jakobi, Tobias

AU - Lai, Eric C.

AU - Nuytens, Justine

AU - Salzman, Julia

AU - Santibanez-Koref, Mauro

AU - Stadler, Peter

AU - Thas, Olivier

AU - Vanden Eynde, Eveline

AU - Verniers, Kimberly

AU - Wen, Guoxia

AU - Westholm, Jakub

AU - Yang, Li

AU - Ye, Chu Yu

AU - Yigit, Nurten

AU - Yuan, Guo Hua

AU - Zhang, Jinyang

AU - Zhao, Fangqing

AU - Vandesompele, Jo

AU - Volders, Pieter Jan

PY - 2023/8

Y1 - 2023/8

N2 - The detection of circular RNA molecules (circRNAs) is typically based on short-read RNA sequencing data processed using computational tools. Numerous such tools have been developed, but a systematic comparison with orthogonal validation is missing. Here, we set up a circRNA detection tool benchmarking study, in which 16 tools detected more than 315,000 unique circRNAs in three deeply sequenced human cell types. Next, 1,516 predicted circRNAs were validated using three orthogonal methods. Generally, tool-specific precision is high and similar (median of 98.8%, 96.3% and 95.5% for qPCR, RNase R and amplicon sequencing, respectively) whereas the sensitivity and number of predicted circRNAs (ranging from 1,372 to 58,032) are the most significant differentiators. Of note, precision values are lower when evaluating low-abundance circRNAs. We also show that the tools can be used complementarily to increase detection sensitivity. Finally, we offer recommendations for future circRNA detection and validation.

AB - The detection of circular RNA molecules (circRNAs) is typically based on short-read RNA sequencing data processed using computational tools. Numerous such tools have been developed, but a systematic comparison with orthogonal validation is missing. Here, we set up a circRNA detection tool benchmarking study, in which 16 tools detected more than 315,000 unique circRNAs in three deeply sequenced human cell types. Next, 1,516 predicted circRNAs were validated using three orthogonal methods. Generally, tool-specific precision is high and similar (median of 98.8%, 96.3% and 95.5% for qPCR, RNase R and amplicon sequencing, respectively) whereas the sensitivity and number of predicted circRNAs (ranging from 1,372 to 58,032) are the most significant differentiators. Of note, precision values are lower when evaluating low-abundance circRNAs. We also show that the tools can be used complementarily to increase detection sensitivity. Finally, we offer recommendations for future circRNA detection and validation.

UR - http://www.scopus.com/inward/record.url?scp=85164517473&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85164517473&partnerID=8YFLogxK

U2 - 10.1038/s41592-023-01944-6

DO - 10.1038/s41592-023-01944-6

M3 - Article

C2 - 37443337

AN - SCOPUS:85164517473

SN - 1548-7091

VL - 20

SP - 1159

EP - 1169

JO - Nature Methods

JF - Nature Methods

IS - 8

ER -

Large-scale benchmarking of circRNA detection tools reveals large differences in sensitivity but not in precision

Abstract

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this