TY - JOUR
T1 - CharaParser+EQ
T2 - Performance evaluation without gold standard
AU - Cui, Hong
AU - Dahdul, Wasila
AU - Dececchi, Alexander T.
AU - Ibrahim, Nizar
AU - Mabee, Paula
AU - Balhoff, James P.
AU - Gopalakrishnan, Hariharan
N1 - Publisher Copyright:
© 2013–2015 Hong Cui, Wasila Dahdul, Alexander T. Dececchi, Nizar Ibrahim, Paula Mabee, James P. Balhoff, Hariharan Gopalakrishnan. All Rights Reserved.
PY - 2015/1
Y1 - 2015/1
N2 - To make phenotypic characters of organisms widely useful for computerized biology research, biocurators manually convert character descriptions to a structured format, for example the Entity-Quality (EQ) format. The manual approach is time consuming and affected by inter-curator variations. In this paper we report a software application, CharaParser+EQ, to our knowledge the first software that produces EQ statements from textual character descriptions. We report a recent experiment that evaluates the performance of the software against three experienced biocurators. While the software is still far from being able to compete with biocurators on this highly intellectual task, the results show (1) CharaParser+EQ's performance (precision and recall) is greatly improved compared to a previous version, (2) the completeness of the ontologies used in the process has significant impact both on the software's EQ generation performance and on the agreement among curators, and (3) unlimited access to external knowledge (published papers, books) by curators has no significant impact on inter-curator agreements. A detailed error analysis that compares machine and curator generated EQs is included.
AB - To make phenotypic characters of organisms widely useful for computerized biology research, biocurators manually convert character descriptions to a structured format, for example the Entity-Quality (EQ) format. The manual approach is time consuming and affected by inter-curator variations. In this paper we report a software application, CharaParser+EQ, to our knowledge the first software that produces EQ statements from textual character descriptions. We report a recent experiment that evaluates the performance of the software against three experienced biocurators. While the software is still far from being able to compete with biocurators on this highly intellectual task, the results show (1) CharaParser+EQ's performance (precision and recall) is greatly improved compared to a previous version, (2) the completeness of the ontologies used in the process has significant impact both on the software's EQ generation performance and on the agreement among curators, and (3) unlimited access to external knowledge (published papers, books) by curators has no significant impact on inter-curator agreements. A detailed error analysis that compares machine and curator generated EQs is included.
KW - EQ statements
KW - Natural Language Processing
KW - Phenotype character curation
KW - curation inconsistency
KW - ontology search
UR - http://www.scopus.com/inward/record.url?scp=84987732864&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84987732864&partnerID=8YFLogxK
U2 - 10.1002/pra2.2015.145052010020
DO - 10.1002/pra2.2015.145052010020
M3 - Article
AN - SCOPUS:84987732864
SN - 2373-9231
VL - 52
SP - 1
EP - 10
JO - Proceedings of the Association for Information Science and Technology
JF - Proceedings of the Association for Information Science and Technology
IS - 1
ER -