TY - JOUR
T1 - An end user evaluation of query formulation and results review tools in three medical meta-search engines
AU - Leroy, Gondy
AU - Xu, Jennifer
AU - Chung, Wingyan
AU - Eggers, Shauna
AU - Chen, Hsinchun
N1 - Funding Information:
This project was sponsored by the following grant: NIH/NLM, “UMLS Enhanced Dynamic Agents to Manage Medical Knowledge”, 1 R01 LM06319-01Al, February 2001–February 2004.
PY - 2007/11
Y1 - 2007/11
N2 - Purpose: Retrieving sufficient relevant information online is difficult for many people because they use too few keywords to search and search engines do not provide many support tools. To further complicate the search, users often ignore support tools when available. Our goal is to evaluate in a realistic setting when users use support tools and how they perceive these tools. Methods: We compared three medical search engines with support tools that require more or less effort from users to form a query and evaluate results. We carried out an end user study with 23 users who were asked to find information, i.e., subtopics and supporting abstracts, for a given theme. We used a balanced within-subjects design and report on the effectiveness, efficiency and usability of the support tools from the end user perspective. Conclusions: We found significant differences in efficiency but did not find significant differences in effectiveness between the three search engines. Dynamic user support tools requiring less effort led to higher efficiency. Fewer searches were needed and more documents were found per search when both query reformulation and result review tools dynamically adjust to the user query. The query reformulation tool that provided a long list of keywords, dynamically adjusted to the user query, was used most often and led to more subtopics. As hypothesized, the dynamic result review tools were used more often and led to more subtopics than static ones. These results were corroborated by the usability questionnaires, which showed that support tools that dynamically optimize output were preferred.
AB - Purpose: Retrieving sufficient relevant information online is difficult for many people because they use too few keywords to search and search engines do not provide many support tools. To further complicate the search, users often ignore support tools when available. Our goal is to evaluate in a realistic setting when users use support tools and how they perceive these tools. Methods: We compared three medical search engines with support tools that require more or less effort from users to form a query and evaluate results. We carried out an end user study with 23 users who were asked to find information, i.e., subtopics and supporting abstracts, for a given theme. We used a balanced within-subjects design and report on the effectiveness, efficiency and usability of the support tools from the end user perspective. Conclusions: We found significant differences in efficiency but did not find significant differences in effectiveness between the three search engines. Dynamic user support tools requiring less effort led to higher efficiency. Fewer searches were needed and more documents were found per search when both query reformulation and result review tools dynamically adjust to the user query. The query reformulation tool that provided a long list of keywords, dynamically adjusted to the user query, was used most often and led to more subtopics. As hypothesized, the dynamic result review tools were used more often and led to more subtopics than static ones. These results were corroborated by the usability questionnaires, which showed that support tools that dynamically optimize output were preferred.
KW - Informatics
KW - Information storage and retrieval
KW - Unified Medical Language System
UR - http://www.scopus.com/inward/record.url?scp=35348864534&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=35348864534&partnerID=8YFLogxK
U2 - 10.1016/j.ijmedinf.2006.08.001
DO - 10.1016/j.ijmedinf.2006.08.001
M3 - Article
C2 - 16996298
AN - SCOPUS:35348864534
VL - 76
SP - 780
EP - 789
JO - International Journal of Medical Informatics
JF - International Journal of Medical Informatics
SN - 1386-5056
IS - 11-12
ER -