MetaSpider: Meta-searching and categorization on the Web

Hsinchun Chen, Haiyan Fan, Michael Chau, Daniel Zeng

Research output: Contribution to journalArticlepeer-review

64 Scopus citations


It has become increasingly difficult to locate relevant information on the Web, even with the help of Web search engines. Two approaches to addressing the low precision and poor presentation of search results of current search tools are studied: meta-search and document categorization. Meta-search engines improve precision by selecting and integrating search results from generic or domain-specific Web search engines or other resources. Document categorization promises better organization and presentation of retrieved results. This article introduces MetaSpider, a meta-search engine that has real-time indexing and categorizing functions. We report in this paper the major components of MetaSpider and discuss related technical approaches. Initial results of a user evaluation study comparing MetaSpider, NorthernLight, and MetaCrawler in terms of clustering performance and of time and effort expended show that MetaSpider performed best in precision rate, but disclose no statistically significant differences in recall rate and time requirements. Our experimental study also reveals that MetaSpider exhibited a higher level of automation than the other two systems and facilitated efficient searching by providing the user with an organized, comprehensive view of the retrieved documents.

Original languageEnglish (US)
Pages (from-to)1134-1147
Number of pages14
JournalJournal of the American Society for Information Science and Technology
Issue number13
StatePublished - Nov 2001

ASJC Scopus subject areas

  • Software
  • Information Systems
  • Human-Computer Interaction
  • Computer Networks and Communications
  • Artificial Intelligence


Dive into the research topics of 'MetaSpider: Meta-searching and categorization on the Web'. Together they form a unique fingerprint.

Cite this