An intelligent personal spider (agent) for dynamic Internet/Intranet searching

Chen Hsinchun, Chung Yi-Ming, Marshall Ramsey, Christopher C. Yang

Research output: Contribution to journalArticlepeer-review

63 Scopus citations


As Internet services based on the World-Wide Web become more popular, information overload has become a pressing research problem. Difficulties with search on Internet will worsen as the amount of on-line information increases. A scalable approach to Internet search is critical to the success of Internet services and other current and future National Information Infrastructure (Nil) applications. As part of the ongoing Illinois Digital Library Initiative project, this research proposes an intelligent personal spider (agent) approach to Internet searching. The approach, which is grounded on automatic textual analysis and general-purpose search algorithms, is expected to be an improvement over the current static and inefficient Internet searches. In this experiment, we implemented Internet personal spiders based on best first search and genetic algorithm techniques. These personal spiders can dynamically take a user's selected starting homepages and search for the most closely related homepages in the web, based on the links and keyword indexing. A plain, static CGI/HTML-based interface was developed earlier, followed by a recent enhancement of a graphical, dynamic Java-based interface. Preliminary evaluation results and two working prototypes (available for Web access) are presented. Although the examples and evaluations presented are mainly based on Internet applications, the applicability of the proposed techniques to the potentially more rewarding Intranet applications should be obvious. In particular, we believe the proposed agent design can be used to locate organization-wide information, to gather new, time-critical organizational information, and to support team-building and communication in Intranets.

Original languageEnglish (US)
Pages (from-to)41-58
Number of pages18
JournalDecision Support Systems
Issue number1
StatePublished - May 1998


  • Agents
  • Evolutionary programming
  • Information retrieval
  • Internet
  • Intranet
  • Java
  • Machine learning
  • Semantic retrieval
  • Spider
  • World-Wide Web

ASJC Scopus subject areas

  • Management Information Systems
  • Information Systems
  • Developmental and Educational Psychology
  • Arts and Humanities (miscellaneous)
  • Information Systems and Management


Dive into the research topics of 'An intelligent personal spider (agent) for dynamic Internet/Intranet searching'. Together they form a unique fingerprint.

Cite this