Design and development of web crawler for indexing an intranet.

• Main Author: Lee, Chee Onn.
• Other Authors: Wee Kim Wee School of Communication and Information
• Format: Theses and Dissertations
• Published: 2008
• Subjects: DRNTU::Library and information science::Libraries::Technologies
• Online Access: http://hdl.handle.net/10356/1523
• Institution: Nanyang Technological University

The paper discusses database and program design of the Web crawler. The Web crawler performance was found to improve by storing the hyperlinks into three different types namely internal HTML URL addresses, internal non-HTML URL addresses and external URL addresses. In addition, the storing of word l...