Design and development of web crawler for indexing an intranet.

• Main Author: Lee, Chee Onn.
• Other Authors: Wee Kim Wee School of Communication and Information
• Format: Theses and Dissertations
• Published: 2008
• Subjects: DRNTU::Library and information science::Libraries::Technologies
• Online Access: http://hdl.handle.net/10356/1523
• Institution: Nanyang Technological University

The paper discusses database and program design of the Web crawler. The Web crawler performance was found to improve by storing the hyperlinks into three different types namely internal HTML URL addresses, internal non-HTML URL addresses and external URL addresses. In addition, the storing of word location such as line number and the word sequence number of the Web page allows phrases or strings of words to be searched and identified effectively. Better handling of comments, scripts and tags by the Web crawler was found to increase the data quality in the data collection process. It was found that enhancing the tags identification modules to handle the tags with attributes allows the Web crawler to be more effective in the retrieval of the data collection process.