DISTRIBUTED CRAWLING ON ONLINE SHOP WEBSITE

• Main Author: Inayati - NIM: 23216038 , Nur’izzah
• Format: Theses
• Language: Indonesia
• Online Access: https://digilib.itb.ac.id/gdl/view/29798
• Institution: Institut Teknologi Bandung

Nowadays, online shops are growing very fast. There are many websites that provide a place for anyone who wants to have an online shop. The increasing number of online shops is currently a problem for Badan Pusat Statistik (Statistics of Indonesia) which is responsible for data collection of all business activities in Indonesia because of the difficulty in obtaining information related to online businesses conducted by respondents and household members. Web crawling and web scraping are several ways to extract data from web pages. Because online shop sites use dynamic pages, simple web crawlers cannot retrieve data from that page. <br /> <br /> <br /> <br /> <br /> This research proposes the mechanism of web crawling web pages with dynamic data that is run in a distributed manner. The data extracted is the data of each shop account at two online shop sites. To extract data automatically, automated extraction mechanisms are designed using semantic analysis. To speed up the crawling process,designed a distributed crawling mechanism using Apache Spark. A prototype was built to test the design that was made. Some experiments used the prototype to determine the performance of the proposed distributed crawling. The experimental results show that automated extraction using semantic analysis provides good results with 100 percent precision and 94.94 percent recall. Distributed crawling can speed up the crawling process and simplify scalability settings. To increase the capacity of the extracted data, simply add resources in the form of a node without needing to change the application..