Applying web scraping for e-commerce retailers

• Main Author: Lim, Yi
• Other Authors: Jun Zhao
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science
• Online Access: https://hdl.handle.net/10356/181139
• Institution: Nanyang Technological University

The rapid growth of e-commerce marketplaces has introduced new opportunities for smaller, emerging retailers to compete with large brands. Designing an optimal pricing and channel strategy across various marketplaces requires a comprehensive view of the market. Current e-commerce market research solutions offer limited success in generating actionable insights for smaller retailers. This project builds on existing solutions, demonstrating the effectiveness of web scraping to extract large amounts of market data for analysis. Using Python web scraping libraries such as Selenium WebDriver and Beautiful Soup, product data was scraped from five major e-commerce marketplaces -- Amazon, ASOS, JDSports, Farfetch, and Footlocker. The scraped data was visualized using Tableau to provide insights in areas including price distribution across marketplaces, competitors' pricing and product strategies, and consumer sentiment. The insights generated provide e-commerce retailers with a comprehensive understanding of marketplace dynamics to determine an optimal pricing and channel strategy. The proposed solution achieves a greater depth of analysis over existing solutions. The advantages of scalability and resource-efficiency makes this project highly applicable to smaller retailers seeking to compete in the e-commerce market.