Collection and analysis on data from Drugs.com

• Main Author: Aw, Teng Teng
• Other Authors: Sun Aixin
• Format: Final Year Project
• Language: English
• Published: 2017
• Subjects: DRNTU::Engineering::Computer science and engineering::Computing methodologies::Document and text processing
• Online Access: http://hdl.handle.net/10356/70267
• Institution: Nanyang Technological University

Internet users rely on the Internet for its convenience and efficiency. Search engines provide convenience and are time-saving. Depending on the source of results, search engines provide plenty of information at an utmost accuracy. For example, professional medical websites such as Drugs.com and Wikipedia are reliable as the authors are professionals with medical knowledge. The public, with no medical knowledge, can access this information and learn more about the prescribed drugs. Also, there are web scrapers on the Internet, known for aiding researchers in extracting data at a much faster speed in a specific time frame. In this report, Scrapy, is a web scraper, which will be used to extract data from Drugs.com. Scrapy is a framework, done in Python and the outputs will be saved in JSON files. Scrapy adapts to the different webpages with different structures using XPath.selectors. The findings will be presented in this report. The aim of this project is to utilize web scraping tools to collect data from Drugs.com and to be further analyzed. Data collected can be used in the future, saving time for researchers intending to do the same. Next, analysis of the collected data will cover aspects of the website, such as the structure and accuracy of information. In addition, this report will analyze the different web scrapers, its costs, complexity level and accuracy of data extracted. To conclude, this report will indicate the recommended choice of the web scrapers.