Computing jobs monitoring dashboard in Malaysia

• Main Author: Tan, Zhen Wei
• Format: Final Year Project / Dissertation / Thesis
• Published: 2022
• Subjects: T Technology (General)
• Online Access: http://eprints.utar.edu.my/4700/1/fyp_CS_2022_TZW.pdf; http://eprints.utar.edu.my/4700/
• Institution: Universiti Tunku Abdul Rahman

This project proposed computing jobs monitoring dashboard in Malaysia and the dashboard will analyze and visualize the scraped data to help job seekers to better understand the current job market in the IT industry. The main motivation to propose this project is that there is vast amount of data available in online job recruitment platform but however, no tools or software are available to analyze that data into meaningful representation to job seekers. This project will focus on scraping data about computing jobs, this is because the IT industry changes and grows rapidly year by year, yet there is no data analysis and statistics about the related industry in Malaysia. Therefore, in this work, a computing jobs monitoring dashboard is proposed to solve the aforementioned issues. The proposed dashboard is able to automatically extract relevant data from online job recruitment platform such as JobStreet and Indeed, analyze the extracted data and visualize them in an interactive manner. The scraped data includes job title, company, location, salary, job requirements, qualifications, years of relevant job experience and application link. Apart from that, the Logistic Regression was used to classify the jobs into different computing jobs categories and a custom Named Entity Recognition (NER) model was built to extract the Information and Communication Technology (ICT) skills from each job requirements. The dashboard displays useful information for job seekers, including popular programming languages and skills, distribution of job opportunities, etc. The proposed dashboard is an interactive dashboard that provide users with several filtering options to view relevant data and information based on certain filtering criteria. In this work, Beautifulsoup has been used to program web scraping scripts and WayScript is used as the main development platform to automate the data scraping and storing them in Azure Blob Storage. In addition to that, the front end of this project is a highly interactive dashboard is developed using Plotly's Dash framework.