FASSSTER Data Pipeline and DevOps

• Main Authors: Tamayo, Lenard Paulo V, Pulmano, Christian E, Santos, Romel John, Buhain, Jay-Arr, Ico, Raven
• Format: text
• Published: Archīum Ateneo 2023
• Subjects: Data pipeline; Data analytics; Data visualization; Analytical, Diagnostic and Therapeutic Techniques and Equipment; Data Science; Epidemiology; Medicine and Health Sciences; Physical Sciences and Mathematics; Public Health
• Online Access: https://archium.ateneo.edu/discs-faculty-pubs/415; https://doi.org/10.1007/978-981-99-3153-8_3
• Institution: Ateneo De Manila University

In data science; the data pipeline serves as a methodological and potentially architectural framework for setting up systems that require near real-time monitoring through dashboards and visualization. The collection; aggregation; and analysis of data related to COVID-19 cases proved to be important in providing the community with the right information at the right time. In the beginning of the pandemic; the data used for interpretation came from different data sources. Some datasets were made available to the public by the Department of Health (DOH) by publishing a Google Drive that contained the datasets in spreadsheet format (http://bit.ly/DataDropPH). Eventually; DOH provided access to a BigQuery database to select groups where data can be automatically extracted on a daily basis. These datasets are extracted and ingested to a data warehouse for further analysis. Various data analysis and modeling techniques are applied to the data. As such; data analysis scripts are written using two popular programming languages; R and Python; to facilitate the processing and transformation of data. The stakeholders then view model outputs in a web-based visualization platform. This chapter describes the FASSSTER data pipeline; from extraction; preprocessing; and processing to produce outputs generated by analytics and models and corresponding data visualization techniques.