DESIGN AND IMPLEMENTATION OF STREAMING DATA PIPELINE FOR FINANCIAL TRANSACTIONS ON PT. XYZ

• Main Author: Faizal Aziz, Ismail
• Format: Final Project
• Language: Indonesia
• Online Access: https://digilib.itb.ac.id/gdl/view/79653
• Institution: Institut Teknologi Bandung

With the increasing volume of data generated by humans, organizations are increasingly relying on data analysis to improve business operations. The process of transforming raw data into useful information requires data analysis. Before raw data can be analyzed, the processes of data collection, cleaning, transformation, and loading are necessary. A data pipeline is a set of tools that function to deliver data through these three processes before it can be analyzed. At PT. XYZ, these three processes take a minimum of one day. This hinders downstream processes such as ad-hoc data analysis, fraud detection, and others. In this final project, the author proposes a streaming data pipeline system to shorten the data collection gap. The proposed solution sources data from the PostgreSQL database, with the data destination being the Google BigQuery data warehouse, and the streaming platform tool being Apache Kafka. In this context, the processed data includes financial transaction data and personal customer data from PT. XYZ. In its implementation, there are limitations such as the pipeline can only use one virtual machine. The data pipeline development process is divided into four main stages: defining requirements, designing the architecture, implementing the design, and testing. Testing conducted on the system includes functional and non-functional requirement testing through end-to-end data processing simulation. The result of the design and implementation of the streaming data pipeline is a pipeline that utilizes tools such as Debezium, Apache Kafka, and Confluent Google BigQuery Sink Connector. Testing on the implemented streaming data pipeline shows that it meets both functional and non- functional requirements. As a comparison, processing the collection of 500 thousand rows of data takes a gap time of 2 to 3 minutes. This shortens the data collection process from one day to a matter of minutes.