Query cost estimation in DBMS with deep learning

• Main Author: Acharya, Atul
• Other Authors: Luo Siqiang
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2023
• Subjects: Engineering::Computer science and engineering
• Online Access: https://hdl.handle.net/10356/166095
• Institution: Nanyang Technological University

Cost and cardinality estimation is considered the Achilles Heel of modern query optimizers. Poor cardinality estimates lead to bad cost estimates resulting in sub-optimal query execution plans being selected which drops the performance of query optimizers. With the recent rise of ML for DB, the database community explored the use of learned methods in cost and cardinality estimation. However none of the methods till date can achieve prediction speeds required for modern database systems. In this project we introduce a novel algorithm (TreeGBM) using Gradient Boosting Trees to solve both cost estimation and cardinality estimation on numeric JOB workloads based on the IMDB dataset. We conducted multiple experiments to improve prediction scores and inference times. Our experiments showed that the TreeGBM was ∼120 times faster than state-of-the-art learned methods while maintaining good prediction scores. We stated possible improvements to our method that could help improve prediction scores and inference times. Future work can add on to the algorithm by using a new predicate embedding algorithm that does not incur much latency and by using prefix tries to encode string values.