State-of-the-art AI integration methods and frameworks

• Main Author: Samson, Sherwin
• Other Authors: Arvind Easwaran
• Format: Final Year Project
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science; Kedro; Metaflow; Luigi; FMU
• Online Access: https://hdl.handle.net/10356/175091
• Institution: Nanyang Technological University

Currently, the integration of AI models and ML pipelines is complex, requiring ad-hoc developments that are error-prone and repetitive. This report evaluates three state-of-the-art frameworks for integrating AI systems: Metaflow, Luigi, and Kedro. These frameworks are thoroughly analyzed based on their features, execution, and integration capabilities for a given ML pipeline. Building upon these state-of-the-art pipelines, an innovative approach of adopting a Function + Data flow (FDF) paradigm is implemented. With FDF, functions are adopted as first-class citizens alongside data within the pipeline. As opposed to Data being the sole currency in traditional Data flow paradigms, functions may be defined through the operations of a pipeline and transmitted together with the data. Subsequently, a novel functionality is introduced through dynamic generation of Functional Mock-up Units (FMUs) from an ML pipeline. After training a model, regardless of the number of functions or transformations used, these elements can be serialized and packaged as an FMU. This approach is an extension of the application of Hybrid AI in smart cities, where AI models can be abstractly simulated and made reproducible on any platform through FMU/FMI packaging. For validation, this automated FMU integration for an FDF pipeline is tested across three case studies using Linear Regression, MLP, and LSTM models. This demonstrates that the dynamic simulation and integration of AI models can be effectively controlled and scaled with this approach.