State-of-the-art AI integration methods and frameworks

Currently, the integration of AI models and ML pipelines is complex, requiring ad-hoc developments that are error-prone and repetitive. This report evaluates three state-of-the-art frameworks for integrating AI systems: Metaflow, Luigi, and Kedro. These frameworks are thoroughly analyzed based on...

全面介紹

Saved in:
書目詳細資料
主要作者: Samson, Sherwin
其他作者: Arvind Easwaran
格式: Final Year Project
語言:English
出版: Nanyang Technological University 2024
主題:
FMU
在線閱讀:https://hdl.handle.net/10356/175091
標簽: 添加標簽
沒有標簽, 成為第一個標記此記錄!
實物特徵
總結:Currently, the integration of AI models and ML pipelines is complex, requiring ad-hoc developments that are error-prone and repetitive. This report evaluates three state-of-the-art frameworks for integrating AI systems: Metaflow, Luigi, and Kedro. These frameworks are thoroughly analyzed based on their features, execution, and integration capabilities for a given ML pipeline. Building upon these state-of-the-art pipelines, an innovative approach of adopting a Function + Data flow (FDF) paradigm is implemented. With FDF, functions are adopted as first-class citizens alongside data within the pipeline. As opposed to Data being the sole currency in traditional Data flow paradigms, functions may be defined through the operations of a pipeline and transmitted together with the data. Subsequently, a novel functionality is introduced through dynamic generation of Functional Mock-up Units (FMUs) from an ML pipeline. After training a model, regardless of the number of functions or transformations used, these elements can be serialized and packaged as an FMU. This approach is an extension of the application of Hybrid AI in smart cities, where AI models can be abstractly simulated and made reproducible on any platform through FMU/FMI packaging. For validation, this automated FMU integration for an FDF pipeline is tested across three case studies using Linear Regression, MLP, and LSTM models. This demonstrates that the dynamic simulation and integration of AI models can be effectively controlled and scaled with this approach.