Workflow system for MapReduce in cloud environment

• Main Author: Wadi, Muntadher Saadoon
• Format: Thesis
• Language: English
• Published: 2017
• Online Access: http://psasir.upm.edu.my/id/eprint/71042/1/FSKTM%202017%206%20-%20IR.pdf; http://psasir.upm.edu.my/id/eprint/71042/
• Institution: Universiti Putra Malaysia

The magnitude of data generated and shared by businesses, public administrations, industrial sectors and scientific research, has increased immeasurably. Apache Hadoop is an open source software framework, which enables a scalable and distributed processing of high volumes of data. MapReduce together with its Hadoop implementation has been widely adopted in many practical applications. A common practice nowadays is to implement MapReduce applications in a high-performance infrastructure, such as cloud computing. A cloud platform can deploy and manage Hadoop clusters. However, there are tasks required advanced knowledge in computer science and cloud computing when using MapReduce technology that prevent the usage of current technologies and software solutions. For example, MapReduce deployment and maintenance, data integration with Hadoop distributed file system or MapReduce job submission. A MapReduce workflow system is one of the solution that could assist MapReduce and Hadoop developers. Besides, it provides a user-friendly execution platform that encapsulating complexity of data analysis steps. In this research, a new workflows system is developed to facilitate the use of collaborating, coordinating and executing operations of MapReduce programs with a graphical user interface based on Hadoop cloud cluster. The experimental results indicate that the developed workflow system can achieve good speed in performance. It is believed that the workflow system is an ideal stereotype for MapReduce and it will play an important role in the era of big data applications in cloud computing.