DEVELOPMENT OF DEGRADED NODE DETECTION DUE TO BANDWIDTH REDUCTION WHEN STARTUP PHASE OF MAPREDUCE HADOOP

• Main Author: Raudi Avinanto - NIM : 13514087, Praditya
• Format: Final Project
• Language: Indonesia
• Online Access: https://digilib.itb.ac.id/gdl/view/29958
• Institution: Institut Teknologi Bandung

Hadoop is a framework and software used to store and process large amounts of data in a distributed manner. To overcome the failure of a node, Hadoop has a feature called speculative execution. But a study entitled Limplock states that speculative execution cannot detect nodes that experience interference on network interface cards (NICs). Another study, entitled Path Based Speculative Execution (PBSE) and has made the solution of this problem, but still has not been able to detect slow nodes at the startup and cleanup phase MapReduce. <br /> <br /> <br /> <br /> <br /> This final project will try to perfect PBSE solution by creating detection mechanism of nodes with a slow NIC during the startup phase (localization operation) MapReduce. The solution offered is creating a new heartbeat sent by NodeManager to ApplicationMaster (AM). The Heartbeat contains the localization progress of each node. With the heartbeat data of all nodes as input, an algorithm will be executed that can detect nodes with slow NICs. <br /> <br /> <br /> <br /> <br /> In normal cases, the detection system that has been created has succeeded in detecting nodes that have a slow NIC and have a performance that is not much different from the original hadoop. <br />