Improving hadoop performance in heterogeneous big data environments by dynamic slot configurations in mapreduce hadoop programming model - M. Ali Enfais, Abolgasem; Amril Jaharadak, Adam; Abdelkarim El Marghani, Amad (International Journal of Engineering and Technology, 2019)
Hadoop has been developed as a platform solution for processing a large scale of data in parallel for different applications in Cloud com-puting. A Hadoop system can be characterized based on three main factors: cluster, workload, and user. Each of these factors can be described in heterogeneous environment, which reflects the heterogeneity degree of the Hadoop system. This paper investigates the effect of heterogeneity in each of these factors on the performance of Hadoop for different schedulers. Three schedulers which consider different levels of Hadoop heterogeneity are used for the analysis: FIFO, Fair sharing, and COSHH (Classification and Optimization based Scheduler for Heterogeneous Hadoop). Performance issues are introduced for Hadoop schedulers and comparative performance analysis between different cases of jobs submission. These jobs are processed in heterogeneous data environments and, under fixed or reconfigurable slot between map and reduce tasks for Hadoop MapReduce java programming clustering model. The results showed that when assigning tunable knob between map and reduce tasks under certain scheduler like FIFO algorithm, the performance enhanced about 81.42% especially in cases of heterogeneity environment where the workload is decreased significantly and the utilization of computational resources in increased obviously.Â Â