Efficient Resource Utilization in Hadoop on Virtual Machine

Hadoop is one of open source software technology that is used for processing large amount of data across clusters of commodity servers in distributed manner. Mainly it is designed to provide high fault tolerance and scale up a single server to thousands numbers of machines. Hadoop uses Hadoop distributed file system (HDFS) which is open source implementation of Google File System (GFS) for data storage. Map/Reduce is the main functionality used for storing data in HDFS. We have environment where Hadoop is deployed in virtual machine in which we can use Kernel-based Virtual Machine (KVM) as virtualization infrastructure. Existing service just use the entire resources available for admitted job. In such situation resource utilization is not proper efficient. It exceeds the limit of minimum resources which is required to finish the job. This configuration is resulting in poor resource utilization with higher cost. So avoid this create a new cluster for each job is assigned. Instead of customer to decide the resources for the job, this model automatically select desired systems for finish the job with minimum resource utilization. Keywords— Hadoop, Cluster, KVM, Virtualization, HDFS