Resource Revocation in Apache Mesos

نویسندگان

  • Stephen Twigg
  • Huy Vo
چکیده

We demonstrate how adding resource revocation to Mesos allows the system to provide latency and resource guarantees to frameworks. Mesos, which uses dominant resource fairness to offers of new resources, was designed initially for primarly MapReduce-like and found to provide weak guarantees with more general workloads. This project resolved this issue by allowing frameworks to explicitely state resource minimums needed to constrain their task latency and then use these constraints to guide Mesos in revoking resources. This solution both minimizes work lost from revocation and leverages the pre-existing DRF algorithm in Mesos to already provide long-term fairness. Offer revocation after a timeout was employed to minimize resource waste due to indecisive frameworks. These adjustments were evaluated on a single machine using synthetic benchmarks designed to simulate problem scenarios that occur when running Mesos in production on large 1000+ node clusters. Evalutation demonstrated frameworks running latency sensitive tasks were able to start new jobs on a fully-utilized Mesos cluster employing revocation as if they were the only framework in the system even in the presence of resource-heavy or indecisive frameworks.

برای دانلود رایگان متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Optimizing CMS build infrastructure via Apache Mesos

The Offline Software of the CMS Experiment at the Large Hadron Collider (LHC) at CERN consists of 6M lines of in-house code, developed over a decade by nearly 1000 physicists, as well as a comparable amount of general use open-source code. A critical ingredient to the success of the construction and early operation of the WLCG was the convergence, around the year 2000, on the use of a homogeneo...

متن کامل

Megos: Enterprise Resource Management in Mesos Clusters

Enterprise data centers increasingly adopt a cloud-like architecture that enables the execution of multiple workloads on a shared pool of resources, reduces the data center footprint and drives down the costs. The Apache Mesos project is emerging as a leading open source resource management technology for server clusters. However, the default Mesos allocation mechanism lacks a number of policy ...

متن کامل

Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center

We present Mesos, a platform for sharing commodity clusters between multiple diverse cluster computing frameworks, such as Hadoop and MPI 1. Sharing improves cluster utilization and avoids per-framework data replication. Mesos shares resources in a fine-grained manner, allowing frameworks to achieve data locality by taking turns reading data stored on each machine. To support the sophisticated ...

متن کامل

Performance Interference of Multi-tenant, Big Data Frameworks in Resource Constrained Private Clouds

In this paper, we investigate and characterize the behavior of “big” and “fast” data analysis frameworks, in multitenant, shared settings for which computing resources (CPU and memory) are limited. Such settings and frameworks are frequently employed in both public and private cloud deployments. Resource constraints stem from both physical limitations (private clouds) and what the user is willi...

متن کامل

Towards a Hybrid Cloud Platform Using Apache Mesos

Hybrid cloud technology is becoming increasingly popular as it merges private and public clouds to bring the best of two worlds together. However, due to the heterogeneous cloud installation, facilitating a hybrid cloud setup is not simple. Despite the availability of some commercial solutions to build a hybrid cloud, an open source implementation is still unavailable. In this paper, we try to ...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

عنوان ژورنال:

دوره   شماره 

صفحات  -

تاریخ انتشار 2012