Support advanced features of resource managers

Summary

This project aims to implement Node Labeling and Preemption API, which are existing feature of resource managers, on REEF. Node Labels allow users to group resources with similar characteristics or with similar ACL policies and Preemption is a way to respect elasticity and SLAs together.

The Project

Even though YARN and Mesos, the resource managers have existing features that supports Node Labeling and Preemption, REEF doesn’t have APIs that makes these features available. To make developers can exploit these features on REEF regardless of which resource manager they use, APIs that make available Node Labeling and Preemption will be implemented on REEF in this project.

Required Deliverables

• Support for the Node Labeling API as it exists in YARN and Mesos in REEF
• Support for the Preemption API as it exists in YARN and Mesos in REEF

Benefits for REEF

When Node Labeling and Preemption has been done in REEF, developer can exploit these features without directly dealing with resource managers such as YARN or Mesos, but through the same interface on REEF. It will increase portability and simplifies development of applications for cluster resource managers.