Contributor
Hwarim Hyun

Implementing Work Stealing on Nemo


Mentors
Taegeon Um, Won Wook SONG
Organization
The Apache Software Foundation

This project aims to optimize the performance of Nemo by dynamically splitting skewed tasks of a job based on the information collected from runtime metrics, executing them in parallel, and merging them afterwards. The goal is achieved through the following four steps: First, gather relevant metrics during execution. Second, using the gathered statistics, determine which task is skew(i.e. needs a longer time to finish) periodically. Third, Split the skewed tasks in half and allocate the latter splitted half to the new task. And lastly, execute the new task and check its performance.