Differentiate Quality of Experience Scheduling for Deep Learning Inferences With Docker Containers in the Cloud

With the prevalence of big-data-driven applications, such as face recognition on smartphones and tailored recommendations from Google Ads, we are road to a lifestyle with significantly more intelligence than ever before. Various neural network powered models running at back end their enable quick responses users. Supporting those requires lots cloud-based computational resources, e.g., CPUs GPUs. The cloud providers charge clients by amount resources that they occupy. Clients have balance budget quality experiences (e.g., response time). leans individual business owners, required Quality Experience (QoE) depends usage scenarios different applications. For instance, an autonomous vehicle real-time response, but unlocking your smartphone can tolerate delays. However, fail offer QoE-based option clients. In this paper, propose DQoES, differentiated experience scheduler for deep learning inferences. DQoES accepts clients' specifications targeted QoEs, dynamically adjusts approach targets. Through extensive experiments, demonstrates it schedule multiple concurrent jobs respect various QoEs achieve up 8x times satisfied when compared existing system