DSAG: A Mixed Synchronous-Asynchronous Iterative Method for Straggler-Resilient Learning

We consider straggler-resilient learning. In many previous works, e.g., in the coded computing literature, straggling is modeled as random delays that are independent and identically distributed between workers. However, practical scenarios, a given worker may straggle over an extended period of time. propose latency model captures this behavior substantiated by traces collected on Microsoft Azure, Amazon Web Services (AWS), small local cluster. Building model, we DSAG, mixed synchronous-asynchronous iterative optimization method, based stochastic average gradient (SAG) combines timely stale results. also dynamic load-balancing strategy to further reduce impact evaluate DSAG for principal component analysis, cast finite-sum problem, large genomics dataset, logistic regression cluster composed 100 workers AWS, find up about 50% faster than SAG, more twice fast methods, particular scenario consider.