In this paper, we present GossipGraD – a gossip communication protocol based Stochastic Gradient Descent (SGD) algorithm for scaling Deep Learning (DL) algorithms on large-scale systems. The salient features of GossipGraD are: 1) reduction in overall communication complexity from Θ(log(p)) for p compute nodes in well-studied SGD to O(1), 2) model diffusion such that compute nodes exchange their...

Hyperparameter tuning is one of the most time-consuming steps in machine learning. Adaptive optimizers, like AdaGrad and Adam, reduce this labor by tuning an individual learning rate for each variable. Lately, researchers have shown interest in simpler methods like momentum SGD as they often yield better results. We ask: can simple adaptive methods based on SGD perform well? We show empirically...

The stochastic gradient descent (SGD) algorithm is widely used for parameter estimation, especially huge datasets and online learning. While this recursive popular computation memory efficiency, quantifying variability randomness of the solutions has been rarely studied. This article aims at conducting statistical inference SGD-based estimates in an setting. In particular, we propose a fully es...

We consider large scale distributed optimization over a set of edge devices connected to central server, where the limited communication bandwidth between server and imposes significant bottleneck for procedure. Inspired by recent advances in federated learning, we propose stochastic gradient descent (SGD) type algorithm that exploits sparsity gradient, when possible, reduce burden. At heart is...

In this paper, we are concerned with differentially private stochastic gradient descent (SGD) algorithms in the setting of convex optimization (SCO). Most existing work requires loss to be Lipschitz continuous and strongly smooth, model parameter uniformly bounded. However, these assumptions restrictive as many popular losses violate conditions including hinge for SVM, absolute robust regressio...

This paper presents a holistic approach to gradient leakage resilient distributed Stochastic Gradient Descent (SGD). First , we analyze two types of strategies for privacy-enhanced federated learning: (i) pruning with random selection or low-rank filtering and (ii) perturbation additive noise d...

In Bayesian approach to probabilistic modeling of data we select a model for probabilities of data that depends on a continuous vector of parameters. For a given data set Bayesian theorem gives a probability distribution of the model parameters. Then the inference of outcomes and probabilities of new data could be found by averaging over the parameter distribution of the model, which is an intr...

Lecture is the most common teaching method used in ethics education, while problem-based learning (PBL) and small group discussion (SGD) have been introduced as more useful methods. This study compared these methods in teaching medical ethics. Twenty students (12 female and 8 male) were randomly assigned into two groups. The PBL method was used in one group, and the other group was taught using...

Submarine groundwater discharge (SGD) and derived nutrient (NO2(-), NO3(-), NH4(+), PO4(3-), and SiO2) and trace element (Cd, Co, Cu, Fe, Mo, Ni, Pb, V and Zn) loadings to the coastal sea were systematically assessed along the coast of Majorca Island, Spain, in a general survey around the island and in three representative coves during 2010. We estimated that brackish water discharges through t...