In recent years, stochastic gradient descent (SGD) based techniques has become the standard tools for training neural networks. However, formal theoretical understanding of why SGD can train neural networks in practice is largely missing. In this paper, we make progress on understanding this mystery by providing a convergence analysis for SGD on a rich subset of two-layer feedforward networks w...

The Mediterranean Sea (MS) is a semienclosed basin that is considered one of the most oligotrophic seas in the world. In such an environment, inputs of allochthonous nutrients and micronutrients play an important role in sustaining primary productivity. Atmospheric deposition and riverine runoff have been traditionally considered the main external sources of nutrients to the MS, whereas the rol...

In this paper, we mainly study the convergence properties of stochastic gradient descent (SGD) as described in Needell et al. [2]. The function to be minimized with SGD is assumed to be strongly convex. Also, its gradients are assumed to be Lipschitz continuous. First, we discuss the superior bound on convergence (of standard SGD) obtained by Needell et al. [2] as opposed to the previous work o...

Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conventional training. We also show that this Stochastic Weight Averaging (SWA) p...

The Saccharomyces Genome Database (SGD) resources, ranging from genetic and physical maps to genome-wide analysis tools, reflect the scientific progress in identifying genes and their functions over the last decade. As emphasis shifts from identification of the genes to identification of the role of their gene products in the cell, SGD seeks to provide its users with annotations that will allow...

Stochastic gradient descent (SGD) and its variants have become more and more popular in machine learning due to their efficiency and effectiveness. To handle large-scale problems, researchers have recently proposed several parallel SGD methods for multicore systems. However, existing parallel SGD methods cannot achieve satisfactory performance in real applications. In this paper, we propose a f...

BACKGROUND Due to limited end-of-life discussions and the absence of palliative care, hospitalisations are frequent at the end of life among nursing home residents in Singapore, resulting in high health-care costs. AIM Our objective was to evaluate the economic impact of Project Care at the End-of-Life for Residents in homes for the Elderly (CARE) programme on nursing home residents compared ...

? ?????? ??????????????? ?????? ??????????? ??????????????? ???????? ???????? ???????. ?????? ?????? ???? ??????????? ? ???????? ????????, ? ????? ????? ????????? ???? ?????? ??????????. ?? ???????? ??? ?? ??????? ?????? ???????????? ????????? ???? ??????????????? ???????????? ?????? (SGD). ????? ?????? ???????????? ?????? ????? ?????? ? ?????????????? ?????? ??????????? ????-?????????. ???????...

Because stochastic gradient descent (SGD) has shown promise optimizing neural networks with millions of parameters and few if any alternatives are known to exist, it has moved to the heart of leading approaches to reinforcement learning (RL). For that reason, the recent result from OpenAI showing that a particular kind of evolution strategy (ES) can rival the performance of SGD-based deep RL me...

This work characterizes the benefits of averaging techniques widely used in conjunction with stochastic gradient descent (SGD). In particular, this work sharply analyzes: (1) mini-batching, a method of averaging many samples of the gradient to both reduce the variance of a stochastic gradient estimate and for parallelizing SGD and (2) tail-averaging, a method involving averaging the final few i...