Understanding the generalization of deep learning has raised lots of concerns recently, where the learning algorithms play an important role in generalization performance, such as stochastic gradient descent (SGD). Along this line, we particularly study the anisotropic noise introduced by SGD, and investigate its importance for the generalization in deep neural networks. Through a thorough empi...

This paper investigates the effect of stale (delayed) gradient updates within the context of asynchronous stochastic gradient descent (Async-SGD) optimization for distributed training of deep neural networks. We demonstrate that our implementation of Async-SGD on a HPC cluster can achieve a tight bound on the gradient staleness while providing near-linear speedup. We propose a variant of the SG...

Sparse learning framework, which is very popular in the field of nature language processing recently due to the advantages of efficiency and generalizability, can be applied to Conditional Random Fields (CRFs) with L1 regularization method. Stochastic gradient descent (SGD) method has been used in training L1-regularized CRFs, because it often requires much less training time than the batch tra...

Recurrent neural network based language models (RNNLM) have been demonstrated to outperform traditional n-gram language models in automatic speech recognition. However, the superior performance is obtained at the cost of expensive model training. In this paper, we propose a sentence-independent subsampling stochastic gradient descent algorithm (SIS-SGD) to speed up the training of RNNLM using p...

This work presents a new algorithm for training recurrent neural networks (although ideas are applicable to feedforward networks as well). The algorithm is derived from a theory in nonconvex optimization related to the diffusion equation. The contributions made in this work are two fold. First, we show how some seemingly disconnected mechanisms used in deep learning such as smart initialization...

Neutrophil-specific granule deficiency (SGD) is a rare disorder characterized by recurrent pyogenic infections, defective neutrophil chemotaxis and bactericidal activity, and lack of neutrophil secondary granule proteins. CCAAT/enhancer binding protein (C/EBP)epsilon, a member of the leucine zipper family of transcription factors, is expressed primarily in myeloid cells, and its knockout mouse ...

Article history: Received 10 August 2007 Received in revised form 24 April 2008 Accepted 25 April 2008 Available online xxxx Algal blooms in Tolo Harbour, Hong Kong have received much attention and submarine groundwater discharge is speculated to be a significant pathway carrying nutrients into the constricted estuary. Plover Cove, a small cove in the Harbour, was selected for SGD analysis usin...

OBJECTIVE This review synthesizes communication interventions that involved the use of speech generating devices (SGD) for individuals with developmental disabilities. METHODS Systematic searches of electronic databases, journals and reference lists identified 35 studies meeting the inclusion criteria. These studies were evaluated in terms of (a) participants, (b) SGD function, (c) SGD charac...

Two types of low cost-per-iteration gradient descent methods have been extensively studied in parallel. One is online or stochastic gradient descent ( OGD/SGD), and the other is randomzied coordinate descent (RBCD). In this paper, we combine the two types of methods together and propose online randomized block coordinate descent (ORBCD). At each iteration, ORBCD only computes the partial gradie...

The SGD-QN algorithm is a stochastic gradient descent algorithm that makes careful use of secondorder information and splits the parameter update into independently scheduled components. Thanks to this design, SGD-QN iterates nearly as fast as a first-order stochastic gradient descent but requires less iterations to achieve the same accuracy. This algorithm won the “Wild Track” of the first PAS...