ویراساینس.
سند علمی

Data Dropout in Arbitrary Basis for Deep Network Regularization

Mostafa Rahmani, George Atia

چکیده

An important problem in training deep networks with high capacity is to ensure that the trained network works well with new inputs. Dropout is an effective regularization technique to boost the network’s generalization in which a random subset of the elements of given data and the extracted features are set to zero during the training process. In this paper, a new randomized regularization technique is proposed in which we withhold a random part of data without necessarily turning off the neurons/data-elements. In the proposed method, of which the conventional dropout is shown to be a special case, random data dropout is performed in an arbitrary basis. In addition, we present a framework to efficiently apply the proposed technique to the convolutional neural networks. The presented numerical experiments show that the proposed technique yields notable performance gain. The proposed approach, dubbed Generalized Dropout, provides a deep insight into the idea of dropout, shows that we can achieve different performance gains using different basis matrices, and opens up a new research question as of how to choose optimal basis matrices that achieve maximal performance gain.

اسناد مشابه

  • Deep Network Regularization via Bayesian Inference of Synaptic Connectivity

    سند علمی۲۰۱۷PAKDDHarris Partaourides, Sotirios Chatzis

    Deep neural networks (DNNs) often require good regularizers to generalize well. Currently, state-of-the-art DNN regularization techniques consist in randomly dropping units and/or connections on each iteration of the training algorithm. Dropout and DropConnect are characteristic

  • Mixed-pooling-dropout for convolutional neural network regularization

    مقاله‌ی دسترسی آزاد۲۰۲۱Journal of King Saud University - Computer and Information SciencesBrahimAit Skourt, AbdelhamidEl Hassani, AichaMajda

  • Functional Discriminant Analysis for Microarray Gene Expression Data via Radial Basis Function Networks

    سند علمی۲۰۰۴Yuko Araki, Sadanori Konishi, Seiya Imoto

    We introduce functional logistic discriminant analysis (FLDA) which is an extension of the classical method of logistic discriminant analysis to data where predictor variables are functions or curves. FLDA approach can effectively classify functions into two distinct classes by i

  • AutoDropout: Learning Dropout Patterns to Regularize Deep Networks

    مقاله‌ی دسترسی آزاد۲۰۲۱Proceedings of the ... AAAI Conference on Artificial Intelligence['Hieu T. Pham', 'Quoc V. Le', '']

    Neural networks are often over-parameterized and hence benefit from aggressive regularization. Conventional regularization methods, such as dropout or weight decay, do not leverage the structures of network's inputs hidden states. As a result, these conventional methods less effe

  • Adaptive dropout for training deep neural networks

    سند علمی۲۰۱۳NIPSJimmy Ba, Brendan J. Frey

    Recently, it was shown that deep neural networks can perform very well if the activities of hidden units are regularized during learning, e.g, by randomly dropping out 50% of their activities. We describe a method called ‘standout’ in which a binary belief network is overlaid on

  • DropELM: Fast neural network regularization with Dropout and DropConnect

    سند علمی۲۰۱۵NeurocomputingAlexandros Iosifidis, Anastasios Tefas, Ioannis Pitas

    In this paper, we propose an extension of the Extreme Learning Machine algorithm for Single-hidden Layer Feedforward Neural network training that incorporates Dropout and DropConnect regularization in its optimization process. We show that both types of regularization lead to the