Neural Network Optimization through searching guided by stochastic methods

In this paper we present an integrated framework for neural network optimization. The problem of a nding a sensible topology and a good set of parameters is resolved by intertwining two processes: topology optimization and parameter adjustment, both embedded in an evolutionary search algorithm. EEcient evolutionary operators can be implemented based on stochastic methods like mutual information, to optimize the input structure, or the correlation coeecient of two functions, i.e., the activation of two hidden neurons. On the other hand, the 'quality' of the topology can not be evaluated without adjusting the parameters of the network, and therefore depends on the initialization and the learning process. The theory of Bayesian learning gives a framework to adjust parameters without making use of additional data, e.g., a validation set to determine hyperparameters as the weighting factor of regularization terms. Since these parameters are all set automatically during learning, it integrates perfectly into the topology optimization process. Furthermore, it provides a criterion, the evidence, to compare diierent models, which can in turn be used as (part of) the optimization criterion for the evolutionary search, since it is positively correlated with the generalization performance. This criterion, the tness function of the evolutionary process, allows the user also to encode additional requirements to the optimization process, e.g., a maximum number of parameters in case of hardware implementation.