The Curious Case of Convex Neural Networks

This paper investigates a constrained formulation of neural networks where the output is convex function input. We show that convexity constraints can be enforced on both fully connected and convolutional layers, making them applicable to most architectures. The include restricting weights (for all but first layer) non-negative using non-decreasing activation function. Albeit simple, these have profound implications generalization abilities network. draw three valuable insights: (a) Input Output Convex Neural Networks (IOC-NNs) self regularize significantly reduce problem overfitting; (b) Although heavily constrained, they outperform base multi layer perceptrons achieve similar performance as compared architectures (c) IOC-NNs robustness noise in train labels. demonstrate efficacy proposed idea thorough experiments ablation studies six commonly used image classification datasets with different network appendix codes for this are available at: https://github.com/sarathsp1729/Convex-Networks.