Information Bottleneck in Deep Learning - A Semiotic Approach


 The information bottleneck principle was recently proposed as a theory meant to explain some of the training dynamics deep neural architectures. Via plane analysis, patterns start emerge in this framework, where two phases can be distinguished: fitting and compression. We take step further study behaviour spatial entropy characterizing layers convolutional networks (CNNs), relation theory. observe pattern formations which resemble compression phases. From perspective semiotics, also known signs sign-using behavior, saliency maps CNN’s exhibit aggregations: are aggregated into supersigns process is called semiotic superization. Superization characterized by decrease interpreted concentration. discuss from superization discover very interesting analogies related informational adaptation model. In practical application, we introduce modification CNN process: progressively freeze with small variation their map representation. Such stopped earlier without significant impact on performance (the accuracy) network, connecting evolution through time network.