Interpreting deep learning models with marginal attribution by conditioning on quantiles

Abstract A vast and growing literature on explaining deep learning models has emerged. This paper contributes to that by introducing a global gradient-based model-agnostic method, which we call Marginal Attribution Conditioning Quantiles (MACQ). Our approach is based analyzing the marginal attribution of predictions (outputs) individual features (inputs). Specifically, consider variable importance fixing (global) output levels, how marginally contribute these fixed levels. MACQ can be seen as counterpart approaches such accumulated local effects, study sensitivities outputs perturbing inputs. Furthermore, allows us separate from interaction effects visualize 3-way relationship between attribution, level, feature value.