The purpose of this paper is to discuss empirical risk minimization when the losses are not necessarily bounded and may have a distribution with heavy tails. In such situations usual empirical averages may fail to provide reliable estimates and empirical risk minimization may provide large excess risk. However, some robust mean estimators proposed in the literature may be used to replace empiri...

In this note we study lower bounds on the empirical minimization algorithm. To explain the basic set up of this algorithm, let (Ω, μ) be a probability space and set X to be a random variable taking values in Ω, distributed according to μ. We are interested in the function learning (noiseless) problem, in which one observes n independent random variables X1, . . . , Xn distributed according to μ...

Elicitation is the study of statistics or properties which are computable via empirical risk minimization. While several recent papers have approached the general question of which properties are elicitable, we suggest that this is the wrong question—all properties are elicitable by first eliciting the entire distribution or data set, and thus the important question is how elicitable. Specifica...

Elicitation is the study of statistics or properties which are computable via empirical risk minimization.While several recent papers have approached the general question of which properties are elicitable, wesuggest that this is the wrong question—all properties are elicitable by first eliciting the entire distributionor data set, and thus the important question is how elicitable. ...

Twin support vector regression (TSVR), as an effective regression machine, solves a pair of smaller-sized quadratic programming problems (QPPs) rather than a single large one as in the classical support vector regression (SVR), which makes the learning speed of TSVR approximately 4 times faster than that of the SVR. However, the empirical risk minimization principle is implemented in TSVR, whic...

Support Vector Machine (SVM) employs Structural Risk Minimization (SRM) principle to generalize better than conventional machine learning methods employing the traditional Empirical Risk Minimization (ERM) principle. When applying SVM to response modeling in direct marketing, however, one has to deal with the practical difficulties: large training data, class imbalance and scoring from binary S...

Given a finite set F of estimators, the problem of aggregation is to construct a new estimator whose risk is as close as possible to the risk of the best estimator in F . It was conjectured that empirical minimization performed in the convex hull of F is an optimal aggregation method, but we show that this conjecture is false. Despite that, we prove that empirical minimization in the convex hul...

Abstract Given a finite set F of estimators, the problem of aggregation is to construct a new estimator that has a risk as close as possible to the risk of the best estimator in F . It was conjectured that empirical minimization performed in the convex hull of F is an optimal aggregation method, but we show that this conjecture is false. Despite that, we prove that empirical minimization in the...

1. Some aspects of classical empirical process theory: uniform laws of large numbers, process convergence and uniform central limit theorems. 2. M-estimation. Asymptotic theory of consistency, rates of convergence and limiting distribution. 3. Non-asymptotic theory of penalized empirical risk minimization; nonasymptotic deviation inequalities for suprema of empirical processes, oracle inequalit...