A Theoretical Analysis on Structured Learning with Noisy Data and its Applications

Abstract: Performances of supervised machine learning will be affected by noises of labeled data badly, which has been well studied by existing theories about learning with noisy data. However these theories only focus on two-class classification problems. In this paper we studied the relation between noise examples and their effects on structured learning. Firstly, we found that noises of labeled data will enlarge in structured learning problems, leading to a higher noise rate in training procedure than on labeled data. Existed theories did not consider the enlargement of noises in structured learning, thus underestimate the complexities of learning problems. Starting from the observation of enlargement of noises, this paper proposed a new theory on learning from noise data with structured predictions. Based on the theory, the concept of “effective size of training data” is proposed to describe the qualities of noisy training data sets in practice. We also analyze the situations when structured learning models will go back to lower order ones in applications. Experimental results are given to confirm the correctness of our theories as well as their practical values on cross-lingual projection and co-training.