Conformal prediction based active learning

Abstract

Conformal prediction uses the degree of strangeness (nonconformity) of new data instances to determine the confidence values of new predictions. Conformal predictors are implemented in conjunction with traditional pattern classification algorithms yielding a set of predicted class labels with guaranteed error rate, a property referred to as validity. Different from Bayesian methods, which require prior knowledge of the distribution that generates the data, conformal prediction is only based on the assumption that the data are independent and identically distributed. ☐ Conformal prediction has been shown to improve the performance of pattern classification algorithms, including support vector machines and neural networks, through active learning. Instances are selected based on their level of uncertainty, instead of being selected at random from an unlabeled pool. Moreover, the quality of the confidence values produced by conformal prediction has been demonstrated in the literature through experimentation, verifying the validity property. ☐ Despite these advances, previous work on conformal prediction considers only uncertainty as the selection criterion for active learning. Selecting a batch of m>1 instances based only on uncertainty may result in the selection of similar instances that do not provide additional information. Moreover, outlier detection is crucial to avoid the selection of instances that are not representative of the data. ☐ In light of the above, we propose novel active learning approaches, within the conformal prediction framework, considering uncertainty, diversity, and representativeness, as the selection criteria. Diversity is used to avoid the selection of similar instances, whereas representativeness is used for outlier detection. This work focuses on the application of conformal prediction to image classification. Experiments conducted on face, object, and emotion recognition databases demonstrate that the proposed active learning approaches improve the performance of a variety of pattern classification algorithms while producing reliable confidence values.