Abstract
Deep neural networks have shown dramatic improvements in a lot of supervised classification tasks. Such models are usually trained with the objective to ultimately minimize the top-1 error rate. Although this approach is very powerful, it averages out the uncertainty of individual samples and does not capture if on a given data point this prediction is reliable or not and why.In real-world scenarios, it can actually be impossible - even for an oracle - to determine the exact label of a given data item because it does not, by itself, contain sufficient evidence to decide between several similar classes. Unlike multi-task classification where each data sample is associated with several labels, here, each item corresponds to exactly one label but this latter is uncertain. For instance, an image of a plant leaf might not be enough to distinguish between several possible species sharing the same leaf morphology. In fine-grained classification problems, most data samples intrinsically contain a certain amount of such label ambiguity even if they are associated with a single hard label. Furthermore, the model itself introduces additional uncertainty in the prediction because it is learned using a finite training dataset.This uncertainty is expected to be progressively reduced by increasing the training set size contrary to the intrinsic ambiguity of the data items which is theoretically irreducible.The goal of this PhD is to study these two types of uncertainties in a decision-theoretic framework. To do so, we propose to move away from the classic top-1 prediction error rate which solely requires to estimate the most probable class. Instead, we pick decision frameworks that force the model to learn more structure about the existing uncertainty. In particular, we focus on two frameworks: (i) adding the opportunity for the classifier to refuse to answer, usually referred to as classification with reject option, and (ii) allowing the classifier to output a set of possible labels rather than a single one, which is known as set-valued classification.We first study how uncertainty information can be exploited to tackle classification with reject option. In this framework, the predictor is a pair containing a classifier and a rejector. By fixing the classifier and focusing on the rejector, we can study how uncertainty information about the classifier can be leveraged to hopefully build a better rejection criterion. Unfortunately, we empirically show that it is difficult to separate both forms of uncertainty and recombine them properly. Based on this observation, we then focus on the first form of uncertainty, task ambiguity, and study natural frameworks to handle it: set-valued classification. There are several ways to predict sets. The most naive approach is to predict the K most probable classes. However, this assumes that all the samples have the same level of ambiguity which is known to be wrong in most cases. Instead, we propose to use average-K: the predictor can output sets of different sizes but on average their size must be equal to K. We then generalize to other adaptive set-valued classification approaches and propose a framework unifying most of them. In particular, we show several ways to construct such classifiers depending on the constraints on the error rate and on the set size and study their relative advantages and weaknesses.