Active Learning with Partial Feedback.

In the large-scale multiclass setting, assigning labels often consists ofanswering multiple questions to drill down through a hierarchy of classes.Here, the labor required per annotation scales with the number of questionsasked. We propose active learning with partial feedback. In this setup, thelearner asks the annotator if a chosen example belongs to a (possiblycomposite) chosen class. The answer eliminates some classes, leaving the agentwith a partial label. Success requires (i) a sampling strategy to choose(example, class) pairs, and (ii) learning from partial labels. Experiments onthe TinyImageNet dataset demonstrate that our most effective method achieves a21% relative improvement in accuracy for a 200k binary question budget.Experiments on the TinyImageNet dataset demonstrate that our most effectivemethod achieves a 26% relative improvement (8.1% absolute) in top1classification accuracy for a 250k (or 30%) binary question budget, compared toa naive baseline. Our work may also impact traditional data annotation. Forexample, our best method fully annotates TinyImageNet with only 482k (with EDCthough, ERC is 491) binary questions (vs 827k for naive method).