Automatic selection of learning bias for active sampling

Abstract

The classification task, when performed by machine learning algorithms, requires previous training on labeled instances. In many applications, the data labeling process is expensive and can affect the predictive performance of classification models. A current solution has been the use of active learning, which investigates strategies for data labeling. Its main goal is to decide which instances should be labeled and added to the training set, reducing the overall labeling costs. However, the strategy normally depends on a learning algorithm, which should be chosen by a machine learning specialist - usually based on a cross-validation procedure. Consequently, there is a deadlock: without the complete training set, the algorithm that will present the best learning curve cannot be known in advance. Ideally, some type of automatic selection should be employed to solve this deadlock. This study investigates the use of meta-learning for automatic algorithm selection in active learning tasks. Experimental results show that meta-learning is able to find correspondences between algorithms and dataset features in order to help active learning to reduce the risks of incurring in unexpected labeling costs.