Tutorial: Prof. Ulf Johansson

by admin last modified Jun 09, 2019 11:45 AM

Ulf Johansson

Predicting with confidence – Conformal Prediction and Venn Predictors

Prof. Ulf Johansson
Department of Computer Science and Informatics
Jönköping University, Sweden
ulf.johansson@ju.se
Date & Time: TBA
Location: TBA
 

Abstract

How good is your prediction? In risk-sensitive applications, it is crucial to be able to assess the quality of a prediction, but traditional classification and regression models don't provide their users with any information regarding the trustworthiness of a prediction.
Conformal predictors, on the other hand, are predictive models that associate each of their predictions with a precise measure of confidence. Given a user-defined significance level E, a conformal predictor outputs, for each test instance, a prediction region (for classification a label set, and for regression a real-valued interval) that, under relatively weak assumptions, contains the true target value with probability 1-E. In other words, given a significance level E, the error rate of a conformal predictor will be exactly E, in the long run. Since all conformal predictors have this remarkable property, called validity, the main goal becomes minimizing the prediction regions, thus maximizing the informativeness. 
The conformal prediction framework allows any traditional classification or regression model to be transformed into a confidence predictor with very little extra work, both in terms of implementation and computational complexity. 
For classification, the definition of validity in conformal prediction is often perceived as somewhat counter-intuitive, since the guarantee only applies a priori, i.e., once we have seen a specific prediction, the probability for that prediction to be wrong is no longer E. With this in mind, we recommend Venn predictors as a very strong alternative to conformal prediction for classification.
Venn predictors are multi-probabilistic predictors with proven validity properties. The standard impossibility result for probabilistic prediction is circumvented in two ways: (i) multiple probabilities for each label are outputted, with one of them being the valid one and (ii) the statistical tests for validity are restricted to calibration.
Hence, conformal prediction and Venn predictors are important tools that every data scientist should carry in their toolboxes, since they represent a straightforward way of associating the predictions of any predictive machine learning algorithm with confidence measures.
This tutorial aims to provide an introduction and an example-oriented presentation of the conformal prediction and Venn prediction frameworks, directed at machine learning researchers and professionals. The goal of the tutorial is to provide attendees with the knowledge necessary for implementing confidence predictors, and to highlight current research on the subject. The tutorial will contain examples of using confidence predictors in Python and KNIME.

The intended audience is machine learning researchers and professionals at intermediate to expert level. The participants are expected to have a good understanding of machine learning and data mining.

 

Biography

Prof. Ulf Johansson holds a M.Sc. in Computer Engineering and Computer Science from Chalmers University of Technology, and a PhD degree in Computer Science from the Institute of Technology, Linköping University, Sweden. Since 2016, he is a full professor in computer science at the School of Engineering, Jönköping University.
Ulf Johansson’s main area of expertise is machine learning algorithms for data analytics. Most of the research is applied, and often co-produced with industry. Application areas include drug discovery, health science, marketing, high-frequency trading, game AI, sports analytics, sales forecasting and gambling. Prof. Johansson has published extensively in the fields of artificial intelligence, machine learning, soft computing and data mining. He is also a regular program committee member of the leading conferences in computational intelligence and machine learning. During the last few years, he has published several papers on conformal prediction and Venn predictors, some presented in top-tier venues like the Machine Learning journal and the ICDM conference.

Filed under: , ,