Tutorial: Prof. Ulf Johansson

by admin last modified Jul 04, 2017 11:34 AM

Ulf Johansson

Predicting with Confidence

Prof. Ulf Johansson
Department of Computer Science and Informatics, Jonkoping University, Sweden

Date & Time: July 17 (Monday), 2017; 03:40pm - 05:40pm
Location: Copper Room


How good is your prediction? In risk-sensitive applications, it is crucial to be able to assess the quality of a prediction, but traditional classification and regression models do not provide their users with any information regarding prediction trustworthiness. Conformal predictors, on the other hand, are predictive models that associate each of their predictions with a precise measure of confidence. Given a user-defined significance level E, a conformal predictor outputs, for each test pattern, a multi-valued prediction region (class label set or real-valued interval) that, under relatively weak assumptions, contain the test pattern's true output value with probability 1-E. In other words, given a significance level E, a conformal predictor makes an erroneous prediction with probability E. The conformal prediction framework allows any traditional classification or regression model to be transformed into a confidence predictor with little extra work, both in terms of implementation and computational complexity. Some key properties of conformal prediction are:

  • We obtain probabilities/error bounds per instance
  •  Probabilities are well-calibrated: 95% means 95%
  •  We do not need to know the priors
  •  We make a single assumption - that the data is exchangeable ~ i.i.d.
  •  We can apply it to any machine learning algorithm
  •  It is rigorously proven and straightforward to implement
  •  There is no magic involved - only mathematics and algorithms

Hence, confidence predictors is an important tool that every data scientist should carry in their toolboxes, and conformal prediction represents a straight-forward way of associating the predictions of any predictive machine learning algorithm with confidence measures.

This tutorial aims to provide an introduction and an example-oriented exposition of the conformal prediction framework, directed at machine learning researchers and professionals. A publicly available Python library, developed by one of the authors of the tutorial, will be used for the running examples.

The goal of the tutorial is to provide attendees with the knowledge necessary for implementing functional conformal predictors, and to highlight current research on the subject.


Prof. Ulf Johansson holds a M.Sc. in Computer Engineering and Computer Science from Chalmers University of Technology, and a PhD degree in Computer Science from the Institute of Technology, Linkoping University, Sweden. Ulf Johansson's research focuses on developing machine learning algorithms for data analytics. Most of the research is applied, and often co-produced with industry. Application areas include drug discovery, health science, marketing, high-frequency trading, game AI, sales forecasting and gambling. In 2011, he had his 15 minutes of fame when called as an expert witness in the Swedish Supreme Court regarding whether Poker is a game of skill or chance. In the court, Prof. Johansson argued that skill predominates over chance using, among other sources, his paper "Fish or Shark - Data Mining Online Poker", originally presented at DMIN 2009. Ulf Johansson has published extensively in the fields of artificial intelligence, machine learning, soft computing and data mining. He is also a regular program committee member of the leading conferences in computational intelligence and machine learning. During the last few years, Prof. Johansson has published several papers on conformal prediction, some presented in top-tier venues like the Machine Learning journal and the ICDM conference.

Filed under: ,