Download ROC Analysis in Artificial Intelligence

José Hernández-Orallo
Cèsar Ferri
Nicolas Lachiche
Peter Flach (Eds.)
ROC Analysis in
Artificial Intelligence
1st International Workshop, ROCAI-2004
Valencia, Spain, August 22, 2004
Proceedings
Workshop associated to ECAI-2004,
The 16th European Conference on Artificial Intelligence
Volume Editors
José Hernández-Orallo, Technical University of Valencia, Spain.
Cèsar Ferri, Technical University of Valencia, Spain.
Nicolas Lachiche, University of Strasbourg, France.
Peter Flach, University of Bristol, UK.
Proceedings of the 1st Intl. Workshop on ROC Analysis in Artificial Intelligence, ROCAI-2004.
Valencia, Spain, August 22, 2004
Depósito Legal:
Impreso en España.
Preface
This volume contains the proceedings of the First International Workshop on ROC Analysis in
Artificial Intelligence, ROCAI-2004. The workshop was held as part of the Sixteenth European
Conference on Artificial Intelligence, ECAI’2004, in Valencia (Spain) on August 22, 2004.
Receiver Operating Characteristic Analysis (ROC Analysis) is a powerful tool for
cost/benefit analysis in decision making. Widely used in psychology and medicine for many
decades, it has been introduced relatively recently in several areas of artificial intelligence:
machine learning, multiagent systems, intelligent decision support and expert systems. In this
context, ROC analysis provides techniques to select possibly optimal models and to discard
suboptimal ones independently from (and prior to specifying) the cost context or the class
distribution. Furthermore, the Area Under the ROC Curve (AUC) has been shown to be a better
evaluation measure than accuracy in contexts with variable misclassification costs and/or
imbalanced datasets. AUC is also the standard measure when using classifiers to rank
examples, and, hence, is used in applications where ranking is crucial, such as campaign design,
model combination, collaboration strategies, and co-learning.
Nevertheless, there are several open questions and limitations that hamper a broader use
and applicability of ROC analysis. Its connections with other evaluation measures is not yet
completely clarified, its incorporation in decision support and expert systems technology just
envisaged, its use for improving the decisions of (communities of) intelligent agents
unexplored, and its use in data mining hasn’t yet reached its full potential. Among the
limitations of ROC analysis, an important one, despite some recent progress, is its possible but
difficult and computationally expensive extension to more than two classes.
The European Conference on Artificial Intelligence offered us a great opportunity to
foster the cross-fertilisation of ideas and applications with related areas in artificial intelligence.
Hence the name given to this first workshop: ROC Analysis in Artificial Intelligence (ROCAI2004). We hope this workshop will be the start of a series of workshops on ROC Analysis.
The technical program of ROCAI-2004 consists of one invited talk, eleven papers and a
round table, organised in several sessions during this one-day workshop. The topics of these
sessions cover a wide spectrum on ROC Analysis theory and applications, including new ideas
and results, recent developments, and new research directions.
We would like to thank everyone who contributed to make this workshop possible. First
of all, we thank all the authors who submitted papers to ROCAI-2004. Each of them was
reviewed by two members from the Program Committee, who finally accepted eleven papers
for presentation. In this regard, we are grateful to the Program Committee and the additional
reviewers for their excellent job. We are also thankful to the institutions who have supported
this conference: the Network of Excellence on Knowledge Discovery (KDNet), the Generalitat
Valenciana, the Technical University of Valencia and DSIC. Finally, we have to express our
gratitude to the ECAI-2004 organisation for the facilities provided.
Valencia, Spain
August 2004
J. Hernández-Orallo, C. Ferri, N. Lachiche and P. Flach
I
II
Program Committee
•
•
•
•
•
•
•
•
•
•
•
Hendrik Blockeel, K.U.Leuven, Belgium.
Stephan Dreiseitl FHS Hagenberg, Austria.
Tom Fawcett, HP Labs, Palo Alto, CA, USA.
Cèsar Ferri, Technical University of Valencia, Spain.
Peter Flach, University of Bristol, UK.
Johannes Fürnkranz, TU Darmstadt, Germany.
José Hernández-Orallo, Technical University of Valencia, Spain.
Nicolas Lachiche, University of Strasbourg, France.
Charles Ling, University of Western Ontario, London, Ontario, Canada.
Maarten van Someren, University of Amsterdam, The Netherlands.
Francesco Tortorella, University of Cassino, Italy.
Organising Committee
•
•
•
•
Cèsar Ferri, Technical University of Valencia, Spain.
Peter Flach, University of Bristol, UK.
José Hernández-Orallo, Technical University of Valencia, Spain.
Nicolas Lachiche, University of Strasbourg, France.
III
IV
Table of Contents
An Empirical Evaluation of Supervised Learning for ROC Area ....................................................... 1
Rich Caruana, Alexandru Niculescu-Mizil
Data Mining in Metric Space: An Empirical Analysis of Supervised
Learning Performance Criteria ................................................................................................................ 9
Rich Caruana, Alexandru Niculescu-Mizil
What ROC Curves Can’t Do (and Cost Curves Can) ......................................................................... 19
Chris Drummond, Robert C. Holte
Cautious Classifiers................................................................................................................................. 27
Cèsar Ferri, José Hernández-Orallo
ROC Optimisation of Safety Related Systems ..................................................................................... 37
Jonathan E. Fieldsend, Richard M. Everson
Precision and Recall Optimisation for Information Access Tasks .................................................... 45
Michelle J. Fisher, Jonathan E. Fieldsend, Richard M. Everson
ROC Analysis of Example Weighting in Subgroup Discovery ......................................................... 55
Branko Kavšek, Nada Lavrač, Ljupčo Todorovski
Confidence Bands for ROC Curves: Methods and an Empirical Study........................................... 61
Sofus A. Macskassy, Foster Provost
Optimizing Area Under Roc Curve with SVMs.................................................................................. 71
Alain Rakotomamonjy
Learning Interestingness Measures in Terminology Extraction.
A ROC-based approach .......................................................................................................................... 81
Mathieu Roche, Jérôme Azé, Yves Kodratoff, Michèle Sebag
Learning Mixtures of Localized Rules by Maximizing the Area Under the
ROC Curve ............................................................................................................................................... 89
Tobias Sing, Niko Beerenwinkel, Thomas Lengauer
Author Index....................................................................................................................................... 97
V
VI