Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Time series wikipedia , lookup
Philosophy of artificial intelligence wikipedia , lookup
Ethics of artificial intelligence wikipedia , lookup
History of artificial intelligence wikipedia , lookup
Intelligence explosion wikipedia , lookup
Existential risk from artificial general intelligence wikipedia , lookup
José Hernández-Orallo Cèsar Ferri Nicolas Lachiche Peter Flach (Eds.) ROC Analysis in Artificial Intelligence 1st International Workshop, ROCAI-2004 Valencia, Spain, August 22, 2004 Proceedings Workshop associated to ECAI-2004, The 16th European Conference on Artificial Intelligence Volume Editors José Hernández-Orallo, Technical University of Valencia, Spain. Cèsar Ferri, Technical University of Valencia, Spain. Nicolas Lachiche, University of Strasbourg, France. Peter Flach, University of Bristol, UK. Proceedings of the 1st Intl. Workshop on ROC Analysis in Artificial Intelligence, ROCAI-2004. Valencia, Spain, August 22, 2004 Depósito Legal: Impreso en España. Preface This volume contains the proceedings of the First International Workshop on ROC Analysis in Artificial Intelligence, ROCAI-2004. The workshop was held as part of the Sixteenth European Conference on Artificial Intelligence, ECAI’2004, in Valencia (Spain) on August 22, 2004. Receiver Operating Characteristic Analysis (ROC Analysis) is a powerful tool for cost/benefit analysis in decision making. Widely used in psychology and medicine for many decades, it has been introduced relatively recently in several areas of artificial intelligence: machine learning, multiagent systems, intelligent decision support and expert systems. In this context, ROC analysis provides techniques to select possibly optimal models and to discard suboptimal ones independently from (and prior to specifying) the cost context or the class distribution. Furthermore, the Area Under the ROC Curve (AUC) has been shown to be a better evaluation measure than accuracy in contexts with variable misclassification costs and/or imbalanced datasets. AUC is also the standard measure when using classifiers to rank examples, and, hence, is used in applications where ranking is crucial, such as campaign design, model combination, collaboration strategies, and co-learning. Nevertheless, there are several open questions and limitations that hamper a broader use and applicability of ROC analysis. Its connections with other evaluation measures is not yet completely clarified, its incorporation in decision support and expert systems technology just envisaged, its use for improving the decisions of (communities of) intelligent agents unexplored, and its use in data mining hasn’t yet reached its full potential. Among the limitations of ROC analysis, an important one, despite some recent progress, is its possible but difficult and computationally expensive extension to more than two classes. The European Conference on Artificial Intelligence offered us a great opportunity to foster the cross-fertilisation of ideas and applications with related areas in artificial intelligence. Hence the name given to this first workshop: ROC Analysis in Artificial Intelligence (ROCAI2004). We hope this workshop will be the start of a series of workshops on ROC Analysis. The technical program of ROCAI-2004 consists of one invited talk, eleven papers and a round table, organised in several sessions during this one-day workshop. The topics of these sessions cover a wide spectrum on ROC Analysis theory and applications, including new ideas and results, recent developments, and new research directions. We would like to thank everyone who contributed to make this workshop possible. First of all, we thank all the authors who submitted papers to ROCAI-2004. Each of them was reviewed by two members from the Program Committee, who finally accepted eleven papers for presentation. In this regard, we are grateful to the Program Committee and the additional reviewers for their excellent job. We are also thankful to the institutions who have supported this conference: the Network of Excellence on Knowledge Discovery (KDNet), the Generalitat Valenciana, the Technical University of Valencia and DSIC. Finally, we have to express our gratitude to the ECAI-2004 organisation for the facilities provided. Valencia, Spain August 2004 J. Hernández-Orallo, C. Ferri, N. Lachiche and P. Flach I II Program Committee • • • • • • • • • • • Hendrik Blockeel, K.U.Leuven, Belgium. Stephan Dreiseitl FHS Hagenberg, Austria. Tom Fawcett, HP Labs, Palo Alto, CA, USA. Cèsar Ferri, Technical University of Valencia, Spain. Peter Flach, University of Bristol, UK. Johannes Fürnkranz, TU Darmstadt, Germany. José Hernández-Orallo, Technical University of Valencia, Spain. Nicolas Lachiche, University of Strasbourg, France. Charles Ling, University of Western Ontario, London, Ontario, Canada. Maarten van Someren, University of Amsterdam, The Netherlands. Francesco Tortorella, University of Cassino, Italy. Organising Committee • • • • Cèsar Ferri, Technical University of Valencia, Spain. Peter Flach, University of Bristol, UK. José Hernández-Orallo, Technical University of Valencia, Spain. Nicolas Lachiche, University of Strasbourg, France. III IV Table of Contents An Empirical Evaluation of Supervised Learning for ROC Area ....................................................... 1 Rich Caruana, Alexandru Niculescu-Mizil Data Mining in Metric Space: An Empirical Analysis of Supervised Learning Performance Criteria ................................................................................................................ 9 Rich Caruana, Alexandru Niculescu-Mizil What ROC Curves Can’t Do (and Cost Curves Can) ......................................................................... 19 Chris Drummond, Robert C. Holte Cautious Classifiers................................................................................................................................. 27 Cèsar Ferri, José Hernández-Orallo ROC Optimisation of Safety Related Systems ..................................................................................... 37 Jonathan E. Fieldsend, Richard M. Everson Precision and Recall Optimisation for Information Access Tasks .................................................... 45 Michelle J. Fisher, Jonathan E. Fieldsend, Richard M. Everson ROC Analysis of Example Weighting in Subgroup Discovery ......................................................... 55 Branko Kavšek, Nada Lavrač, Ljupčo Todorovski Confidence Bands for ROC Curves: Methods and an Empirical Study........................................... 61 Sofus A. Macskassy, Foster Provost Optimizing Area Under Roc Curve with SVMs.................................................................................. 71 Alain Rakotomamonjy Learning Interestingness Measures in Terminology Extraction. A ROC-based approach .......................................................................................................................... 81 Mathieu Roche, Jérôme Azé, Yves Kodratoff, Michèle Sebag Learning Mixtures of Localized Rules by Maximizing the Area Under the ROC Curve ............................................................................................................................................... 89 Tobias Sing, Niko Beerenwinkel, Thomas Lengauer Author Index....................................................................................................................................... 97 V VI