Evaluation of Computational Intelligence Decision Makers: Receiver Operating Characteristic (ROC), Jackknife, Bootstrap and other Statistical Methodologies

WCCI 2014 Tutorial

The course will cover the broad range of performance metrics that go beyond the simple calculation of success rate or accuracy on a limited test set. These include sensitivity, specificity, positive and negative predictive value, receiver operating characteristic (ROC) curve, and area under the ROC curve (AUC). Particular attention will be paid to the ROC curve formalism, and a hands-on demonstration will be given. Participants will be asked to bring a laptop or other computational device and will take part in simulated reader studies to better understand ROC methodology and to compete for prizes. Both parametric and nonparametric uncertainty calculation techniques will be discussed, and the importance of good data hygiene in maintaining complete separation of training and test data will be illustrated.

Tutorial Slides in PDF

Tutorial Slides in Powerpoint

Brief Bio

David Brown is recently retired from a 40 year career in medical imaging and CAD performance assessment methodology research at the U.S. Food and Drug Administration. He is an SPIE fellow and member of the college of fellows of the International Neural Network Society, of which he was a long time Board member and former President.











Brief Bio

Frank Samuelson received his Ph.D. in Astrophysics from Iowa State University, and was a post-doctoral fellow at the Los Alamos National Laboratory. Since 2004 he has worked for the US Food and Drug Administration in the evaluation of imaging devices.








Construction of the ROC Curve

Construction of the ROC Curve


Dependence of ROC Curve on Separation of Distributions

Dependence of ROC Curve on Separation of Distributions














人工智能分类决策算法的评估：受试者工作特征(ROC)曲线，刀切(Jackknife)，自助(Bootstrap)，以及其他统计方法

本课程将涵盖二值分类中的多种性能评价指标，超越简单的基于有限样本的成功率或正确率。这些包括灵敏度，特异度，阳性预测值，阴性预测值，受试者工作特征(ROC)曲线，ROC曲线下面积等。重点将放在ROC曲线的形成，并给于现场实习演示。为了帮助理解ROC方法，建议参与者自带一台手提电脑或者其他计算机装置以参与一个模拟医学图像读片实验，优者有奖励。本课程还将讨论参数化和非参数化的不确定度（方差，可信区间）计算技术，并将说明在训练和测试算法中保持好的数据“卫生”的重要性。