Marriott Marquis Houston
Houston, Texas, United States
15 - 20 February 2020
Course (SC1295)
From Analytic to Clinical Validation: Moving AI/ML into Practice
  • Instructors:
  • William Y. Hsu, Univ. of California, Los Angeles (United States);
  • Matthew S. Brown, Univ. of California, Los Angeles (United States);
  • Robert M. Nishikawa, Univ. of Pittsburgh (United States);
  • Elizabeth A. Krupinski, Emory Univ. School of Medicine (United States);
Tuesday 18 February 2020
8:30 AM - 5:30 PM

Member Price $565.00
Non-Member Price $660.00
Student Member Price $294.00
  • Course Level:
  • Intermediate
  • CEU:
  • 0.7
Artificial Intelligence (AI) is increasingly being used in a wide variety of medical imaging applications. Most of the focus, however, is on algorithm and scheme development, but this is only part of the picture. In order to have an impact on clinical decision making, workflow and patient care these AI tools must be evaluated using real-world cases and actual clinical providers that are expected to use them in routine care. The techniques used to conduct these types of studies are less well known in this field thus investigators need to be trained the proper study design and analysis methods. This course will cover basic principles, techniques, and process for validating models developed using artificial intelligence (AI)/machine learning (ML) techniques. The primary goal of this course is to help the audience understand and apply fundamental principles related to designing, executing, and interpreting model evaluation studies. The course will be organized around two parts: analytic validation and clinical validation. In the first half, the audience will be exposed to approaches for performing a technical validation of a prediction model, including different study designs, appropriate statistical tests, metrics, dataset considerations, and decision curve analysis. The second half will cover the process of undertaking clinical validation that would address real-world use of models, regulatory and deployment issues. Topics include workflow integration, prospective clinical trials, reader impact studies, and regulatory approvals. Examples will focus on imaging-related models that are drawn from literature and the instructors’ personal experiences in prognostic modeling, computer-aided diagnosis, and imaging biomarker development.
Learning Outcomes
  • Describe the process of analytic and clinical validation, elucidating the steps involved and the considerations in designing an evaluation
  • Explain basic concepts (e.g., training/testing/validation, cross-validation) and metrics (e.g., precision/recall) related to algorithm evaluation
  • Choose the appropriate study design and metrics for comparing algorithms, depending on data, model, and objective
  • Interpret various metrics to determine whether one algorithm is superior over others for a specific task
  • Identify sources of potential biases that may influence model performance due to characteristics of the training or target populations
  • Identify potential steps in moving beyond an algorithm paper to deploying AI/ML in clinical practice.
  • Identify appropriate methods to assess the impact of AI techniques and tools on observer accuracy.
Data scientists, health information technology practitioners, and clinician informaticists who wish to learn about the process of evaluating prediction models and the considerations involved in assessing the suitability and impact of adopting a model in a clinical environment. Basic biostatistics (hypothesis testing, statistical tests) and a working knowledge of machine learning concepts (how models are constructed, types of machine learning algorithms) are assumed.
About the
William Y. Hsu PhD is an Associate Professor of Radiological Sciences at the University of California, Los Angeles and a member of the Medical & Imaging Informatics group. Dr. Hsu’s research interest is in data integration and machine and reinforcement learning with applications in improving diagnostic decisions support to improve early detection of cancers. He has developed and evaluated many machine learning-based algorithms that utilize clinical, imaging, and molecular data as inputs. In his role as a Deputy Editor for the Radiology: Artificial Intelligence journal, he has developed an interest in improving the transparency and consistency of reporting of model evaluations.
Matthew S. Brown PhD is a Professor of Radiological Sciences at the University of California, Los Angeles and the Director of the Center for Computer Vision and Imaging Biomarkers (CVIB) at UCLA. Dr. Brown’s research interests include medical image segmentation, quantitative analysis, and imaging biomarker development. He is a co-founder of MedQIA, a company that provides image analysis services for clinical trials to the biotech, pharma, and medical device industries.
Robert M. Nishikawa PhD is Professor of Radiology at the University of Pittsburgh. His research interests are in developing quantitative imaging techniques for breast imaging; image quality assessment; and evaluation of imaging technologies, specifically, the clinical effectiveness of computer aids for radiologists. He has over 200 scientific publications and has been a consultant for several medical imaging companies on digital imaging and computer-aided diagnosis. He is a fellow of SPIE, AAPM, AIMBE, and the Society of Breast Imaging.
Elizabeth A. Krupinski PhD is an Experimental Psychologist with research interests in medical image perception, observer performance, medical decision making, and human factors as they pertain to radiology and telemedicine. The goal of her research is to improve our understanding of the perceptual and cognitive mechanisms underlying the interpretation of medical images in order to reduce errors, improve training, and optimize the reading environment, thereby improving patient care and outcomes.
Course level is Intermediate to Advanced.
Back to Top