Share Email Print

Proceedings Paper

Hazards of data leakage in machine learning: a study on classification of breast cancer using deep neural networks
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

With the renewed interest in developing machine learning methods for medical imaging using deep-learning approaches, it is essential to reexamine data leakage. In this study, we simulated data leakage in the form of feature leakage, where a classifier was trained on the training set, but the feature selection was influenced by the performance on the validation set. A pre-trained deep-learning convolutional neural network (DCNN) without fine-tuning was used as a feature extractor for malignant and benign mass classification in mammography. A feature selection algorithm was trained in the wrapper mode with a cost function tuned to follow the performance metric on the validation set. Linear discriminant analysis (LDA) classifier was trained to classify masses on mammographic patches. Mammograms from 1,882 patient cases with 4,577 unique patches were partitioned by patient into 3,222 for training and 508 for validation, while 847 were sequestered as unseen independent test set to evaluate the generalization error. The effects of the finite sample size on data leakage were studied by varying the training and validation set sizes from 10% to 100% of the available sets. The area under the receiver operating characteristic curve (AUC) was used as the performance metric. The results show that the performance on the validation set could be overestimated, having AUCs of 0.75 to 0.99 for various sample sizes, whereas the independent test performance could realistically only reach an AUC of 0.72. The analysis indicates that deep learning can risk a high inflation in performance and proper housekeeping rules should be followed when designing and developing deep learning methods in medical imaging.

Paper Details

Date Published: 16 March 2020
PDF: 6 pages
Proc. SPIE 11314, Medical Imaging 2020: Computer-Aided Diagnosis, 1131416 (16 March 2020); doi: 10.1117/12.2549313
Show Author Affiliations
Ravi K. Samala, Univ. of Michigan (United States)
Heang-Ping Chan, Univ. of Michigan (United States)
Lubomir Hadjiiski, Univ. of Michigan (United States)
Sathvik Koneru, Univ. of Michigan (United States)

Published in SPIE Proceedings Vol. 11314:
Medical Imaging 2020: Computer-Aided Diagnosis
Horst K. Hahn; Maciej A. Mazurowski, Editor(s)

© SPIE. Terms of Use
Back to Top
Sign in to read the full article
Create a free SPIE account to get access to
premium articles and original research
Forgot your username?