• Medical Imaging
    2018 Event Highlights
    Special Events
    Grand Challenge 2019
    2019 Awards
    Travel to San Diego
    Onsite Services
    For Authors and Presenters
    For Chairs and Committees
Town and Country Resort & Convention Center
San Diego, California, United States
16 - 21 February 2019
Search Program:
Print PageEmail PageView My Schedule

Cancer Cellularity Challenge

Challenge image 2019


SPIE-AAPM-NCI BreastPathQ: Cancer Cellularity Challenge


The SPIE (The international society for optics and photonics), along with the American Association of Physicists in Medicine (AAPM), and the National Cancer Institute (NCI), will conduct a “Grand Challenge” on the development of quantitative biomarkers for the determination of cancer cellularity from whole slide images (WSI) of breast cancer hematoxylin and eosin (H&E) stained pathological slides. As part of the 2019 SPIE Medical Imaging Conference, the BreastPathQ Challenge will provide a unique opportunity for participants to compare their algorithms with those of others from academia, industry, and government in a structured, direct way using the same data sets.

This year, we have introduced a cancer cellularity scoring challenge for tumor burden assessment in breast pathology. Participants will be tasked to develop an automated method for analyzing histology patches extracted from whole slide images and assign a category/score reflecting cancer cellularity in each. Currently, this task is performed manually and relies upon expert interpretation of complex tissue structures. Furthermore, reproducibility of cancer cellularity scores is a concern in current practice, therefore a fully automated method holds great promise for increasing throughput and reducing inter- and intra-observer variability. 

For the BreastPathQ challenge, participants are asked to provide a floating-point score between 0 and 100% for each patch where the score represents the percentage of cellularity within the tumor bed of the patch. The training set consists of 2,579 patches of varying cellularity and the task will be to assign scores to an independent test set containing 1,121 patches. Participants are also required to submit a two-page manuscript describing their proposed solution.

Read the SPIE Press Release

Draft Timeline:

- Training data release: October 15, 2018
- Release date of validation set cases without truth: November 1, 2018
- Release date of test set cases without truth and validation truth:
   December 1, 2018
- Challenge submission close date: December 20, 2018
- Winners contacted: January 4, 2019
- Challenge results presented: February 16, 2019 at the
   SPIE Medical Imaging Conference

(From Peikari et al, Cytometry Part A, 91A: 10781087, 2017, Rajan et al, Cancer, 100:1365, 2004)

Neoadjuvant treatment (NAT) of breast cancer (BCa) is an option for patients with the locally advanced disease. Moreover, tumor response to the therapy provides useful information for patient management. In addition to the treatment effect on tumor size, NAT may alter the tumor cellularity. Tumor size many not decrease, but the overall cellularity may be markedly reduced, making residual tumor cellularity an important factor in assessing response.

The pathological examination of the tissue sections after surgery is the gold-standard to estimate the residual tumor and the assessment of cellularity is an important component of tumor burden assessment. Cellularity within the tumor bed is defined as the percentage area of the overall tumor bed that is comprised of tumor cells (invasive or in situ).  The most acceptable methods of assessing residual cancer burden in ongoing clinical trials and in clinical practice in some centers follows the algorithm proposed by an international working group. This algorithm takes into account several parameters including tumor cellularity. In the current clinical practice, tumor cellularity is manually estimated by pathologists on hematoxylin and eosin (H&E) stained slides, the quality, and reliability of which might be impaired by inter-observer variability which potentially affects prognostic power assessment in NAT trials.

Challenge Proposal

This challenge will invite participant to develop image analysis and machine learning algorithms to automatically assess cellularity in pathology whole slide image patches in terms of a continuous score. The reference standard is clinical assessment of cellularity by two breast pathologists.

A table of available training and test data available from Sunnybrook Health Sciences Centre to support this trial is given in the table below.

chart image

In this challenge, participants will be provided with a training set. Each have been assigned a tumor cellularity score by expert pathologists as reference standards. A validation set without truth will be released so that participants can gauge the performance of their algorithm before final submission of test results.  Participants will be able to submit their validation results and feedback will be provided to ensure proper data format and preliminary performance. Validation truth will be release in conjunction with the test data set to allow for full training using both training and validation data. The training and validation sets has reference standards from 1 pathologist, while the test set has reference standards from 2 pathologists.

The top three performers will be invited to present their algorithm and performance results at the 2019 SPIE Medical Imaging meeting in San Diego, CA.

Performance Metrics
Rank-based Kendall’s tau-b, averaged across the individual reference standard from pathologists 1 and 2

Challenge Platform
Harvard Challenge Platform via Jayashree Kalpathy-Cramer
Data distribution
The patch data will be distributed through the Harvard Challenge platform via Jayashree Kalpathy-Cramer
Long-term distribution of the full resolution WSI images will be through the Cancer Imaging Archive (TCIA).

Result and Prizes
Results to be distributed: January 4, 2019
Prizes: Complimentary registration and invitation to present method at SPIE Medical Imaging 2019 for up to three winners.


Anne Martel University of Toronto (anne.martel@sunnybrook.ca)
Shazia Akbar, University of Toronto (shazia.akbar@sunnybrook.ca)
Sharon Nofech-Mozes, Sunnybrook Health Sciences Centre
Nick Petrick, U.S. FDA (nicholas.petrick@fda.hhs.gov)
Marios Gavrielides, U.S. FDA (marios.gavrielides@fda.hhs.gov)
Berkman Sahiner, U.S. FDA (berkman.sahiner@fda.hhs.gov)
Kenny Cha, U.S. FDA (kenny.cha@fda.hhs.gov)
Sam Armato, University of Chicago (s-armato@uchicago.edu)
Karen Drukker, University of Chicago (kdrukker@uchicago.edu)
Lubomir Hadjiiski, University of Michigan (lhadjisk@umich.edu)
Keyvan Farahani, NIH/NCI (farahank@mail.nih.gov)
Jayashree Kalpathy-Cramer, Harvard University
Diane Cline, SPIE (diane@spie.org)
Joel Saltz, Stony Brook University (joel.saltz@stonybrookmedicine.edu)
John Tomaszewski, Kaleida Health (jtomaszewski@KaleidaHealth.org)
Aaron Ward, Western University (aaron.ward@uwo.ca)
Horst Hahn, Fraunhofer MEVIS (horst.hahn@mevis.fraunhofer.de)
Kensaku Mori, Nagoya University (mori@nuie.nagoya-u.ac.jp)
SPIE-MI CAD Technical committee team
SPIE-MI Pathology Technical committee team

Receive email updates about SPIE Medical Imaging

Browse Defense, Security, and Sensing 2011 papers

Journal of Medical Imaging

Get email updates for the Journal of Medical Imaging.