High-Throughput Truthing of Whole Slide Images to Validate Artificial Intelligence Algorithms

Study Details

Information about this study has been provided by the principal investigator.

We are crowdsourcing pathologists to collect annotations of the density of tumor-infiltrating lymphocytes in breast cancer to create a validation dataset for evaluating algorithm performance. Through this process, we will learn about and develop tools to evaluate pathologist agreement of their correlated quantitative evaluations. The images and annotations will be available to perception researchers to learn and develop tools to evaluate pathologist agreement. The images and annotations will also be available to algorithm developers to validate their algorithm performance in a submission to the FDA/CDRH.

Visit Study Website exit disclaimer


Brandon Gallas


Katherine Elfer


Participant Qualifications

  • Pathologists

Principal Investigator

Brandon Gallas, Ph.D.


Center for Devices and Radiological Health, U.S. Food and Drug Administration

IRB Approval Number

Exempt determination # 2019-CDRH-109

These experiments are not conducted by the National Cancer Institute (NCI). NCI features studies on this page as a resource to the extramural community. The listing of any study, PI, or organization does not imply endorsement in the study’s validity or any research entity. For questions, contact Todd Horowitz.

Last Updated
November 23, 2020