Mercer Science and Engineering Fair | Fostering STEM Education in Mercer County, NJAffiliated with Regeneron ISEF and Thermo Fisher Junior Innovators Challenge

Problem being addressed
1. To evaluate the efficacy of combining predictive artificial intelligence and an image similarity model to risk stratify thyroid nodules, using a retrospective study.

Hypothesis
1. Incorporating image similarity analysis into predictive AI models is a reliable method for assessing malignancy in thyroid nodules using static ultrasound images.

Procedures
1. Prior to Data Collection
  1. Read articles about Thyroid cancer physiology, prevalence, and current standards of diagnosis, evaluation and treatment.
  2. Discussed these articles in depth via multiple zoom sessions with Dr. Johnson to get a foundational understanding of the topic.
  3. Also reviewed how the AI tool was developed and discussed how AI and machine learning can shape the future of medicine.
2. Data Collection
  1. Downloaded Stanford AIMI open-source data set of 192 thyroid nodule images gathered between April 2017 to May 2018.
  2. Organized data into an Excel spreadsheet according to age, gender, nodule size, TI-RAD scores, and histopathology.
    1. The inclusion criteria were males and females, aged 18 years with thyroid surgery or biopsy at participating sites with a definitive diagnosis by cytology or pathology.
    2. The exclusion criteria were patients below the age of 18; indeterminate thyroid nodules without a definitive diagnosis; ultrasound images of thyroid nodules containing annotations, markings, writings, or crosshair within the nodule and whole thyroid nodule not visible in the ultrasound section; metastasis to the thyroid from other malignancies as well as lymphoma of the thyroid were also excluded; multinodular goiters without a clearly separable nodule on ultrasound images and nodules that underwent radioactive iodine treatment, ethanol ablation, radiofrequency ablation or laser ablation.
  3. Organized the images from the data sets into separate folders for malignant vs. benign nodules.
  4. Fed each image through the AiBx algorithm, recorded results (TI-RAD scores and benign vs. malignant) in an Excel spreadsheet, and documented the results alongside the actual results (please see Excel data sheets).
  5. Worked with an Endocrinologist at Princeton Medical Group (private endocrinology practice that does in-house thyroid ultrasounds and FNAs) to obtain second data set that had 118 thyroid nodule images from 2018-2023. The images, TI-RAD scores, and histopathology results were provided by the Endocrinologist. Permission was obtained from the institution.
  6. Repeated same process as in steps ii-iv for this data set.
3. Data Analysis
  1. Calculated raw data for the data sets comparing published data to predicted data by the algorithm.
    1. Recorded which results were consistent between the algorithm and the data set, as well as which results differed.
  2. Used raw data to create a table of true positive, true negative, false positive, and false negative values.
  3. Used this information to calculate negative predictive value, positive predictive value, sensitivity, and specificity using respective formulas.
    1. Negative predictive value: (True Negatives)/(True Negatives + False Negatives)
    2. Positive predictive value: (True Positives)/(True Positives + False Positives)
    3. Sensitivity: (True Positives)/(True Positives + False Negatives)
    4. Specificity: (True Negatives)/(True Negatives + False Positives)

	Stanford Data	Private Data
Sensitivity	1	0.91
Specificity	0.55	0.95
PPV	0.18	0.8
NPV	1	0.98
AUC ROC	0.78	0.93

Calculated AUCROC, Pearson correlation coefficient, Polychoric Correlation, and Cohen’s Kappa using Python and R Studio statistical analysis tools. These results were used to verify how accurate the algorithm was in predicting the malignancy of a nodule as well as its correlation with the published TI-RAD score.
The risk reduction formula was also calculated to predict how many potential FNAs could have been avoided if this algorithm was used in a real-life setting.
1. Formula: (Correctly predicted AiBx malignant nodules)/(Malignant nodules defined by FNA histopathology)

Bibliography

1. Popoveniuc G, Jonklaas J. Thyroid Nodules. Med Clin North Am. 2012;96(2):329-349. doi:10.1016/j.mcna.2012.02.002

2. Kamran SC, Marqusee E, Kim MI, et al. Thyroid nodule size and prediction of cancer. J Clin Endocrinol Metab. 2013;98(2):564-570. doi:10.1210/jc.2012-2968

3. Jasim S, Dean DS, Gharib H. Fine-Needle Aspiration of the Thyroid Gland. In: Feingold KR, Anawalt B, Blackman MR, et al., eds. Endotext. MDText.com, Inc.; 2000. Accessed February 19, 2024. http://www.ncbi.nlm.nih.gov/books/NBK285544/

4. Yip L, Farris C, Kabaker AS, et al. Cost Impact of Molecular Testing for Indeterminate Thyroid Nodule Fine-Needle Aspiration Biopsies. J Clin Endocrinol Metab. 2012;97(6):1905-1912. doi:10.1210/jc.2011-3048

5. Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25(1):44-56. doi:10.1038/s41591-018-0300-7

6. Tessler FN, Thomas J. Artificial Intelligence for Evaluation of Thyroid Nodules: A Primer. Thyroid Off J Am Thyroid Assoc. 2023;33(2):150-158. doi:10.1089/thy.2022.0560

7. Krupinski EA. Current perspectives in medical image perception. Atten Percept Psychophys. 2010;72(5):10.3758/APP.72.5.1205. doi:10.3758/APP.72.5.1205

8. Tessler FN, Middleton WD, Grant EG, et al. ACR Thyroid Imaging, Reporting and Data System (TI-RADS): White Paper of the ACR TI-RADS Committee. J Am Coll Radiol JACR. 2017;14(5):587-595. doi:10.1016/j.jacr.2017.01.046

9. Yamashita R, Kapoor T, Alam MN, et al. Toward Reduction in False-Positive Thyroid Nodule Biopsies with a Deep Learning-based Risk Stratification System Using US Cine-Clip Images. Radiol Artif Intell. 2022;4(3):e210174. doi:10.1148/ryai.210174

10. Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: Machine Learning in Python. J Mach Learn Res. 2011;12(85):2825-2830.

11. Wildman-Tobriner B, Taghi-Zadeh E, Mazurowski MA. Artificial Intelligence (AI) Tools for Thyroid Nodules on Ultrasound, From the AJR Special Series on AI Applications. AJR Am J Roentgenol. 2022;219(4):1-8. doi:10.2214/AJR.22.27430

12. Park VY, Han K, Seong YK, et al. Diagnosis of Thyroid Nodules: Performance of a Deep Learning Convolutional Neural Network Model vs. Radiologists. Sci Rep. 2019;9(1):17843. doi:10.1038/s41598-019-54434-1

13. He LT, Chen FJ, Zhou DZ, et al. A Comparison of the Performances of Artificial Intelligence System and Radiologists in the Ultrasound Diagnosis of Thyroid Nodules. Curr Med Imaging. 2022;18(13):1369-1377. doi:10.2174/1573405618666220422132251

14. Barredo Arrieta A, Díaz-Rodríguez N, Del Ser J, et al. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Inf Fusion. 2020;58:82-115. doi:10.1016/j.inffus.2019.12.012

15. What is Explainable AI? - Unite.AI. Accessed February 19, 2024. https://www.unite.ai/what-is-explainable-ai/

16. Cadario R, Longoni C, Morewedge CK. Understanding, explaining, and utilizing medical artificial intelligence. Nat Hum Behav. 2021;5(12):1636-1642. doi:10.1038/s41562-021-01146-0

17. Chen H, Gomez C, Huang CM, Unberath M. Explainable medical imaging AI needs human-centered design: guidelines and evidence from a systematic review. Npj Digit Med. 2022;5(1):1-15. doi:10.1038/s41746-022-00699-2

18. McNamara M. Explainable AI: What is it? How does it work? And what role does data play? Published February 22, 2022. Accessed February 19, 2024. https://www.netapp.com/blog/explainable-ai/

19. Tsopra R, Fernandez X, Luchinat C, et al. A framework for validating AI in precision medicine: considerations from the European ITFoC consortium. BMC Med Inform Decis Mak. 2021;21(1):274. doi:10.1186/s12911-021-01634-3

20. Riley RD, Archer L, Snell KIE, et al. Evaluation of clinical prediction models (part 2): how to undertake an external validation study. BMJ. 2024;384:e074820. doi:10.1136/bmj-2023-074820

Questions and Answers

Iniitial project questions

1. What was the major objective of your project and what was your plan to achieve it?

The major objective of this project is to potentially help people avoid invasive procedures and decrease healthcare costs and patient burden, as FNAs can be daunting and a source of stress for patients. My plan to achieve it was through contacting Dr. Johnson Thomas who developed an AI algorithm called AiBx that predicted malignancy risk of thyroid nodules using ultrasound images. We met via Zoom where he explained all the details of this tool and what it was intended for. Part of our discussion included conducting studies with large data sets to show the accuracy of this AI tool and if it can be used in the health care setting reliably in the future. I expressed my interest in conducting such a study and worked together to find available datasets to get the project started.

2. What were the major tasks you had to perform in order to complete your project?

Obtaining and analyzing the data was a big part of what I did. I helped with the statistical analysis and wrote the abstract, discussion, results, and conclusion of the paper.

3. What is new or novel about your project?

Image similalrity was not used in the other AI tools that are currently available for thyroid nodule prediction. Adding this extra piece of information allows the AI to continue to learn and change its recommendations based on what it got right and what it got wrong.

4. What was the most challenging part of completing your project?

a. What problems did you encounter, and how did you overcome them?

The most challenging part was obtaining a usuable dataset. Stanford's dataset was the only one that was publically available. To obtain other data sets, I had to ask many practices if they would be willing to provide addtional data sets. Luckily, one agreed to do so.

b. What did you learn from overcoming these problems?

I learned that perserverance pays off and not to give up easily.

5. If you were going to do this project again, are there any things you would you do differently the next time?

As stated in the discussion, would consider designing the study with video of thyroid nodules instead of static images. Would also consider using larger data sets if feasible.

6. Did working on this project give you any ideas for other projects?

Yes, it did. I am thinking about creating an App for this algorithm to use in real time

7. How did COVID-19 affect the completion of your project?

It did not affect my project

Combining Image similarity and Predictive AI Models to Decrease Subjectivity in Thyroid Nodule Diagnosis and Improve Malignancy Prediction

Abstract:

Bibliography/Citations:

Additional Project Information

Research Plan:

Questions and Answers