Performance of AI versus humans in detecting coronary stenosis

In a post-hoc analysis of the Dutch PACIFIC-1 study, a novel AI algorithm outperformed physicians in detecting obstructive stenosis on CT angiography, particularly in patients with extensive coronary artery disease.

This summary is based on the publication of Bernardo R, Nurmohamed NS, Bom MJ, et al. - Diagnostic accuracy in coronary CT angiography analysis: artificial intelligence versus human assessment. Open Heart. 2025 Jan 11;12(1):e003115. doi: 10.1136/openhrt-2024-003115.

Introduction and methods

Background

Visual assessment of coronary CT angiography (CCTA) is time-consuming, highly dependent on the reader’s experience, and prone to interobserver variability [1,2]. As this may lead to overestimation of coronary stenosis and unnecessary interventions [2,3], increased accuracy and reproducibility of CCTA analysis are warranted.

Atherosclerosis imaging quantitative CT (AI-QCT) is a novel artificial intelligence (AI)–guided algorithm for assessment of coronary stenosis that has demonstrated high accuracy against invasive quantitative coronary angiography (QCA) and other modalities [3-6]. However, its performance has not been directly compared with that of human CCTA readers with different levels of experience.

Aim of the study

The study aim was to evaluate the diagnostic accuracy of AI-QCT and visual assessment by physician readers for coronary stenosis detection.

Methods

This was a post-hoc analysis of the previously published PACIFIC-1 (Prospective Comparison of Cardiac PET/CT, SPECT/CT Perfusion Imaging and CT Coronary Angiography With Invasive Coronary Angiography) study, a prospective, observational, controlled, clinical study conducted at the Amsterdam University Medical Centers in Amsterdam, the Netherlands from January 2012 through October 2014 [7]. In this study, 208 consecutive patients with new-onset stable chest pain and suspected coronary artery disease (CAD) underwent CCTA and invasive QCA.

CCTA images were interpreted independently by a level 3 reader (i.e., experienced radiologist with >5 years of work experience in clinical assessment of CT scans) and two level 2 readers (i.e., cardiologists in training), who were all blinded to each other’s interpretations, patient characteristics, and QCA results. For every major epicardial vessel, percent stenosis and stenosis category were scored according to the Coronary Artery Disease Reporting and Data System (CAD-RADS) expert consensus [8].

The external performance of AI-QCT and clinical readers in predicting obstructive CAD were compared with that of blinded core laboratory QCA (with stenosis ≥50% and ≥70% as reference standards) using an area under the curve (AUC) analysis.

Main results

• For the detection of stenosis ≥50% on a per-patient level, AI-QCT achieved an AUC of 0.91 (95%CI: 0.87–0.95), which was higher than the level 3 reader (0.77; 95%CI: 0.70–0.83; P<0.001) and both level 2 readers (0.79; 95%CI: 0.72–0.85 and 0.76; 95%CI: 0.69–0.83, respectively; both P<0.001).
• Similar results were found for the detection of stenosis ≥70%.
• At the per-vessel level, AI-QCT (AUC: 0.86; 95%CI: 0.82–0.89) and the level 3 reader (AUC: 0.82; 95%CI: 0.79–0.86; P=0.098) showed a comparable performance in detecting stenosis ≥50% on QCA, whereas the level 2 readers achieved the same albeit lower AUC (0.69; 95%CI: 0.65–0.74; P<0.001).
• Interobserver correlation analysis of CAD-RADS scores indicated AI-QCT had the highest concordance with QCA (98/206; 48%), followed by the level 3 reader (30%) and the two level 2 readers (19% and 24%, respectively).
• In patients with total plaque volumes above the median (i.e., relatively extensive CAD), AI-QCT achieved an AUC of 0.88 (95%CI: 0.82–0.94) in detecting stenosis ≥50%, which was higher than the level 3 reader (0.77; 95%CI: 0.68–0.86; P=0.015) and both level 2 readers (0.72; 95%CI: 0.62–0.82; P=0.007 and 0.72; 95% CI 0.62–0.83; P=0.004).

Conclusion

In this post-hoc analysis of the single-center PACIFIC-1 study, AI-QCT had the highest concordance with the reference standard invasive QCA in detecting obstructive stenosis. Visual assessment by experienced radiologists showed a similar performance as AI-QCT, but cardiologists in training performed less well, particularly in patients with extensive CAD. The authors conclude that “the most prominent benefit of AI-QCT implementation through a standardized approach to CCTA analysis may be in centers lacking highly experienced readers.”

Find this article online at Open Heart.

References

1. Jonas RA, Weerakoon S, Fisher R, et al. Interobserver variability among expert readers quantifying plaque volume and plaque characteristics on coronary CT angiography: a CLARIFY trial sub-study. Clin Imaging 2022;91:19–25.
2. Lu MT, Meyersohn NM, Mayrhofer T, et al. Central Core Laboratory versus Site Interpretation of Coronary CT Angiography: Agreement and Association with Cardiovascular Events in the PROMISE Trial. Radiology 2018;287:87–95.
3. Griffin WF, Choi AD, Riess JS, et al. AI Evaluation of Stenosis on Coronary CTA, Comparison With Quantitative Coronary Angiography and Fractional Flow Reserve: A CREDENCE Trial Substudy. JACC Cardiovasc Imaging 2023;16:193–205.
4. Choi AD, Marques H, Kumar V, et al. CT Evaluation by Artificial Intelligence for Atherosclerosis, Stenosis and Vascular Morphology (CLARIFY): A Multi-center, international study. J Cardiovasc Comput Tomogr 2021;15:470–6.
5. Lipkin I, Telluri A, Kim Y, et al. Coronary CTA With AI-QCT Interpretation: Comparison With Myocardial Perfusion Imaging for Detection of Obstructive Stenosis Using Invasive Angiography as Reference Standard. AJR Am J Roentgenol 2022;219:407–19.
6. Kim Y, Choi AD, Telluri A, et al. Atherosclerosis Imaging Quantitative Computed Tomography (AI-QCT) to guide referral to invasive coronary angiography in the randomized controlled CONSERVE trial. Clin Cardiol 2023;46:477–83.
7. Danad I, Raijmakers PG, Driessen RS, et al. Comparison of Coronary CT Angiography, SPECT, PET, and Hybrid Imaging for Diagnosis of Ischemic Heart Disease Determined by Fractional Flow Reserve. JAMA Cardiol 2017;2:1100–7.
8. Cury RC, Leipsic J, Abbara S, et al. CAD-RADSTM 2.0 - 2022 Coronary Artery Disease-Reporting and Data System: An Expert Consensus Document of the Society of Cardiovascular Computed Tomography (SCCT). J Cardiovasc Comput Tomogr 2022;16:536–57.