Human-AI Collaboration In Radiology: The Case Of Pulmonary Embolism

Abstract: We study how radiologists use AI to diagnose pulmonary embolism (PE), the third leading cause of cardiovascular death in the U.S. We track over 100,000 scans interpreted by nearly 400 radiologists during the staggered rollout of an FDA-approved diagnostic platform in a large hospital system. When AI flags PE, radiologists agree 84% of the time; when AI predicts no PE, they agree 97%. Disagreement evolves substantially: radiologists initially reject AI-positive PEs in 30% of cases, dropping to 12% by year two. Despite a 16% increase in scan volume, diagnostic speed remains stable while per-radiologist monthly volumes nearly double. We document substantial heterogeneity in AI collaboration: some radiologists reject AI-flagged PEs half the time while others accept nearly always; female radiologists are 6 percentage points less likely to override AI than male radiologists. Moderate AI engagement correlates with highest agreement, whereas both low and high engagement show more disagreement. Follow-up imaging reveals that when radiologists override AI to diagnose PE, 54% of subsequent scans show both agreeing on no PE within 30 days. Patient outcomes differ systematically by agreement category. When both AI and radiologist agree on PE presence, 30-day mortality increases 3.4 percentage points relative to concordant negatives. When only the radiologist diagnoses PE, mortality increases 5.5 percentage points. When only the AI detects PE but the radiologist does not, mortality increases 4.0 percentage points. Radiologist overrides to diagnose PE correlate with higher rates of right heart failure and respiratory failure than AI positives rejected by radiologists. These patterns may reflect radiologists identifying clinically significant cases that AI misses, radiologist diagnoses triggering more aggressive treatment, or diagnostic errors where radiologists see PE in sicker patients who lack it. The modest mortality differences across disagreement categories suggest patient severity matters more than who detects the PE.