Generalist Models in Specialized Domains: Evaluating Contrastive Language-image Pre-training for Zero-shot Anomaly Detection in Brain MRI

Zero-shot anomaly detection (ZSAD) is gaining traction in medical imaging as a way to identify abnormalities without task-specific supervision. In this work, we benchmark state-of-the-art CLIP-based ZSAD models —originally developed for industrial inspection —on brain metastasis detection using the BraTS-MET dataset. We evaluate both general-purpose and medical-adapted variants across multiple training paradigms with little to no supervision, emulating real-world scenarios with scarce labeled imaging data. While the models can apply general knowledge to medical images, we show that their accuracy remains limited, especially in peripheral brain regions, and that substantial but still suboptimal performance gains are achieved only via domain-specific fine-tuning. Our findings highlight current limitations in spatial consistency when using 2D-based approaches for 3D problems, and suggest that adaptation is required to make CLIP-based ZSAD viable for clinical use.