AI safety in medicine is crucial to prevent catastrophic errors in patient care with the rise of large language models.