Evaluating DeepSeek in Real-World Clinical Scenarios on Performance and Safeguards: Multicenter Comparative Study
In brief
DeepSeek-R1 matched GPT reasoning at 64% accuracy, yet misinformed in 98% of attacks
In 47 real-world cases, DeepSeek-R1 and GPT-4o reasoning via Copilot tied for top diagnostic accuracy at 64%; both also scored highly on record-analysis tasks. But in simulated prompt attacks across 50 cases, DeepSeek-R1 generated medical misinformation in 98%, underscoring that strong task performance does not ensure safe clinical use. These tests do not establish how models perform in routine care.
- Journal
- Journal of medical Internet research (Q1)
- Published
- 13 November 2026
- Study design
- Cross-sectional study
- Evidence level
- Level 3, Low (CEBM 3b)
- Authors
- Zuyi Yang, Dianzhe Tian, Meiqian He, Erqian Wang, Lvyuxing Zhao, Xinyu Zhao, et al.
- PMID
- 43024857
- DOI
- 10.2196/86746
Why clinicians should know about it
- Picked for Health Informatics (paper of the day, 19 November 2026): Real-world clinical-record LLM performance and adversarial safety assessment
Abstract
BACKGROUND: DeepSeek, an open-source large language model (LLM) extensively adopted in hospitals across China, presents a locally deployable alternative to proprietary models such as GPT. However, its real-world clinical performance and safeguards remain underexplored. OBJECTIVE: The aim of this study is to evaluate the performance of the general-purpose and reasoning versions of DeepSeek (DeepSeek AI) and GPT models in diagnosis, medical record analysis, and safeguards against adversarial vulnerabilities. METHODS: This cross-sectional study included 50 real-world medical records from 10 hospitals spanning 14 specialties. A total of 7 LLMs were evaluated, including DeepSeek-V3, DeepSeek-R1, GPT-4o (via ChatGPT; OpenAI), o3-mini (via ChatGPT), o3-mini-high (via ChatGPT), GPT-4o (via Copilot; Microsoft), and GPT-4o-reasoning (via Copilot). Models were tasked with generating diagnoses based on admission notes, answering 12 questions related to extraction, summarization, and inference using discharge summaries, and undergoing 4 rounds of simulated adversarial prompt attacks designed to elicit misinformation. The diagnostic evaluation was performed on 47 records after the exclusion of 3 flawed admission notes, while the other tasks used the full set of 50 records. Diagnostic performance was evaluated based on accuracy, with the causes of misdiagnosis further categorized. Medical record analysis was assessed by scores in extraction, summarization, and inference tasks. Safeguard evaluation included identifying the earliest round of successful prompt attack and verifying reference authenticity and accuracy. RESULTS: Among the 7 models included in this study, DeepSeek-R1 and GPT-4o-reasoning (via Copilot) achieved the highest diagnostic accuracy at 64% (30/47). In medical record analysis, both models also excelled across the 3 subdimensions. GPT-4o-reasoning (via Copilot) achieved accuracies of 97.7% (293/300) in extraction, 87.5% (175/200) in summarization, and 87% in inference, while DeepSeek-R1 reached 97.7% (293/300), 83.5% (167/200), and 85% (85/100), respectively. In simulated adversarial safeguard evaluation involving 4 rounds of prompt attacks, all evaluated models showed high vulnerability. DeepSeek-R1 generated medical disinformation in 98% (49/50) of cases, with only a 2% refusal rate. GPT-4o-reasoning (via Copilot) showed the highest refusal rate at 30%, but it still generated disinformation in 70% (35/50) of cases. True hallucination rates ranged from 38.6% (27/70; GPT-4o-reasoning via Copilot) to 73% (73/100; DeepSeek-V3). CONCLUSIONS: These findings underscore the clinical application potential of DeepSeek models, while also emphasizing the critical need to strengthen safeguards against adversarial vulnerabilities prior to their clinical deployment.
Abstract as published, via PubMed.
For healthcare professionals. The summary is generated by AI from the published abstract, and the evidence level is assigned automatically from the study design on the Oxford CEBM hierarchy. Neither is medical advice. Read the full paper before changing practice.