01The problem
Brazil's Sistema Único de Saúde (SUS, Unified Health System) serves over 200 million people, guided by official clinical guidelines published by the Ministry of Health — Clinical Protocols and Therapeutic Guidelines (PCDTs), Protocols of Use, Oncology Diagnostic and Therapeutic Guidelines, National Guidelines, and Care Pathways. They specify diagnostic criteria, dosages, treatment regimens, and monitoring parameters, and adherence is mandatory for SUS managers.
Yet today's LLMs — even frontier models — are trained predominantly on English-language medical literature. Before this work, no benchmark measured factual accuracy grounded in Brazilian guidelines, and no open model had been specifically adapted to the official PCDTs of the SUS.
02Pipeline
From the 178 clinical guidelines (~5.4M tokens), we generate 17,800 synthetic documents (~70M tokens) in three complementary formats — paraphrases, Wikipedia-style articles, and question-answer pairs — using four different generators (GPT-4.1-mini, GPT-5-nano, GPT-OSS-20B, Qwen3-235B). This corpus is used for continual pre-training of Qwen2.5-14B-Instruct, followed by GRPO using the train split of HealthBench-BR as the reward signal.
03The benchmarks
As one of the central contributions, we release two open benchmarks derived directly from the official guidelines — designed to reproducibly measure how much a model has absorbed of Brazilian clinical knowledge:
HealthBench-BR
1,780 items · paired T/F
Paired true/false statements. For each fact, we generate a true version and a false version that modifies one critical detail (dose, route, interval). Getting it right requires precise factual recall.
PCDT-QA
890 items · LLM-as-judge
Open-ended questions paired with reference answers, judged by GPT-4.1 in a binary correct/incorrect decision. They span broad topics down to specific factual queries.
Examples
↔ Click the tabs to switch between the two datasets (content is in Portuguese, since these are Brazilian protocols):
04Main results
The final model outperforms all proprietary baselines — including Gemini 3.1 Pro, the strongest of them — on both benchmarks. Even Google AI Overview (search + LLM, with an inherent advantage: it abstains on 11.7% of HB-BR questions and 4.9% of PCDT-QA questions) falls behind:
| Model | HealthBench-BR | PCDT-QA |
|---|---|---|
| Qwen2.5-14B (baseline) | 59.4 | 27.9 |
| + CPT | 69.6 | 66.3 |
| + CPT (4 generators) | 71.1 | 86.3 |
| + CPT (4 generators) + GRPO | 83.9 | 85.4 |
| GPT-4.1 | 73.3 | 70.3 |
| GPT-5.2 (high) | 78.5 | 78.2 |
| Claude Sonnet 4.6 | 77.6 | 70.3 |
| Gemini 3.1 Pro | 79.3 | 80.0 |
| Google AI Overview | 70.5 | 77.3 |
05Generator diversity matters more than scale
The largest gain came from combining four different generator models rather than a single one. With one generator (GPT-4.1-mini), CPT reaches 66.3% on PCDT-QA. With four, it jumps to 86.3% — +20 points.
Notably, generator size alone does not translate into gains: Qwen3-235B (235B parameters) does not dominate any benchmark by itself, and self-augmenting with the target Qwen2.5-14B-Instruct is the weakest case (60.2% HB / 50.1% PCDT) — its outputs are too short on average (~1,594 tokens vs. 3,471–4,551 for the others), omitting dosages and monitoring details. Diversity of phrasing and style matters more than raw token volume.
06RL fixes the sycophancy CPT introduces
HealthBench-BR is perfectly balanced (50% true / 50% false). The base Qwen2.5-14B is well calibrated (49.9% "True"), but CPT introduces an agreement bias (58.9%) — likely because the corpus tends to phrase things affirmatively. GRPO corrects this: our best model returns to 53.0%, and the accuracy gain on HB is +12.8 points over CPT alone.
Among frontier models, Claude Sonnet 4.6 (49.2%) and GPT-5.2-high (48.7%) are well calibrated; Gemini 3.1 Pro shows moderate bias (63.3%). Google AI Overview is the most sycophantic system evaluated: it predicts "True" 69.6% of the time — concerning given that it is currently the default AI-generated source of clinical information in Google Search.
07RAG pushes performance further
Combining our best model with retrieval (top-10 chunks from the guidelines) reaches 95.3% on HB-BR and 93.3% on PCDT-QA. An interesting detail: retrieval type interacts with task format — BM25 wins on HB (lexical verification of dosages/intervals), embeddings win on PCDT-QA (semantic retrieval). And there's a counterintuitive observation: CPT alone + RAG hurts HB-BR (73.9% vs. 90.0% from baseline + RAG), because the trained model tends to trust its parameters and ignore the evidence. GRPO reverses this completely — the verdict reward teaches the model to ground its answer in the retrieved context.
08Catastrophic forgetting
Training on a clinical domain can degrade general capabilities. We evaluate on 5 out-of-distribution benchmarks (HealthQA-BR, DrBode, MedPT, BLUEX, IFEval), comparing Full FT, LoRA, and Replay (1:1 mix with FineWeb). Findings: Full FT sits on the favorable side of the knowledge↔retention frontier; LoRA, counterintuitively, loses on both axes — consistent with the intruder dimensions effect reported by Shuttleworth et al.
| Data | Method | HB | PCDT | HealthQA | DrBode | MedPT | BLUEX | IFEval | Avg |
|---|---|---|---|---|---|---|---|---|---|
| — | Baseline | 59.4 | 27.9 | 67.8 | 65.3 | 56.4 | 75.0 | 79.7 | 68.8 |
| 1 gen | Full FT | 69.6 | 66.3 | 66.0 | 62.7 | 56.4 | 73.5 | 75.8 | 66.9 |
| LoRA | 67.2 | 65.2 | 64.1 | 56.6 | 53.8 | 71.1 | 68.6 | 62.8 | |
| Replay | 63.8 | 65.6 | 67.0 | 64.5 | 55.9 | 72.7 | 77.1 | 67.4 | |
| 4 gen | Full FT | 71.1 | 86.3 | 63.3 | 61.2 | 54.1 | 72.8 | 69.3 | 64.1 |
| LoRA | 70.3 | 77.1 | 60.8 | 58.3 | 52.1 | 68.9 | 65.6 | 61.1 | |
| Replay | 69.2 | 80.0 | 65.7 | 59.2 | 54.0 | 73.2 | 69.3 | 64.3 |
09Ethical considerations
The model is intended to support, not replace, clinical decisions. Even at 83.9% accuracy, residual errors may involve consequential details (dosages, contraindications). It is a research model, not a certified medical device, and should only be used under professional supervision.
Since all training data comes from public Ministry of Health documents, privacy risks from clinical records are avoided — but biases from the source documents and the generator models may persist. Our sycophancy results reinforce that clinical AI systems should be evaluated not only for accuracy but also for their tendency to confirm false user assertions. Releasing open weights trained on Brazilian protocols is a step toward equitable access for Portuguese-speaking clinicians and patients historically underserved by English-centric LLMs.