There are thousands of diseases with substantial unmet therapeutic needs and tens of thousands of existing drugs that could, in principle, be tested against them. Yet only a small fraction of possible drug-disease combinations will ever be evaluated experimentally due to the raw combinatorial scale.
Drug-repurposing methods aim to narrow this space to a tractable set of hypotheses worth testing further. Existing benchmarks evaluate models on whether they can recover held-out drug-disease associations. For LLMs, however, many of these associations are likely to have appeared in the pretraining corpus, making it difficult to distinguish scientific reasoning from retrieval of known relationships. RepurposingBench addresses this problem by constructing evaluation sets from drug-repurposing hypotheses that are outside of a model's knowledge. The focus is to evaluate the model on tasks for which it must reason over biomedical knowledge to rediscover known hypotheses rather than simply recall them.
Constructing the benchmark
To construct the benchmark, we search the web for repurposing hypotheses published in the form of newly initiated clinical trials, case reports, preprints, patents, and other public disclosures of existing drugs being investigated for new diseases in vivo. For each candidate, we search for all mentions and record the date of the first mention of the drug-disease association. Each model's knowledge of the drug is then assessed under a variety of prompts. If the drug-disease association is not recovered by any model, then it becomes part of the test set. If the drug-disease association was not recovered and the earliest date of mention is after 2-16-2026, GPT-5.6's knowledge cutoff, the candidate becomes a part of the prospective set.
Knowledge probe prompts:
- "What are all the diseases that [drug] has been used to treat or has been tested as a treatment for? List every disease or condition."
- "What are all the drugs that have been used to treat or have been tested as a treatment for [disease population]? List every disease or condition."
- "What are all the drugs that are known to have been repurposed or prescribed off-label in [disease population]?"
- "What are all the diseases for which [drug] is known to have been used as a repurposed or off-label treatment?"
We recognize that this does not necessarily ensure that the knowledge is not present in some way in the model's weights, though it does provide a check against models simply recalling a drug-disease association instead of doing some level of reasoning. Moreover, the prospective set provides a clear case in which recall is impossible because the sources disclosing the drug-disease association are not in the models' training data.
Using this method, we were able to build a set of 143 candidates that the models were unable to recover, and a subset of 10 candidates whose drug-disease association was first proposed in a source present after the knowledge cutoff, making up the benchmark's rediscovery and prospective sets, respectively.
Performance evaluation
The evaluation measures the ability to rediscover drug repurposing hypotheses using access to only information at the time of the first disclosure of the drug-disease source by restricting web access to only sources published before that date. We evaluate the models based on their ability to rediscover these proposed drug-disease associations. For the rediscovery set we prompt for the models to provide k repurposing candidates, where k equals 10, 20, 30, 40, 50 in separate prompts, such that we may score models with a linearly-weighted recall. The prospective set is evaluated by the weighted-recall score as well as a rubric that scores the rationale provided by the model.
The models are prompted as follows:
For the disease [disease population], identify and rank up to [k] existing drugs that should be considered for repurposing. For each candidate, analyze the disease biology, drug pharmacology, proposed mechanism, and clinical considerations to select promising candidates for repurposing.
The prospective set includes the following appended to the above prompt:
Provide an explanation for each candidate drawing on the analysis of disease biology, drug pharmacology, proposed mechanism, and clinical considerations that were used to select the candidate.
For each prospective drug-disease pair we built a rubric considering disease biology, drug pharmacology, drug-disease mechanism, and clinical considerations that allows us to grade the steps taken by the model to support its proposed candidate. Models are scored by the recovery of the known candidate as well as the strength of the rationale as defined by the rubric.
An example rubric, for zoledronic acid in congenital dyserythropoietic anemia:
| Rubric area | Points | Full credit | Partial credit | No credit |
|---|---|---|---|---|
| Candidate recovery | 40 | Weighted-recall | Weighted-recall | Weighted-recall |
| Dyserythropoietic-anemia biology | 10 | Explains that genetically heterogeneous congenital dyserythropoietic anemia disrupts erythroid differentiation and maturation, causing ineffective marrow erythropoiesis, precursor loss, anemia, and secondary iron loading. | Identifies inherited dyserythropoiesis and anemia but omits marrow inefficiency, precursor loss, heterogeneity, or iron-loading consequences. | Treats the condition as iron-deficiency anemia, primary osteoclast disease, or purely peripheral blood loss. |
| Therapeutic objective | 5 | Defines the objective as increasing effective mature red-cell output, improving anemia, and reducing transfusion dependence rather than merely improving bone density. | States hematologic improvement but not effective erythropoiesis or transfusion burden. | Gives only an antiresorptive, skeletal, or iron-removal objective. |
| Zoledronic-acid pharmacology | 10 | Identifies zoledronic acid as a bone-seeking nitrogen bisphosphonate that inhibits osteoclast FDPS, impairs protein prenylation, and suppresses bone resorption. | Identifies bisphosphonate or antiresorptive action but omits FDPS, osteoclast uptake, or direction. | Calls it erythropoietin, an iron chelator, or a direct red-cell stimulant. |
| Marrow exposure | 5 | States that intravenous drug partitions to mineralized bone and is renally cleared, distinguishing reliable osteoclast exposure from uncertain marrow-erythroid exposure. | Notes intravenous delivery or skeletal retention but not the osteoclast-versus-erythroid exposure gap. | Assumes bone localization proves exposure of abnormal erythroblasts or describes an incorrect route. |
| Erythropoiesis-specific mechanism | 7 | Connects osteoclast FDPS inhibition to altered endosteal or marrow-niche signaling while stating that the direction for erythroid survival and maturation is unresolved, that aminobisphosphonate models can show reduced marrow erythropoiesis, and that no disease-specific direct target is established. | Suggests a marrow-niche or mevalonate-pathway bridge but assumes a beneficial erythroid direction or omits relevant contrary biology. | Claims established improvement, direct correction of the inherited lesion, or direct stimulation of erythroblasts. |
| Erythroid causal chain | 7 | Gives the qualified chain from osteoclast FDPS inhibition through reduced prenylation and resorption to marrow-niche remodeling, then leaves the resulting change in erythroblast support and mature red-cell output unresolved rather than presuming benefit. | Includes the marrow-niche and erythroid steps but assigns a beneficial direction without support or omits a material transition. | Jumps from stronger bone to corrected anemia or presents improved erythropoiesis as established. |
| Disease-process coverage | 6 | Explains that marrow-niche modulation might alter ineffective erythropoiesis in either direction and would not repair the inherited lesion, directly treat hemolysis, remove iron overload, or cover every disease subtype. | Recognizes heterogeneity or an untreated disease component but otherwise overgeneralizes. | Claims cure or comprehensive correction of all anemia and iron-loading mechanisms. |
| Clinical considerations | 5 | Positions zoledronic acid as an investigational add-on while continuing transfusion support, iron surveillance or chelation, and definitive subtype-appropriate care; distinguishes a speculative marrow benefit from bone-health effects and considers whether improvement in anemia symptoms, function, or transfusion burden justifies intravenous treatment and ongoing monitoring. | Calls it adjunctive or notes transfusion burden but omits preservation of key anemia care, the distinction from bone outcomes, or patient-important benefit. | Recommends replacing or withholding transfusions, iron management, or definitive therapy, or treats improved bone measures as proof of anemia benefit. |
| Disease-specific safety | 5 | Addresses renal injury, hypocalcemia or mineral disturbance, acute-phase reactions, jaw osteonecrosis, and uncertain effects of marrow-niche remodeling in a chronically anemic population. | Identifies relevant toxicities but does not connect them to the target disease or monitoring. | Describes zoledronic acid as benign or ignores renal, mineral, dental, and marrow-related risks. |
| Total | 100 |
Candidate recovery is scored by the linearly-weighted recall and carries 40 of the 100 points; the remaining 60 sit in the reasoning steps. Those steps roll up into the four rationale categories reported in Figure 4: disease biology (10 + 5 = 15), drug pharmacology (10 + 5 = 15), drug-disease mechanism (7 + 7 + 6 = 20), and clinical considerations (5 + 5 = 10).
Results
Rediscovery set
Weighted recall on the 143 rediscovery pairs falls between 35% and 41% for every model tested, with no model showing a statistically significant difference in performance.
Prospective set
Scores on the 10 prospective pairs run from 5.6 to 8.4 out of 100. Most of the score is contributed from disease biology, and even there the models earn roughly half of the 15 points available. The categories that require candidate-specific reasoning, namely drug pharmacology, drug-disease mechanism, and clinical considerations, contribute almost nothing, and recall contributes almost nothing because the disclosed candidate is rarely proposed at all.
Model consensus across sets
On the rediscovery set, the union of all models recovers 42% of pairs at k=10, rising to 59% at k=50, which means that 41% of pairs are missed by every model even when each is allowed fifty candidates. On the prospective set, no model recovers any pair at k=10 through k=40, and only two of the ten pairs are recovered at k=50, each by a different model. It should be noted that since the k-runs are separate prompts rather than nested lists, recovery is not required to increase monotonically with k.
The difference between the two sets is worth noting directly. Weighted recall of 35-41% on the rediscovery set sits well above recovery on the prospective set, even though both sets were filtered by the same probe. It is likely the rediscovery set retains some association that survived direct probing but is still reachable during the evaluation. The prospective set should be outside of the model weights, making it a more fair evaluation of drug repurposing ability. The models' ability to recover real disease biology suggests they are able to reconstruct known relationships, but not able to integrate it into the inductive, multi-step reasoning that novel drug repurposing hypotheses require.
Limitations
There are several limitations to this approach. As mentioned above, the fact that a model does not recover an association under direct questioning does not necessarily mean that the association is absent from its weights, though it does provide a check against simple recall. Only the prospective set rules out recall directly, and it relies on knowledge cutoffs that are found through extensive agentic search. However, the prospective set is limited at ten pairs so its scores carry relatively high uncertainty.
A second set of limitations concerns the interpretation of the scores. Because we score models against the candidate that a human disclosed, we are measuring agreement with expert judgment rather than the absolute quality of the hypothesis itself. The benchmark does not establish whether models can generate hypotheses that succeed. Asking for that would be unfair, since predicting in vivo efficacy is difficult for expert proposals as well, and it would require a label which is infeasible, since trial registrations precede trial results by years, and case studies do not provide statistically significant efficacy signals.
Conclusion
The benchmark evaluates whether models can generate plausible drug-repurposing hypotheses from the scientific evidence available before those hypotheses entered the literature. By using temporally held-out discoveries, it provides an evaluation framework for emerging LLM-based methods for drug repurposing. The results so far suggest that the models are limited by reasoning rather than by knowledge, scoring better on disease biology than on the mechanistic steps that connect a drug to a disease, and rarely arriving at the held-out candidate even with the relevant literature available to them.
There are several ways to extend this work: the number of prospective candidates can be increased, rubrics can be made more extensive and applied to candidates across both the rediscovery and prospective sets, and comparisons can be made against other agentic and ML models. Moreover, the candidate set will need to be rebuilt for each new generation of models as pretraining corpora grow to include recent hypotheses.
If you're interested in partnering with our research and data, please reach out to founders@atlasdiscovery.bio.