Coverage Degradation under Violated Exchangeability: Calibration and Conformal Prediction for Drug-Drug Interaction Prediction on Structurally Novel Drugs

Introduction

Adverse drug-drug interactions (DDIs) are a common and largely preventable cause of harm, particularly in patients who take several medicines at once. Because the number of possible drug pairs grows quadratically with the number of drugs, only a small fraction can ever be studied clinically. Computational models that predict whether, and how, two drugs interact from their chemical structure are therefore an active research area. Early work framed the task as classification over interaction types derived from DrugBank descriptions [1], and later models learned drug representations from interaction networks [2] or directly from molecular graphs [3], [4].

For a newly approved or investigational compound, no interactions have been recorded, so the practically important setting is the cold start. Recent graph models are evaluated in exactly this way: whole drugs, rather than drug pairs, are held out, and performance is reported separately for test pairs that contain one unseen drug (S1) and pairs in which both drugs are unseen (S2) [3][5]. These studies consistently show that accuracy drops sharply in S1 and further in S2. They report accuracy, however, and say little about whether the models know when they are likely to be wrong.

If such predictions are used to prioritize candidate interactions for expert review, knowing when they are unreliable matters as much as their accuracy. Two families of tools address it. Calibration asks whether a model’s stated confidence matches its observed accuracy [6]. Conformal prediction wraps any trained classifier so that it returns a set of plausible labels that contains the true label with a user-chosen probability, for example 90% [7], [8]. This guarantee is distribution-free, but it rests on one assumption: the calibration data and the test data must be exchangeable [7], [8]. A drug that never appeared in training violates that assumption by construction, so the guarantee no longer applies, and theory gives no bound on the resulting loss without further information about the shift [9]. How much empirical coverage degrades in DDI prediction, and whether the degradation depends on how structurally unusual the new drug is, has not been measured. Main contributions of the paper are:

• ColdCal is a benchmark for uncertainty quantification in multi-class DDI type prediction that grades novelty along two axes: structural distance of the held-out drugs (random, Butina-0.6 and Butina-0.7 splits) and pair novelty (S1 and S2), with an in-distribution calibration set and an in-distribution test set as references;

• A systematic evaluation of three models of increasing specificity to the cold-start problem, a fingerprint-based multilayer perceptron, SSI-DDI [3] and DSN-DDI [4], combined with temperature scaling, deep ensembles and three split-conformal procedures over nine splits;

• Evidence empirical coverage, which matches its 90% target in distribution, degrades to 0.13-0.66 on unseen drugs, in proportion to structural novelty, and that standard remedies do not repair it; and

• An analysis linking coverage degradation to the new drug’s similarity to the training set, reported separately for S1 and S2 pairs, together with released splits, audit tables and code.

  The rest of the paper is organized as follows. Section 2 reviews related work. Section 3 describes the data, the splitting protocol, the models and the uncertainty methods. Section 4 gives the experimental setup, Section 5 presents and discusses the results, and Section 6 concludes.

Complete Article

The complete article, including all figures, tables, equations and algorithms, is available in the official publication PDF.

View Full PDF

Conclusion

We introduced ColdCal, a benchmark for calibration and conformal coverage in DDI type prediction under graded drug novelty. For a fingerprint MLP, SSI-DDI and DSN-DDI, uncertainty estimates that were well calibrated on known drugs degraded substantially on new ones, where exchangeability is violated: empirical conformal coverage at a 90% target fell to 0.43-0.66 for pairs with one new drug and to 0.13-0.44 for pairs with two, and the degradation grew with structural novelty. Temperature scaling, deep ensembles and class-conditional calibration did not repair it, and adaptive sets did so only on S1 and only at a substantial cost in set size. Because coverage tracks similarity to the training drugs within both pair types, our next step is to test similarity-stratified and weighted conformal calibration on ColdCal, and to extend the benchmark to further datasets and longer-trained graph models. ColdCal is a benchmarking study of DrugBank interaction-type prediction and of uncertainty behavior on that task; it is not a validated clinical decision system, and its results should not be used to guide prescribing decisions.

Data Availability

The ColdCal split files are openly available on Kaggle at https://www.kaggle.com/datasets/bajpayeesingha/coldcal-splits. The evaluation code and model training scripts are available in the Kaggle notebooks https://www.kaggle.com/code/bajpayeesingha/coldcal-03-ssiddi, https://www.kaggle.com/code/bajpayeesingha/coldcal-04-dsn, and https://www.kaggle.com/code/bajpayeesingha/coldcal-05-paper, whose outputs contain the saved model outputs and result tables reported in this paper. The SSI-DDI model outputs are available at https://www.kaggle.com/datasets/bajpayeesingha/coldcal-results-ssiddi. The DrugBank-derived drug–drug interaction dataset is the version distributed with the official SSI-DDI implementation https://github.com/kanz76/SSI-DDI, which is subject to DrugBank's licensing terms. SSI-DDI and DSN-DDI were run from their official public repositories, https://github.com/kanz76/SSI-DDI and https://github.com/microsoft/Drug-Interaction-Research/tree/DSN-DDI-for-DDI-Prediction, respectively.

References

1. J. Y. Ryu, H. U. Kim, and S. Y. Lee, “Deep learning improves prediction of drug-drug and drug–food interactions, ” Proceedings of the National Academy of Sciences, vol. 115, no. 18, pp. E4304–E4311, 2018, doi: 10.1073/pnas.1803294115
2. M. Zitnik, M. Agrawal, and J. Leskovec, “Modeling polypharmacy side effects with graph convolutional networks, ” Bioinformatics, vol. 34, no. 13, pp. i457–i466, 2018, doi: 10.1093/bioinformatics/bty294
3. A. K. Nyamabo, H. Yu, and J.-Y. Shi, “SSI-DDI: Substructure-substructure interactions for drug–drug interaction prediction, ” Briefings in Bioinformatics, vol. 22, no. 6, p. bbab133, 2021, doi: 10.1093/bib/bbab133
4. Z. Li, S. Zhu, B. Shao, X. Zeng, T. Wang, and T.-Y. Liu, “DSN-DDI: An accurate and generalized framework for drug-drug interaction prediction by dual-view representation learning, ” Briefings in Bioinformatics, vol. 24, no. 1, p. bbac597, 2023, doi: 10.1093/bib/bbac597
5. P. Dewulf, M. Stock, and B. De Baets, “Cold-start problems in data-driven prediction of drug-drug interaction effects, ” Pharmaceuticals, vol. 14, no. 5, p. 429, 2021, doi: 10.3390/ph14050429
6. C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks, ” in Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, 2017, pp. 1321–1330.
7. V. Vovk, A. Gammerman, and G. Shafer, Algorithmic Learning in a Random World. New York, NY, USA: Springer, 2005.
8. A. N. Angelopoulos and S. Bates, “Conformal prediction: A gentle introduction, ” Foundations and Trends in Machine Learning, vol. 16, no. 4, pp. 494–591, 2023, doi: 10.1561/2200000101
9. R. F. Barber, E. J. Candès, A. Ramdas, and R. J. Tibshirani, “Conformal prediction beyond exchangeability, ” The Annals of Statistics, vol. 51, no. 2, pp. 816–845, 2023, doi: 10.1214/23-AOS2276
10. P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks, ” in Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada, 2018.
11. M. Nickel, V. Tresp, and H.-P. Kriegel, “A three-way model for collective learning on multi-relational data, ” in Proceedings of the 28th International Conference on Machine Learning, Bellevue, WA, USA, 2011, pp. 809–816.
12. Z. Wu, B. Ramsundar, E. N. Feinberg, J. Gomes, C. Geniesse, A. S. Pappu, K. Leswing, and V. Pande, “MoleculeNet: A benchmark for molecular machine learning, ” Chemical Science, vol. 9, no. 2, pp. 513–530, 2018, doi: 10.1039/C7SC02664A
13. D. Butina, “Unsupervised data base clustering based on Daylight’s fingerprint and Tanimoto similarity: A fast and automated way to cluster small and large data sets, ” Journal of Chemical Information and Computer Sciences, vol. 39, no. 4, pp. 747–750, 1999, doi: 10.1021/ci9803381
14. D. Rogers and M. Hahn, “Extended-connectivity fingerprints, ” Journal of Chemical Information and Modeling, vol. 50, no. 5, pp. 742–754, 2010, doi: 10.1021/ci100050t
15. B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles, ” in Advances in Neural Information Processing Systems, vol. 30, Long Beach, CA, USA, 2017.
16. Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V. Dillon, B. Lakshminarayanan, and J. Snoek, “Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift, ” in Advances in Neural Information Processing Systems, vol. 32, Vancouver, Canada, 2019.
17. M. Sadinle, J. Lei, and L. Wasserman, “Least ambiguous set-valued classifiers with bounded error levels, ” Journal of the American Statistical Association, vol. 114, no. 525, pp. 223–234, 2019, doi: 10.1080/01621459.2017.1395341
18. Y. Romano, M. Sesia, and E. J. Candès, “Classification with valid and adaptive coverage, ” in Advances in Neural Information Processing Systems, vol. 33, Virtual Conference, 2020.
19. T. Ding, A. N. Angelopoulos, S. Bates, M. I. Jordan, and R. J. Tibshirani, “Class-conditional conformal prediction with many classes, ” in Advances in Neural Information Processing Systems, vol. 36, New Orleans, LA, USA, 2023.
20. R. J. Tibshirani, R. F. Barber, E. J. Candès, and A. Ramdas, “Conformal prediction under covariate shift, ” in Advances in Neural Information Processing Systems, vol. 32, Vancouver, Canada, 2019.
21. D. S. Wishart et al., “DrugBank 5.0: A major update to the DrugBank database for 2018, ” Nucleic Acids Research, vol. 46, no. D1, pp. D1074–D1082, 2018, doi: 10.1093/nar/gkx1037
22. RDKit Contributors, “RDKit: Open-source cheminformatics software, ” RDKit. Accessed: October 24, 2026. [Online]. Available: https://www.rdkit.org
23. A. K. Nyamabo, “SSI-DDI: Official implementation, ” GitHub repository. Accessed: October 24, 2026. [Online]. Available: https://github.com/kanz76/SSI-DDI
24. Microsoft, “DSN-DDI: Official implementation (branch DSN-DDI-for-DDI-Prediction), ” GitHub repository. Accessed: October 24, 2026. [Online]. Available: https://github.com/microsoft/Drug-Interaction-Research/tree/DSN-DDI-for-DDI-Prediction
25. A. Paszke et al., “PyTorch: An imperative style, high-performance deep learning library, ” in Advances in Neural Information Processing Systems, vol. 32, Vancouver, Canada, 2019.
26. M. Fey and J. E. Lenssen, “Fast graph representation learning with PyTorch Geometric, ” in ICLR Workshop on Representation Learning on Graphs and Manifolds, New Orleans, LA, USA, 2019.
27. M. P. Naeini, G. F. Cooper, and M. Hauskrecht, “Obtaining well calibrated probabilities using Bayesian binning, ” in Proceedings of the 29th AAAI Conference on Artificial Intelligence, Austin, TX, USA, 2015, pp. 2901–2907.
28. Y. Geifman and R. El-Yaniv, “Selective classification for deep neural networks, ” in Advances in Neural Information Processing Systems, vol. 30, Long Beach, CA, USA, 2017.