Research Articles Open Access

Coverage Degradation under Violated Exchangeability: Calibration and Conformal Prediction for Drug-Drug Interaction Prediction on Structurally Novel Drugs

Bajpayee Singha , Ajay Kumar Sinha *
* Corresponding author: Ajay Kumar Sinha ( ajaysinha@stumail.ysu.edu.cn )
Available online November 9, 2026
PDF

Abstract

Deep learning models for drug-drug interaction (DDI) prediction are increasingly evaluated on drugs absent from training, yet the reliability of their uncertainty estimates in this cold-start setting has not been examined. We present ColdCal, a benchmark that measures calibration and conformal coverage for multi-class DDI type prediction as drug novelty increases. Using DrugBank data covering 1,704 drugs, 191,870 interacting pairs and 86 interaction types, we held out about 20% of drugs under three regimes of increasing structural novelty (random selection and Butina clustering at Tanimoto cutoffs of 0.6 and 0.7) and evaluated pairs containing one or two unseen drugs. A fingerprint-based multilayer perceptron and two graph neural networks, SSI-DDI and DSN-DDI, were combined with temperature scaling, deep ensembles, and three split-conformal procedures over nine splits. In distribution, all models were well calibrated, and conformal sets met the 90% target. For unseen drugs, where exchangeability is violated and the guarantee no longer applies, empirical coverage of least-ambiguous conformal sets fell to 0.43-0.66 when one drug was new and to 0.13-0.44 when both were new, decreasing steadily with structural novelty. Temperature scaling, ensembling and class-conditional calibration recovered little, and adaptive prediction sets restored coverage only by enlarging sets to 34-68 of 86 types for the perceptron. Coverage rose with the new drug's similarity to training drugs for both pair types but stayed below target in every similarity bin. In this benchmark, uncertainty calibrated on known drugs did not transfer to new drugs, which motivates shift-aware recalibration.

Keywords

Drug-drug interaction uncertainty quantification conformal prediction model calibration cold-start prediction

Introduction

Adverse drug-drug interactions (DDIs) are a common and largely preventable cause of harm, particularly in patients who take several medicines at once. Because the number of possible drug pairs grows quadratically with the number of drugs, only a small fraction can ever be studied clinically. Computational models that predict whether, and how, two drugs interact from their chemical structure are therefore an active research area. Early work framed the task as classification over interaction types derived from DrugBank descriptions [1], and later models learned drug representations from interaction networks [2] or directly from molecular graphs [3], [4].

For a newly approved or investigational compound, no interactions have been recorded, so the practically important setting is the cold start. Recent graph models are evaluated in exactly this way: whole drugs, rather than drug pairs, are held out, and performance is reported separately for test pairs that contain one unseen drug (S1) and pairs in which both drugs are unseen (S2) [3][5]. These studies consistently show that accuracy drops sharply in S1 and further in S2. They report accuracy, however, and say little about whether the models know when they are likely to be wrong.

If such predictions are used to prioritize candidate interactions for expert review, knowing when they are unreliable matters as much as their accuracy. Two families of tools address it. Calibration asks whether a model’s stated confidence matches its observed accuracy [6]. Conformal prediction wraps any trained classifier so that it returns a set of plausible labels that contains the true label with a user-chosen probability, for example 90% [7], [8]. This guarantee is distribution-free, but it rests on one assumption: the calibration data and the test data must be exchangeable [7], [8]. A drug that never appeared in training violates that assumption by construction, so the guarantee no longer applies, and theory gives no bound on the resulting loss without further information about the shift [9]. How much empirical coverage degrades in DDI prediction, and whether the degradation depends on how structurally unusual the new drug is, has not been measured. Main contributions of the paper are:

  • ColdCal is a benchmark for uncertainty quantification in multi-class DDI type prediction that grades novelty along two axes: structural distance of the held-out drugs (random, Butina-0.6 and Butina-0.7 splits) and pair novelty (S1 and S2), with an in-distribution calibration set and an in-distribution test set as references;

  • A systematic evaluation of three models of increasing specificity to the cold-start problem, a fingerprint-based multilayer perceptron, SSI-DDI [3] and DSN-DDI [4], combined with temperature scaling, deep ensembles and three split-conformal procedures over nine splits;

  • Evidence empirical coverage, which matches its 90% target in distribution, degrades to 0.13-0.66 on unseen drugs, in proportion to structural novelty, and that standard remedies do not repair it; and

  • An analysis linking coverage degradation to the new drug’s similarity to the training set, reported separately for S1 and S2 pairs, together with released splits, audit tables and code.

    The rest of the paper is organized as follows. Section 2 reviews related work. Section 3 describes the data, the splitting protocol, the models and the uncertainty methods. Section 4 gives the experimental setup, Section 5 presents and discusses the results, and Section 6 concludes.

Complete Article

The complete article, including all figures, tables, equations and algorithms, is available in the official publication PDF.

Conclusion

We introduced ColdCal, a benchmark for calibration and conformal coverage in DDI type prediction under graded drug novelty. For a fingerprint MLP, SSI-DDI and DSN-DDI, uncertainty estimates that were well calibrated on known drugs degraded substantially on new ones, where exchangeability is violated: empirical conformal coverage at a 90% target fell to 0.43-0.66 for pairs with one new drug and to 0.13-0.44 for pairs with two, and the degradation grew with structural novelty. Temperature scaling, deep ensembles and class-conditional calibration did not repair it, and adaptive sets did so only on S1 and only at a substantial cost in set size. Because coverage tracks similarity to the training drugs within both pair types, our next step is to test similarity-stratified and weighted conformal calibration on ColdCal, and to extend the benchmark to further datasets and longer-trained graph models. ColdCal is a benchmarking study of DrugBank interaction-type prediction and of uncertainty behavior on that task; it is not a validated clinical decision system, and its results should not be used to guide prescribing decisions.

Data Availability

The ColdCal split files are openly available on Kaggle at https://www.kaggle.com/datasets/bajpayeesingha/coldcal-splits. The evaluation code and model training scripts are available in the Kaggle notebooks https://www.kaggle.com/code/bajpayeesingha/coldcal-03-ssiddi, https://www.kaggle.com/code/bajpayeesingha/coldcal-04-dsn, and https://www.kaggle.com/code/bajpayeesingha/coldcal-05-paper, whose outputs contain the saved model outputs and result tables reported in this paper. The SSI-DDI model outputs are available at https://www.kaggle.com/datasets/bajpayeesingha/coldcal-results-ssiddi. The DrugBank-derived drug–drug interaction dataset is the version distributed with the official SSI-DDI implementation https://github.com/kanz76/SSI-DDI, which is subject to DrugBank's licensing terms. SSI-DDI and DSN-DDI were run from their official public repositories, https://github.com/kanz76/SSI-DDI and https://github.com/microsoft/Drug-Interaction-Research/tree/DSN-DDI-for-DDI-Prediction, respectively.

References

  1. J. Y. Ryu, H. U. Kim, and S. Y. Lee, “Deep learning improves prediction of drug-drug and drug–food interactions, ” Proceedings of the National Academy of Sciences, vol. 115, no. 18, pp. E4304–E4311, 2018, doi: 10.1073/pnas.1803294115
  2. M. Zitnik, M. Agrawal, and J. Leskovec, “Modeling polypharmacy side effects with graph convolutional networks, ” Bioinformatics, vol. 34, no. 13, pp. i457–i466, 2018, doi: 10.1093/bioinformatics/bty294
  3. A. K. Nyamabo, H. Yu, and J.-Y. Shi, “SSI-DDI: Substructure-substructure interactions for drug–drug interaction prediction, ” Briefings in Bioinformatics, vol. 22, no. 6, p. bbab133, 2021, doi: 10.1093/bib/bbab133
  4. Z. Li, S. Zhu, B. Shao, X. Zeng, T. Wang, and T.-Y. Liu, “DSN-DDI: An accurate and generalized framework for drug-drug interaction prediction by dual-view representation learning, ” Briefings in Bioinformatics, vol. 24, no. 1, p. bbac597, 2023, doi: 10.1093/bib/bbac597
  5. P. Dewulf, M. Stock, and B. De Baets, “Cold-start problems in data-driven prediction of drug-drug interaction effects, ” Pharmaceuticals, vol. 14, no. 5, p. 429, 2021, doi: 10.3390/ph14050429
  6. C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks, ” in Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, 2017, pp. 1321–1330.
  7. V. Vovk, A. Gammerman, and G. Shafer, Algorithmic Learning in a Random World. New York, NY, USA: Springer, 2005.
  8. A. N. Angelopoulos and S. Bates, “Conformal prediction: A gentle introduction, ” Foundations and Trends in Machine Learning, vol. 16, no. 4, pp. 494–591, 2023, doi: 10.1561/2200000101
  9. R. F. Barber, E. J. Candès, A. Ramdas, and R. J. Tibshirani, “Conformal prediction beyond exchangeability, ” The Annals of Statistics, vol. 51, no. 2, pp. 816–845, 2023, doi: 10.1214/23-AOS2276
  10. P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks, ” in Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada, 2018.
  11. M. Nickel, V. Tresp, and H.-P. Kriegel, “A three-way model for collective learning on multi-relational data, ” in Proceedings of the 28th International Conference on Machine Learning, Bellevue, WA, USA, 2011, pp. 809–816.
  12. Z. Wu, B. Ramsundar, E. N. Feinberg, J. Gomes, C. Geniesse, A. S. Pappu, K. Leswing, and V. Pande, “MoleculeNet: A benchmark for molecular machine learning, ” Chemical Science, vol. 9, no. 2, pp. 513–530, 2018, doi: 10.1039/C7SC02664A
  13. D. Butina, “Unsupervised data base clustering based on Daylight’s fingerprint and Tanimoto similarity: A fast and automated way to cluster small and large data sets, ” Journal of Chemical Information and Computer Sciences, vol. 39, no. 4, pp. 747–750, 1999, doi: 10.1021/ci9803381
  14. D. Rogers and M. Hahn, “Extended-connectivity fingerprints, ” Journal of Chemical Information and Modeling, vol. 50, no. 5, pp. 742–754, 2010, doi: 10.1021/ci100050t
  15. B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles, ” in Advances in Neural Information Processing Systems, vol. 30, Long Beach, CA, USA, 2017.
  16. Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V. Dillon, B. Lakshminarayanan, and J. Snoek, “Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift, ” in Advances in Neural Information Processing Systems, vol. 32, Vancouver, Canada, 2019.
  17. M. Sadinle, J. Lei, and L. Wasserman, “Least ambiguous set-valued classifiers with bounded error levels, ” Journal of the American Statistical Association, vol. 114, no. 525, pp. 223–234, 2019, doi: 10.1080/01621459.2017.1395341
  18. Y. Romano, M. Sesia, and E. J. Candès, “Classification with valid and adaptive coverage, ” in Advances in Neural Information Processing Systems, vol. 33, Virtual Conference, 2020.
  19. T. Ding, A. N. Angelopoulos, S. Bates, M. I. Jordan, and R. J. Tibshirani, “Class-conditional conformal prediction with many classes, ” in Advances in Neural Information Processing Systems, vol. 36, New Orleans, LA, USA, 2023.
  20. R. J. Tibshirani, R. F. Barber, E. J. Candès, and A. Ramdas, “Conformal prediction under covariate shift, ” in Advances in Neural Information Processing Systems, vol. 32, Vancouver, Canada, 2019.
  21. D. S. Wishart et al., “DrugBank 5.0: A major update to the DrugBank database for 2018, ” Nucleic Acids Research, vol. 46, no. D1, pp. D1074–D1082, 2018, doi: 10.1093/nar/gkx1037
  22. RDKit Contributors, “RDKit: Open-source cheminformatics software, ” RDKit. Accessed: October 24, 2026. [Online]. Available: https://www.rdkit.org
  23. A. K. Nyamabo, “SSI-DDI: Official implementation, ” GitHub repository. Accessed: October 24, 2026. [Online]. Available: https://github.com/kanz76/SSI-DDI
  24. Microsoft, “DSN-DDI: Official implementation (branch DSN-DDI-for-DDI-Prediction), ” GitHub repository. Accessed: October 24, 2026. [Online]. Available: https://github.com/microsoft/Drug-Interaction-Research/tree/DSN-DDI-for-DDI-Prediction
  25. A. Paszke et al., “PyTorch: An imperative style, high-performance deep learning library, ” in Advances in Neural Information Processing Systems, vol. 32, Vancouver, Canada, 2019.
  26. M. Fey and J. E. Lenssen, “Fast graph representation learning with PyTorch Geometric, ” in ICLR Workshop on Representation Learning on Graphs and Manifolds, New Orleans, LA, USA, 2019.
  27. M. P. Naeini, G. F. Cooper, and M. Hauskrecht, “Obtaining well calibrated probabilities using Bayesian binning, ” in Proceedings of the 29th AAAI Conference on Artificial Intelligence, Austin, TX, USA, 2015, pp. 2901–2907.
  28. Y. Geifman and R. El-Yaniv, “Selective classification for deep neural networks, ” in Advances in Neural Information Processing Systems, vol. 30, Long Beach, CA, USA, 2017.
11 1

Similar Articles

11-18 of 18

You may also start an advanced similarity search for this article.