Data Pitfalls in Explainable ML for UPJ Obstruction in Horseshoe Kidney

* Corresponding author: Harshwardhansinh Chauhan ( hwkc392@gmail.com )
Available online November 19, 2026
PDF

Abstract

Assessment of ureteropelvic junction obstruction (UPJO) in horseshoe kidney (HSK) is difficult, and curated data for this congenital anomaly are scarce, which makes synthetic data attractive for building machine-learning (ML) processes. How much can be concluded from such data is rarely evaluated. We report a reproducible methodological case study on a synthetic cohort of 1,000 HSK records (no real patients) and ask how far headline performance and explanation plots can be trusted. Post-treatment fields were excluded to prevent target leakage. On a balanced 80:20 holdout (N = 200), XGBoost reached ROC-AUC 0.997 and accuracy 0.955, and Logistic Regression, SVM, and Random Forest performed comparably (ROC-AUC 0.994–0.996). In repeated cross-validation, all four models reached ROC-AUC 0.990–0.993, while logistic regression reached 0.983. When a random fraction of labels was flipped (10 independent draws), observed ROC-AUC fell to 0.93 at 5%, 0.87-0.88 at 10%, and 0.76–0.77 at 20%, whereas Gaussian measurement noise of 1.0 × SD on APD_mm and T₁/₂ reduced it only to 0.95-0.96. As separate process checks that are not evidence about HSK or UPJO, the same workflow reached ROC-AUC ≥ 0.999 on the easy public UCI Chronic Kidney Disease dataset (n = 400) but only 0.81-0.85 on a harder public urine-analysis dataset (n = 79). Near-perfect metrics on the synthetic cohort therefore reflect rule-based labels, not diagnostic certainty. We give reporting and stress-testing recommendations for ML studies that use synthetic clinical data. This is a methodological study, not a clinically validated diagnostic tool.

Keywords

Diuretic renography explainable artificial intelligence (XAI) extreme gradient boosting (XGBoost) horseshoe kidney label noise

Introduction

Horseshoe kidney (HSK) is the most common renal fusion anomaly, affecting about 1 in 400 to 1 in 800 people [1], [2]. Failure of axial rotation and arrested cephalad migration produce a fibrous or parenchymal isthmus joining the lower poles across the midline [1]. The renal pelves therefore lie anteriorly, the ureters cross the isthmus, ureteral insertions are often high, and anomalous lower-pole arteries arising directly from the aorta or iliac vessels can compress the collecting system [2], [3]. These features predispose patients to urinary stasis, recurrent nephrolithiasis and ureteropelvic junction obstruction (UPJO) [2], [3].

Assessing UPJO in this anomalous anatomy is difficult. A typical work-up combines ultrasound, including the Society for Fetal Urology (SFU) hydronephrosis grade and the maximal anteroposterior pelvic diameter (APD) [4-5], with diuretic renography (e.g., ^99mTc-MAG3) to obtain differential renal function (DRF), drainage half-time (T_1/2) and the shape of the clearance curve are shown in Figure 1 and Figure 2. In HSK, however, delayed tracer washout is common and may reflect gravity-dependent pooling in anteriorly rotated calyces rather than true mechanical obstruction [2], [6]. Even in anatomically normal kidneys, the decision to operate is made from several imperfect surrogates rather than a single gold standard [4-5]. Objective decision support that separates benign stasis from progressive obstruction needing pyeloplasty would therefore be valuable [6].

Machine-learning (ML) models such as extreme gradient boosting (XGBoost) [7] perform well on many tabular medical prediction tasks [8], but clinical translation is limited by their opacity and by the scarcity of data for rare congenital anomalies [9]. Explainable-AI (XAI) methods such as SHapley Additive exPlanations (SHAP) are often used to address the first problem [9], [10], and synthetic data are increasingly used to address the second [11], [12]. Synthetic tabular data allow processes to be built safely, but they need caution: artifacts of the generating process can inflate accuracy while omitting the difficulties of noisy biological cohorts [11], [13]. When both performance and explanations are computed on the same synthetic data, it is easy to mistake a model that has recovered the generator’s rule for a model that has learned clinical physiology.

This study asks three methodological questions.

  • RQ1: How much of the near-ceiling performance on a synthetic HSK-UPJO cohort is explained by a simple, low-dimensional rule?

  • RQ2: What do SHAP explanations reveal on such data, and what can they not establish?

  • RQ3: How sensitive are the headline metrics to realistic sources of error, namely reference-standard (label) disagreement and measurement noise on the key imaging variables?

We train an explainable workflow (XGBoost, Random Forest, support vector machine (SVM) and Logistic Regression, with SHAP [9], [10]) on the synthetic HSK-UPJO dataset and then stress-test it. Our contributions are: (i) a leakage-aware feature set with a documented audit of flawed fields in the source file; (ii) simple and rule-based baselines with repeated cross-validation that quantify how much signal the two dominant variables carry; (iii) label-noise and measurement-noise simulations, each repeated over 10 independent draws, that quantify how quickly the metrics degrade; (iv) supplementary runs on two public real-world datasets, one easy (UCI Chronic Kidney Disease) and one hard (urine analysis), which show that the same process produces near-ceiling or modest metrics depending on the task, and which are not presented as evidence about HSK or UPJO; and (v) release of the code, a corrected dataset and a single script that regenerates every reported number and figure. No real HSK patient is analyzed, and no clinical validity is claimed.

The rest of the paper is organized as follows. Section 2 reviews related work and states the research gap. Section 3 describes the dataset, the leakage controls, the models and the validation and stress-testing protocol. Section 4 reports the results, with the synthetic-cohort experiments (Sections 4.1-4.5) kept separate from the supplementary public-dataset experiments (Section 4.6). Section 5 discusses the findings against previous studies and their practical implications, Section 6 lists the limitations and Section 7 concludes.

Complete Article

The complete article, including all figures, tables, equations and algorithms, is available in the official publication PDF.

Conclusion

This study presented a reproducible, explainable machine-learning workflow (XGBoost, Random Forest, SVM, Logistic Regression and TreeSHAP) for UPJO classification in a synthetic cohort of horseshoe kidneys, and used it to show why results from such data must not be read as clinical evidence. The models reached near-ceiling discrimination, but a two-variable logistic regression came within about 0.01 of them (RQ1); SHAP recovered threshold-like rules that mirror clinical conventions because the generator encoded them, not because the model learned physiology (RQ2), and observed performance fell sharply under label noise while remaining stable under measurement noise (RQ3). The same process was near-perfect on an easy public CKD dataset but reached only 0.81-0.85 on a harder public urine-analysis dataset, which confirms that such metrics depend on the task. The contribution of this work is methodological rather than clinical: a protocol of leakage audit, rule-recovering baselines, noise stress tests and real-data contrasts, together with reporting recommendations. Researchers who use synthetic clinical data should release the data and generator, report simple baselines and noise-sensitivity analyses and avoid clinical interpretation. Validation on multi-institutional HSK-UPJO registries remains essential before any translational claim.

References

  1. K. Taghavi, J. Kirkpatrick, and S. A. Mirjalili, “The horseshoe kidney: Surgical anatomy and embryology, ” J. Pediatr. Urol., vol. 12, no. 5, pp. 275–280, Oct. 2016. https://doi.org/doi: 10.1016/j.jpurol.2016.04.033
  2. M. I. A. Elmaadawy, S. W. Kim, S. K. Kang, S. W. Han, and Y. S. Lee, “A retrospective analysis of ureteropelvic junction obstructions in patients with horseshoe kidney, ” Transl. Androl. Urol., vol. 10, no. 11, pp. 4173–4180, Nov. 2021. https://doi.org/doi: 10.21037/tau-21-471
  3. A. W. Partin, R. R. Dmochowski, L. R. Kavoussi, C. A. Peters, and A. J. Wein, Eds., Campbell-Walsh-Wein Urology, 12th ed. Philadelphia, PA, USA: Elsevier, 2020.
  4. H. T. Nguyen et al., “The Society for Fetal Urology consensus statement on the evaluation and management of antenatal hydronephrosis, ” J. Pediatr. Urol., vol. 6, no. 3, pp. 212–231, Jun. 2010. https://doi.org/doi: 10.1016/j.jpurol.2010.02.205
  5. I. Gordon, A. Piepsz, and R. Sixt, “Guidelines for standard and diuretic renogram in children, ” Eur. J. Nucl. Med. Mol. Imaging, vol. 38, no. 6, pp. 1175–1188, Jun. 2011. https://doi.org/doi: 10.1007/s00259-011-1811-3
  6. J. H. Ross and R. Kay, “Ureteropelvic junction obstruction in anomalous kidneys, ” Urol. Clin. North Am., vol. 25, no. 2, pp. 219–225, May 1998. https://doi.org/doi: 10.1016/S0094-0143(05)70010-0
  7. T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system, ” in Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining (KDD ’16), San Francisco, CA, USA, 2016, pp. 785–794. https://doi.org/doi: 10.1145/2939672.2939785
  8. L. Grinsztajn, E. Oyallon, and G. Varoquaux, “Why do tree-based models still outperform deep learning on typical tabular data?, ” in Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 35, Datasets and Benchmarks Track, 2022, pp. 507–520. https://doi.org/doi: 10.52202/068431-0037
  9. S. M. Lundberg et al., “From local explanations to global understanding with explainable AI for trees, ” Nature Mach. Intell., vol. 2, no. 1, pp. 56–67, Jan. 2020. https://doi.org/doi: 10.1038/s42256-019-0138-9
  10. S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions, ” in Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 30, 2017, pp. 4765–4774.
  11. R. J. Chen, M. Y. Lu, T. Y. Chen, D. F. K. Williamson, and F. Mahmood, “Synthetic data in machine learning for medicine and healthcare, ” Nature Biomed. Eng., vol. 5, no. 6, pp. 493–497, Jun. 2021. https://doi.org/doi: 10.1038/s41551-021-00751-8
  12. A. Gonzales, G. Guruswamy, and S. R. Smith, “Synthetic data in health care: A narrative review, ” PLOS Digit. Health, vol. 2, no. 1, Art. no. e0000082, Jan. 2023. https://doi.org/doi: 10.1371/journal.pdig.0000082
  13. S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine-learning-based science, ” Patterns, vol. 4, no. 9, Art. no. 100804, Sep. 2023. https://doi.org/doi: 10.1016/j.patter.2023.100804
  14. E. S. Blum et al., “Early detection of ureteropelvic junction obstruction using signal analysis and machine learning: A dynamic solution to a dynamic problem, ” J. Urol., vol. 199, no. 3, pp. 847–852, Mar. 2018. https://doi.org/doi: 10.1016/j.juro.2017.09.147
  15. J. J. Cerrolaza et al., “Quantitative ultrasound for measuring obstructive severity in children with hydronephrosis, ” J. Urol., vol. 195, no. 4, pt. 1, pp. 1093–1099, Apr. 2016. https://doi.org/doi: 10.1016/j.juro.2015.10.173
  16. L. Erdman et al., “Predicting obstructive hydronephrosis based on ultrasound alone, ” in Medical Image Computing and Computer-Assisted Intervention-MICCAI 2020, Lecture Notes in Computer Science, vol. 12263. Cham, Switzerland: Springer, 2020, pp. 493–503. https://doi.org/doi: 10.1007/978-3-030-59716-0_47
  17. L. Erdman et al., “The Hydronephrosis Severity Index guides paediatric antenatal hydronephrosis management based on artificial intelligence applied to ultrasound images alone, ” Sci. Rep., vol. 14, no. 1, Art. no. 22748, Oct. 2024. https://doi.org/doi: 10.1038/s41598-024-72271-9
  18. A. Khondker et al., “Predicting obstruction risk using common ultrasonography parameters in paediatric hydronephrosis with machine learning, ” BJU Int., vol. 133, no. 1, pp. 79–86, Jan. 2024. https://doi.org/doi: 10.1111/bju.16159
  19. A. Khondker et al., “Longitudinal image-based prediction of surgical intervention in infants with hydronephrosis using deep learning: Is a single ultrasound enough?, ” PLOS Digit. Health, vol. 4, no. 8, Art. no. e0000939, Aug. 2025. https://doi.org/doi: 10.1371/journal.pdig.0000939
  20. E. Drysdale et al., “Personalized application of machine learning algorithms to identify pediatric patients at risk for recurrent ureteropelvic junction obstruction after dismembered pyeloplasty, ” World J. Urol., vol. 40, no. 2, pp. 593–599, Feb. 2022. https://doi.org/doi: 10.1007/s00345-021-03879-z
  21. C. A. Alici and B. Tokar, “Prediction of surgical necessity in children with ureteropelvic junction obstruction using machine learning, ” Ir. J. Med. Sci., vol. 194, no. 2, pp. 583–590, Apr. 2025. https://doi.org/doi: 10.1007/s11845-025-03895-7
  22. C. Wolmer, J. Delmas, S. Pecorelli, E. Dobremez, C. Ferdynus, and L. Harper, “Predicting clinically significant events in children with ureteropelvic junction obstruction, ” Front. Pediatr., vol. 12, Art. no. 1409170, 2024. https://doi.org/doi: 10.3389/fped.2024.1409170
  23. M. T. Ribeiro, S. Singh, and C. Guestrin, “‘Why should I trust you?’: Explaining the predictions of any classifier, ” in Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining (KDD ’16), San Francisco, CA, USA, 2016, pp. 1135–1144. https://doi.org/doi: 10.1145/2939672.2939778
  24. S. M. Lundberg et al., “Explainable machine-learning predictions for the prevention of hypoxaemia during surgery, ” Nature Biomed. Eng., vol. 2, no. 10, pp. 749–760, Oct. 2018. https://doi.org/doi: 10.1038/s41551-018-0304-0
  25. I. E. Kumar, S. Venkatasubramanian, C. Scheidegger, and S. A. Friedler, “Problems with Shapley-value-based explanations as feature importance measures, ” in Proc. 37th Int. Conf. Mach. Learn. (ICML), PMLR, vol. 119, 2020, pp. 5491–5500.
  26. D. Slack, S. Hilgard, E. Jia, S. Singh, and H. Lakkaraju, “Fooling LIME and SHAP: Adversarial attacks on post hoc explanation methods, ” in Proc. AAAI/ACM Conf. AI, Ethics, Soc. (AIES), New York, NY, USA, 2020, pp. 180–186. https://doi.org/doi: 10.1145/3375627.3375830
  27. M. Ghassemi, L. Oakden-Rayner, and A. L. Beam, “The false hope of current approaches to explainable artificial intelligence in health care, ” Lancet Digit. Health, vol. 3, no. 11, pp. e745–e750, Nov. 2021. https://doi.org/doi: 10.1016/S2589-7500(21)00208-9
  28. C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, ” Nature Mach. Intell., vol. 1, no. 5, pp. 206–215, May 2019. https://doi.org/doi: 10.1038/s42256-019-0048-x
  29. J. Walonoski et al., “Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record, ” J. Amer. Med. Inform. Assoc., vol. 25, no. 3, pp. 230–238, Mar. 2018. https://doi.org/doi: 10.1093/jamia/ocx079
  30. L. Xu, M. Skoularidou, A. Cuesta-Infante, and K. Veeramachaneni, “Modeling tabular data using conditional GAN, ” in Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 32, 2019, pp. 7335–7345.
  31. J. Yoon, L. N. Drumright, and M. van der Schaar, “Anonymization through data synthesis using generative adversarial networks (ADS-GAN), ” IEEE J. Biomed. Health Inform., vol. 24, no. 8, pp. 2378–2388, Aug. 2020. https://doi.org/doi: 10.1109/JBHI.2020.2980262
  32. M. Giuffrè and D. L. Shung, “Harnessing the power of synthetic data in healthcare: Innovation, application, and privacy, ” npj Digit. Med., vol. 6, no. 1, Art. no. 186, Oct. 2023. https://doi.org/doi: 10.1038/s41746-023-00927-3
  33. A. Alaa, B. van Breugel, E. S. Saveliev, and M. van der Schaar, “How faithful is your synthetic data? Sample-level metrics for evaluating and auditing generative models, ” in Proc. 39th Int. Conf. Mach. Learn. (ICML), PMLR, vol. 162, 2022, pp. 290–306.
  34. B. Frénay and M. Verleysen, “Classification in the presence of label noise: A survey, ” IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 5, pp. 845–869, May 2014. https://doi.org/doi: 10.1109/TNNLS.2013.2292894
  35. C. G. Northcutt, L. Jiang, and I. L. Chuang, “Confident learning: Estimating uncertainty in dataset labels, ” J. Artif. Intell. Res., vol. 70, pp. 1373–1411, 2021. https://doi.org/doi: 10.1613/jair.1.12125
  36. C. G. Northcutt, A. Athalye, and J. Mueller, “Pervasive label errors in test sets destabilize machine learning benchmarks, ” in Proc. Neural Information Processing Systems Track on Datasets and Benchmarks, vol. 1, 2021.
  37. D. Karimi, H. Dou, S. K. Warfield, and A. Gholipour, “Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis, ” Med. Image Anal., vol. 65, Art. no. 101759, Oct. 2020. https://doi.org/doi: 10.1016/j.media.2020.101759
  38. S. Kaufman, S. Rosset, C. Perlich, and O. Stitelman, “Leakage in data mining: Formulation, detection, and avoidance, ” ACM Trans. Knowl. Discov. Data, vol. 6, no. 4, Art. no. 15, Dec. 2012. https://doi.org/doi: 10.1145/2382577.2382579
  39. M. Rosenblatt, L. Tejavibulya, R. Jiang, S. Noble, and D. Scheinost, “Data leakage inflates prediction performance in connectome-based machine learning models, ” Nature Commun., vol. 15, no. 1, Art. no. 1829, Feb. 2024. https://doi.org/doi: 10.1038/s41467-024-46150-w
  40. S. Whalen, J. Schreiber, W. S. Noble, and K. S. Pollard, “Navigating the pitfalls of applying machine learning in genomics, ” Nature Rev. Genet., vol. 23, no. 3, pp. 169–181, Mar. 2022. https://doi.org/doi: 10.1038/s41576-021-00434-9
  41. M. Roberts et al., “Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans, ” Nature Mach. Intell., vol. 3, pp. 199–217, Mar. 2021. https://doi.org/doi: 10.1038/s42256-021-00307-0
  42. S. Varma and R. Simon, “Bias in error estimation when using cross-validation for model selection, ” BMC Bioinformatics, vol. 7, Art. no. 91, Feb. 2006. https://doi.org/doi: 10.1186/1471-2105-7-91
  43. E. Christodoulou, J. Ma, G. S. Collins, E. W. Steyerberg, J. Y. Verbakel, and B. Van Calster, “A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models, ” J. Clin. Epidemiol., vol. 110, pp. 12–22, Jun. 2019. https://doi.org/doi: 10.1016/j.jclinepi.2019.02.004
  44. B. Van Calster, D. J. McLernon, M. van Smeden, L. Wynants, and E. W. Steyerberg, “Calibration: The Achilles heel of predictive analytics, ” BMC Med., vol. 17, no. 1, Art. no. 230, Dec. 2019. https://doi.org/doi: 10.1186/s12916-019-1466-7
  45. S. G. Finlayson et al., “The clinician and dataset shift in artificial intelligence, ” N. Engl. J. Med., vol. 385, no. 3, pp. 283–286, Jul. 2021. https://doi.org/doi: 10.1056/NEJMc2104626
  46. G. Varoquaux and V. Cheplygina, “Machine learning for medical imaging: Methodological failures and recommendations for the future, ” npj Digit. Med., vol. 5, no. 1, Art. no. 48, Apr. 2022. https://doi.org/doi: 10.1038/s41746-022-00592-y
  47. G. S. Collins et al., “TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods, ” BMJ, vol. 385, Art. no. e078378, Apr. 2024. https://doi.org/doi: 10.1136/bmj-2023-078378
  48. J. C. C. Kwong et al., “Standardized reporting of machine learning applications in urology: The STREAM-URO framework, ” Eur. Urol. Focus, vol. 7, no. 4, pp. 672–682, Jul. 2021. https://doi.org/doi: 10.1016/j.euf.2021.07.004
  49. R. F. Wolff et al., “PROBAST: A tool to assess the risk of bias and applicability of prediction model studies, ” Ann. Intern. Med., vol. 170, no. 1, pp. 51–58, Jan. 2019. https://doi.org/doi: 10.7326/M18-1376
  50. R. D. Riley et al., “Calculating the sample size required for developing a clinical prediction model, ” BMJ, vol. 368, Art. no. m441, Mar. 2020. https://doi.org/doi: 10.1136/bmj.m441
  51. A. C. Davison and D. V. Hinkley, Bootstrap Methods and Their Application. Cambridge, U. K.: Cambridge University Press, 1997. https://doi.org/doi: 10.1017/CBO9780511802843
  52. L. Rubini, P. Soundarapandian, and P. Eswaran, “Chronic Kidney Disease, ” UCI Machine Learning Repository, 2015. Available: https://archive.ics.uci.edu/dataset/336/chronic+kidney+disease. https://doi.org/doi: 10.24432/C5G020
  53. D. F. Andrews and A. M. Herzberg, Data: A Collection of Problems from Many Fields for the Student and Research Worker. New York, NY, USA: Springer-Verlag, 1985. https://doi.org/doi: 10.1007/978-1-4612-5098-2
4 1

Similar Articles

You may also start an advanced similarity search for this article.