Abstract
Assessment of ureteropelvic junction obstruction (UPJO) in horseshoe kidney (HSK) is difficult, and curated data for this congenital anomaly are scarce, which makes synthetic data attractive for building machine-learning (ML) processes. How much can be concluded from such data is rarely evaluated. We report a reproducible methodological case study on a synthetic cohort of 1,000 HSK records (no real patients) and ask how far headline performance and explanation plots can be trusted. Post-treatment fields were excluded to prevent target leakage. On a balanced 80:20 holdout (N = 200), XGBoost reached ROC-AUC 0.997 and accuracy 0.955, and Logistic Regression, SVM, and Random Forest performed comparably (ROC-AUC 0.994–0.996). In repeated cross-validation, all four models reached ROC-AUC 0.990–0.993, while logistic regression reached 0.983. When a random fraction of labels was flipped (10 independent draws), observed ROC-AUC fell to 0.93 at 5%, 0.87-0.88 at 10%, and 0.76–0.77 at 20%, whereas Gaussian measurement noise of 1.0 × SD on APD_mm and T₁/₂ reduced it only to 0.95-0.96. As separate process checks that are not evidence about HSK or UPJO, the same workflow reached ROC-AUC ≥ 0.999 on the easy public UCI Chronic Kidney Disease dataset (n = 400) but only 0.81-0.85 on a harder public urine-analysis dataset (n = 79). Near-perfect metrics on the synthetic cohort therefore reflect rule-based labels, not diagnostic certainty. We give reporting and stress-testing recommendations for ML studies that use synthetic clinical data. This is a methodological study, not a clinically validated diagnostic tool.
Keywords
Introduction
Horseshoe kidney (HSK) is the most common renal fusion anomaly, affecting about 1 in 400 to 1 in 800 people [1], [2]. Failure of axial rotation and arrested cephalad migration produce a fibrous or parenchymal isthmus joining the lower poles across the midline [1]. The renal pelves therefore lie anteriorly, the ureters cross the isthmus, ureteral insertions are often high, and anomalous lower-pole arteries arising directly from the aorta or iliac vessels can compress the collecting system [2], [3]. These features predispose patients to urinary stasis, recurrent nephrolithiasis and ureteropelvic junction obstruction (UPJO) [2], [3].
Assessing UPJO in this anomalous anatomy is difficult. A typical work-up combines ultrasound, including the Society for Fetal Urology (SFU) hydronephrosis grade and the maximal anteroposterior pelvic diameter (APD) [4-5], with diuretic renography (e.g., ^99mTc-MAG3) to obtain differential renal function (DRF), drainage half-time (T_1/2) and the shape of the clearance curve are shown in Figure 1 and Figure 2. In HSK, however, delayed tracer washout is common and may reflect gravity-dependent pooling in anteriorly rotated calyces rather than true mechanical obstruction [2], [6]. Even in anatomically normal kidneys, the decision to operate is made from several imperfect surrogates rather than a single gold standard [4-5]. Objective decision support that separates benign stasis from progressive obstruction needing pyeloplasty would therefore be valuable [6].
Machine-learning (ML) models such as extreme gradient boosting (XGBoost) [7] perform well on many tabular medical prediction tasks [8], but clinical translation is limited by their opacity and by the scarcity of data for rare congenital anomalies [9]. Explainable-AI (XAI) methods such as SHapley Additive exPlanations (SHAP) are often used to address the first problem [9], [10], and synthetic data are increasingly used to address the second [11], [12]. Synthetic tabular data allow processes to be built safely, but they need caution: artifacts of the generating process can inflate accuracy while omitting the difficulties of noisy biological cohorts [11], [13]. When both performance and explanations are computed on the same synthetic data, it is easy to mistake a model that has recovered the generator’s rule for a model that has learned clinical physiology.
This study asks three methodological questions.
RQ1: How much of the near-ceiling performance on a synthetic HSK-UPJO cohort is explained by a simple, low-dimensional rule?
RQ2: What do SHAP explanations reveal on such data, and what can they not establish?
RQ3: How sensitive are the headline metrics to realistic sources of error, namely reference-standard (label) disagreement and measurement noise on the key imaging variables?
We train an explainable workflow (XGBoost, Random Forest, support vector machine (SVM) and Logistic Regression, with SHAP [9], [10]) on the synthetic HSK-UPJO dataset and then stress-test it. Our contributions are: (i) a leakage-aware feature set with a documented audit of flawed fields in the source file; (ii) simple and rule-based baselines with repeated cross-validation that quantify how much signal the two dominant variables carry; (iii) label-noise and measurement-noise simulations, each repeated over 10 independent draws, that quantify how quickly the metrics degrade; (iv) supplementary runs on two public real-world datasets, one easy (UCI Chronic Kidney Disease) and one hard (urine analysis), which show that the same process produces near-ceiling or modest metrics depending on the task, and which are not presented as evidence about HSK or UPJO; and (v) release of the code, a corrected dataset and a single script that regenerates every reported number and figure. No real HSK patient is analyzed, and no clinical validity is claimed.
The rest of the paper is organized as follows. Section 2 reviews related work and states the research gap. Section 3 describes the dataset, the leakage controls, the models and the validation and stress-testing protocol. Section 4 reports the results, with the synthetic-cohort experiments (Sections 4.1-4.5) kept separate from the supplementary public-dataset experiments (Section 4.6). Section 5 discusses the findings against previous studies and their practical implications, Section 6 lists the limitations and Section 7 concludes.
Complete Article
The complete article, including all figures, tables, equations and algorithms, is available in the official publication PDF.
Conclusion
This study presented a reproducible, explainable machine-learning workflow (XGBoost, Random Forest, SVM, Logistic Regression and TreeSHAP) for UPJO classification in a synthetic cohort of horseshoe kidneys, and used it to show why results from such data must not be read as clinical evidence. The models reached near-ceiling discrimination, but a two-variable logistic regression came within about 0.01 of them (RQ1); SHAP recovered threshold-like rules that mirror clinical conventions because the generator encoded them, not because the model learned physiology (RQ2), and observed performance fell sharply under label noise while remaining stable under measurement noise (RQ3). The same process was near-perfect on an easy public CKD dataset but reached only 0.81-0.85 on a harder public urine-analysis dataset, which confirms that such metrics depend on the task. The contribution of this work is methodological rather than clinical: a protocol of leakage audit, rule-recovering baselines, noise stress tests and real-data contrasts, together with reporting recommendations. Researchers who use synthetic clinical data should release the data and generator, report simple baselines and noise-sensitivity analyses and avoid clinical interpretation. Validation on multi-institutional HSK-UPJO registries remains essential before any translational claim.
References
- K. Taghavi, J. Kirkpatrick, and S. A. Mirjalili, “The horseshoe kidney: Surgical anatomy and embryology, ” J. Pediatr. Urol., vol. 12, no. 5, pp. 275–280, Oct. 2016. https://doi.org/doi: 10.1016/j.jpurol.2016.04.033
- M. I. A. Elmaadawy, S. W. Kim, S. K. Kang, S. W. Han, and Y. S. Lee, “A retrospective analysis of ureteropelvic junction obstructions in patients with horseshoe kidney, ” Transl. Androl. Urol., vol. 10, no. 11, pp. 4173–4180, Nov. 2021. https://doi.org/doi: 10.21037/tau-21-471
- A. W. Partin, R. R. Dmochowski, L. R. Kavoussi, C. A. Peters, and A. J. Wein, Eds., Campbell-Walsh-Wein Urology, 12th ed. Philadelphia, PA, USA: Elsevier, 2020.
- H. T. Nguyen et al., “The Society for Fetal Urology consensus statement on the evaluation and management of antenatal hydronephrosis, ” J. Pediatr. Urol., vol. 6, no. 3, pp. 212–231, Jun. 2010. https://doi.org/doi: 10.1016/j.jpurol.2010.02.205
- I. Gordon, A. Piepsz, and R. Sixt, “Guidelines for standard and diuretic renogram in children, ” Eur. J. Nucl. Med. Mol. Imaging, vol. 38, no. 6, pp. 1175–1188, Jun. 2011. https://doi.org/doi: 10.1007/s00259-011-1811-3
- J. H. Ross and R. Kay, “Ureteropelvic junction obstruction in anomalous kidneys, ” Urol. Clin. North Am., vol. 25, no. 2, pp. 219–225, May 1998. https://doi.org/doi: 10.1016/S0094-0143(05)70010-0
- T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system, ” in Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining (KDD ’16), San Francisco, CA, USA, 2016, pp. 785–794. https://doi.org/doi: 10.1145/2939672.2939785
- L. Grinsztajn, E. Oyallon, and G. Varoquaux, “Why do tree-based models still outperform deep learning on typical tabular data?, ” in Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 35, Datasets and Benchmarks Track, 2022, pp. 507–520. https://doi.org/doi: 10.52202/068431-0037
- S. M. Lundberg et al., “From local explanations to global understanding with explainable AI for trees, ” Nature Mach. Intell., vol. 2, no. 1, pp. 56–67, Jan. 2020. https://doi.org/doi: 10.1038/s42256-019-0138-9
- S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions, ” in Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 30, 2017, pp. 4765–4774.
- R. J. Chen, M. Y. Lu, T. Y. Chen, D. F. K. Williamson, and F. Mahmood, “Synthetic data in machine learning for medicine and healthcare, ” Nature Biomed. Eng., vol. 5, no. 6, pp. 493–497, Jun. 2021. https://doi.org/doi: 10.1038/s41551-021-00751-8
- A. Gonzales, G. Guruswamy, and S. R. Smith, “Synthetic data in health care: A narrative review, ” PLOS Digit. Health, vol. 2, no. 1, Art. no. e0000082, Jan. 2023. https://doi.org/doi: 10.1371/journal.pdig.0000082
- S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine-learning-based science, ” Patterns, vol. 4, no. 9, Art. no. 100804, Sep. 2023. https://doi.org/doi: 10.1016/j.patter.2023.100804
- E. S. Blum et al., “Early detection of ureteropelvic junction obstruction using signal analysis and machine learning: A dynamic solution to a dynamic problem, ” J. Urol., vol. 199, no. 3, pp. 847–852, Mar. 2018. https://doi.org/doi: 10.1016/j.juro.2017.09.147
- J. J. Cerrolaza et al., “Quantitative ultrasound for measuring obstructive severity in children with hydronephrosis, ” J. Urol., vol. 195, no. 4, pt. 1, pp. 1093–1099, Apr. 2016. https://doi.org/doi: 10.1016/j.juro.2015.10.173
- L. Erdman et al., “Predicting obstructive hydronephrosis based on ultrasound alone, ” in Medical Image Computing and Computer-Assisted Intervention-MICCAI 2020, Lecture Notes in Computer Science, vol. 12263. Cham, Switzerland: Springer, 2020, pp. 493–503. https://doi.org/doi: 10.1007/978-3-030-59716-0_47
- L. Erdman et al., “The Hydronephrosis Severity Index guides paediatric antenatal hydronephrosis management based on artificial intelligence applied to ultrasound images alone, ” Sci. Rep., vol. 14, no. 1, Art. no. 22748, Oct. 2024. https://doi.org/doi: 10.1038/s41598-024-72271-9
- A. Khondker et al., “Predicting obstruction risk using common ultrasonography parameters in paediatric hydronephrosis with machine learning, ” BJU Int., vol. 133, no. 1, pp. 79–86, Jan. 2024. https://doi.org/doi: 10.1111/bju.16159
- A. Khondker et al., “Longitudinal image-based prediction of surgical intervention in infants with hydronephrosis using deep learning: Is a single ultrasound enough?, ” PLOS Digit. Health, vol. 4, no. 8, Art. no. e0000939, Aug. 2025. https://doi.org/doi: 10.1371/journal.pdig.0000939
- E. Drysdale et al., “Personalized application of machine learning algorithms to identify pediatric patients at risk for recurrent ureteropelvic junction obstruction after dismembered pyeloplasty, ” World J. Urol., vol. 40, no. 2, pp. 593–599, Feb. 2022. https://doi.org/doi: 10.1007/s00345-021-03879-z
- C. A. Alici and B. Tokar, “Prediction of surgical necessity in children with ureteropelvic junction obstruction using machine learning, ” Ir. J. Med. Sci., vol. 194, no. 2, pp. 583–590, Apr. 2025. https://doi.org/doi: 10.1007/s11845-025-03895-7
- C. Wolmer, J. Delmas, S. Pecorelli, E. Dobremez, C. Ferdynus, and L. Harper, “Predicting clinically significant events in children with ureteropelvic junction obstruction, ” Front. Pediatr., vol. 12, Art. no. 1409170, 2024. https://doi.org/doi: 10.3389/fped.2024.1409170
- M. T. Ribeiro, S. Singh, and C. Guestrin, “‘Why should I trust you?’: Explaining the predictions of any classifier, ” in Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining (KDD ’16), San Francisco, CA, USA, 2016, pp. 1135–1144. https://doi.org/doi: 10.1145/2939672.2939778
- S. M. Lundberg et al., “Explainable machine-learning predictions for the prevention of hypoxaemia during surgery, ” Nature Biomed. Eng., vol. 2, no. 10, pp. 749–760, Oct. 2018. https://doi.org/doi: 10.1038/s41551-018-0304-0
- I. E. Kumar, S. Venkatasubramanian, C. Scheidegger, and S. A. Friedler, “Problems with Shapley-value-based explanations as feature importance measures, ” in Proc. 37th Int. Conf. Mach. Learn. (ICML), PMLR, vol. 119, 2020, pp. 5491–5500.
- D. Slack, S. Hilgard, E. Jia, S. Singh, and H. Lakkaraju, “Fooling LIME and SHAP: Adversarial attacks on post hoc explanation methods, ” in Proc. AAAI/ACM Conf. AI, Ethics, Soc. (AIES), New York, NY, USA, 2020, pp. 180–186. https://doi.org/doi: 10.1145/3375627.3375830
- M. Ghassemi, L. Oakden-Rayner, and A. L. Beam, “The false hope of current approaches to explainable artificial intelligence in health care, ” Lancet Digit. Health, vol. 3, no. 11, pp. e745–e750, Nov. 2021. https://doi.org/doi: 10.1016/S2589-7500(21)00208-9
- C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead, ” Nature Mach. Intell., vol. 1, no. 5, pp. 206–215, May 2019. https://doi.org/doi: 10.1038/s42256-019-0048-x
- J. Walonoski et al., “Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record, ” J. Amer. Med. Inform. Assoc., vol. 25, no. 3, pp. 230–238, Mar. 2018. https://doi.org/doi: 10.1093/jamia/ocx079
- L. Xu, M. Skoularidou, A. Cuesta-Infante, and K. Veeramachaneni, “Modeling tabular data using conditional GAN, ” in Adv. Neural Inf. Process. Syst. (NeurIPS), vol. 32, 2019, pp. 7335–7345.
- J. Yoon, L. N. Drumright, and M. van der Schaar, “Anonymization through data synthesis using generative adversarial networks (ADS-GAN), ” IEEE J. Biomed. Health Inform., vol. 24, no. 8, pp. 2378–2388, Aug. 2020. https://doi.org/doi: 10.1109/JBHI.2020.2980262
- M. Giuffrè and D. L. Shung, “Harnessing the power of synthetic data in healthcare: Innovation, application, and privacy, ” npj Digit. Med., vol. 6, no. 1, Art. no. 186, Oct. 2023. https://doi.org/doi: 10.1038/s41746-023-00927-3
- A. Alaa, B. van Breugel, E. S. Saveliev, and M. van der Schaar, “How faithful is your synthetic data? Sample-level metrics for evaluating and auditing generative models, ” in Proc. 39th Int. Conf. Mach. Learn. (ICML), PMLR, vol. 162, 2022, pp. 290–306.
- B. Frénay and M. Verleysen, “Classification in the presence of label noise: A survey, ” IEEE Trans. Neural Netw. Learn. Syst., vol. 25, no. 5, pp. 845–869, May 2014. https://doi.org/doi: 10.1109/TNNLS.2013.2292894
- C. G. Northcutt, L. Jiang, and I. L. Chuang, “Confident learning: Estimating uncertainty in dataset labels, ” J. Artif. Intell. Res., vol. 70, pp. 1373–1411, 2021. https://doi.org/doi: 10.1613/jair.1.12125
- C. G. Northcutt, A. Athalye, and J. Mueller, “Pervasive label errors in test sets destabilize machine learning benchmarks, ” in Proc. Neural Information Processing Systems Track on Datasets and Benchmarks, vol. 1, 2021.
- D. Karimi, H. Dou, S. K. Warfield, and A. Gholipour, “Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis, ” Med. Image Anal., vol. 65, Art. no. 101759, Oct. 2020. https://doi.org/doi: 10.1016/j.media.2020.101759
- S. Kaufman, S. Rosset, C. Perlich, and O. Stitelman, “Leakage in data mining: Formulation, detection, and avoidance, ” ACM Trans. Knowl. Discov. Data, vol. 6, no. 4, Art. no. 15, Dec. 2012. https://doi.org/doi: 10.1145/2382577.2382579
- M. Rosenblatt, L. Tejavibulya, R. Jiang, S. Noble, and D. Scheinost, “Data leakage inflates prediction performance in connectome-based machine learning models, ” Nature Commun., vol. 15, no. 1, Art. no. 1829, Feb. 2024. https://doi.org/doi: 10.1038/s41467-024-46150-w
- S. Whalen, J. Schreiber, W. S. Noble, and K. S. Pollard, “Navigating the pitfalls of applying machine learning in genomics, ” Nature Rev. Genet., vol. 23, no. 3, pp. 169–181, Mar. 2022. https://doi.org/doi: 10.1038/s41576-021-00434-9
- M. Roberts et al., “Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans, ” Nature Mach. Intell., vol. 3, pp. 199–217, Mar. 2021. https://doi.org/doi: 10.1038/s42256-021-00307-0
- S. Varma and R. Simon, “Bias in error estimation when using cross-validation for model selection, ” BMC Bioinformatics, vol. 7, Art. no. 91, Feb. 2006. https://doi.org/doi: 10.1186/1471-2105-7-91
- E. Christodoulou, J. Ma, G. S. Collins, E. W. Steyerberg, J. Y. Verbakel, and B. Van Calster, “A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models, ” J. Clin. Epidemiol., vol. 110, pp. 12–22, Jun. 2019. https://doi.org/doi: 10.1016/j.jclinepi.2019.02.004
- B. Van Calster, D. J. McLernon, M. van Smeden, L. Wynants, and E. W. Steyerberg, “Calibration: The Achilles heel of predictive analytics, ” BMC Med., vol. 17, no. 1, Art. no. 230, Dec. 2019. https://doi.org/doi: 10.1186/s12916-019-1466-7
- S. G. Finlayson et al., “The clinician and dataset shift in artificial intelligence, ” N. Engl. J. Med., vol. 385, no. 3, pp. 283–286, Jul. 2021. https://doi.org/doi: 10.1056/NEJMc2104626
- G. Varoquaux and V. Cheplygina, “Machine learning for medical imaging: Methodological failures and recommendations for the future, ” npj Digit. Med., vol. 5, no. 1, Art. no. 48, Apr. 2022. https://doi.org/doi: 10.1038/s41746-022-00592-y
- G. S. Collins et al., “TRIPOD+AI statement: Updated guidance for reporting clinical prediction models that use regression or machine learning methods, ” BMJ, vol. 385, Art. no. e078378, Apr. 2024. https://doi.org/doi: 10.1136/bmj-2023-078378
- J. C. C. Kwong et al., “Standardized reporting of machine learning applications in urology: The STREAM-URO framework, ” Eur. Urol. Focus, vol. 7, no. 4, pp. 672–682, Jul. 2021. https://doi.org/doi: 10.1016/j.euf.2021.07.004
- R. F. Wolff et al., “PROBAST: A tool to assess the risk of bias and applicability of prediction model studies, ” Ann. Intern. Med., vol. 170, no. 1, pp. 51–58, Jan. 2019. https://doi.org/doi: 10.7326/M18-1376
- R. D. Riley et al., “Calculating the sample size required for developing a clinical prediction model, ” BMJ, vol. 368, Art. no. m441, Mar. 2020. https://doi.org/doi: 10.1136/bmj.m441
- A. C. Davison and D. V. Hinkley, Bootstrap Methods and Their Application. Cambridge, U. K.: Cambridge University Press, 1997. https://doi.org/doi: 10.1017/CBO9780511802843
- L. Rubini, P. Soundarapandian, and P. Eswaran, “Chronic Kidney Disease, ” UCI Machine Learning Repository, 2015. Available: https://archive.ics.uci.edu/dataset/336/chronic+kidney+disease. https://doi.org/doi: 10.24432/C5G020
- D. F. Andrews and A. M. Herzberg, Data: A Collection of Problems from Many Fields for the Student and Research Worker. New York, NY, USA: Springer-Verlag, 1985. https://doi.org/doi: 10.1007/978-1-4612-5098-2