Research Articles Open Access

How Duplicate Samples Inflate Machine-Learning Evaluation: A Controlled Cross-Dataset Study of Train-Test Leakage

Ehsan Sarwar * , Ayesha Ali
* Corresponding author: Ehsan Sarwar ( mehsan.sarwaruol@gmail.com )
Available online January 1, 2026
PDF

Abstract

Machine-learning evaluation assumes that the test set contains observations that were not available during model fitting. Exact duplicates that cross the train-test boundary break that assumption and can raise reported performance without improving generalization to new observations. This study measures that effect under controlled conditions. Four public classification datasets and one synthetic dataset are evaluated with logistic regression, 5-nearest neighbors, a decision tree, and a random forest. For each dataset, ten classified 70/30 train-test splits are created. The training partition is left unchanged, while 0, 5, 10, 20, 30, or 50 percent of the test records are replaced by exact class-matched copies of training records. This design isolates evaluation contamination from changes in training data. At 50 percent overlap, the mean inflation is 1.60 percentage points for logistic regression and 1.70 for 5-nearest neighbors. It rises to 7.15 points for the decision tree and 3.49 for the random forest. The largest single condition occurs on the synthetic dataset, where decision-tree accuracy rises by 18.37 points. At the same leakage level, decision-tree and random-forest accuracy on the duplicated subset is 100 percent, while their accuracy on the remaining uncontaminated records is 84.94 and 92.97 percent, respectively. Across the 20 dataset-model combinations, clean-test error is strongly associated with inflation at 50 percent overlap (Spearman ρ = 0.81, p = 1.67 × 10⁻⁵). The results show that duplicate leakage does not affect every classifier equally and can be most misleading when a flexible model has modest clean-test performance. 

Keywords

Data leakage duplicate samples machine learning reproducibility cross-validation

Introduction

A classifier is expected to be evaluated on observations that were not used to fit it. This requirement is simple to state, yet it is easy to violate when a dataset contains repeated records, repeated entities, near-duplicates, or preprocessing steps that use information from the test partition. Once train and test data are no longer independent, the reported score no longer measures the intended generalization problem. It measures performance on a mixture of new observations and information that has already entered the learning process.

Leakage has been discussed in data mining for more than a decade. Kaufman et al. formalized leakage as information that would not legitimately be available at prediction time. More recent work has shown that the problem is not confined to competitions or poorly documented processes. Kapoor and Narayanan reviewed leakage across scientific applications and found that it contributed to overoptimistic results in many fields. Rosenblatt et al. reached a related conclusion in connectome-based prediction: repeated subjects and leaky feature selection could materially raise measured performance. These studies differ in domain, but they share the same evaluation concern. The test data must remain genuinely external to model development.

Exact duplicate leakage is a narrow but useful case to study since the contamination mechanism can be controlled exactly . A flexible classifier, when faced with a training-set record reappearing in the test set, may simply echo the training decision rather than generalizing from the surrounding structure. A decision tree can classify the duplicate with the same fitted partitions; an ensemble can inherit the same benefit across many trees. However , a linear model can not memorize an arbitrary sample like that . The effect of duplication should therefore depend on both the overlap and the family of models.

The literature contains strong warnings about leakage. The practical measurement question is: how much can exact overlap inflate common classification metrics, and does the answer differ across models and datasets? A controlled experiment can answer that question without mixing a number of leakage mechanisms. The training set can be fixed, the test set can be contaminated only and one can specify directly the overlap fraction. This design isolates the effect of duplicate evaluation samples from the effects of resampling, feature selection, hyperparameter tuning, or changes in the training distribution.

This design is used in this study. We evaluate four common classifiers on four public classification datasets and one synthetic dataset. Each dataset is trained with ten stratified splits. For each split, the fitted model is kept constant and the test overlap is increased from 0 to 50 percent. We report accuracy, macro-F1, balanced accuracy and Matthews correlation coefficient (MCC). The analysis further distinguishes predictions on duplicated test records from predictions on the rest of uncontaminated records.

The experiment is guided by three questions:

  • RQ1: How does increasing the fraction of train-test overlap impact the reported classification performance?

  • RQ2: Are memory-capable tree and neighbor methods more sensitive to duplicate leakage than a linear classifier?

  • RQ3: When clean-test performance is lower, does leakage inflation increase?

This paper does not model all types of leakage. It isolates one mechanism such that its effect is directly measured. The evidence we obtain offers a pragmatic warning for the construction of benchmarks, classroom experiments, applied machine-learning papers, and any workflow in which duplicate or repeated observations are able to cross the evaluation boundary.

Complete Article

The complete article, including all figures, tables, equations and algorithms, is available in the official publication PDF.

Conclusion

Exact train-test duplicates can raise machine-learning evaluation scores without improving the fitted model. The size of that distortion depends on the classifier and on the underlying clean-test difficulty. At 50 percent exact overlap, mean accuracy rises by 7.15 percentage points for the decision tree and 3.49 points for random forest. Logistic regression and 5-NN rise by 1.60 and 1.70 points, respectively. The most affected condition is the decision tree on the synthetic dataset, where the reported score increases by 18.37 points.

The subset analysis explains why. Decision-tree and random-forest accuracy on duplicated test records reaches 100 percent at the 50 percent overlap condition, while accuracy on the remaining uncontaminated records is lower. The resulting aggregate score is a mixture of memorised and truly unseen observations. The relationship between inflation and clean-test error is also robust across dataset-model combinations (ρ=0.81), which suggests that leakage can be especially misleading when baseline performance leaves room for apparent improvement.

The practical implication is straightforward: one must check evaluation integrity at the data-split level, not infer from the final metric. We propose cheap precautions such as exact-overlap audits, entity-aware grouping, train-only preprocessing, and separate reporting of contaminated and uncontaminated performance. They do not address all types of leakage, but they stop a basic data-management error from being mistaken for model progress.

References

  1. S. Kaufman, S. Rosset, C. Perlich, and O. Stitelman, “Leakage in data mining: Formulation, detection, and avoidance, ” ACM Trans. Knowl. Discov. Data, vol. 6, no. 4, pp. 15:1–15:21, 2012, doi: 10.1145/2382577.2382579
  2. S. Kapoor and A. Narayanan, “Leakage and the reproducibility crisis in machine-learning-based science, ” Patterns, vol. 4, no. 9, p. 100804, 2023, doi: 10.1016/j.patter.2023.100804
  3. J. D. Rosenblatt et al., “Data leakage inflates prediction performance in connectome-based machine learning models, ” Nature Communications, vol. 15, 2024, doi: 10.1038/s41467-024-46150-w
  4. F. Pedregosa et al., “Scikit-learn: Machine learning in Python, ” J. Mach. Learn. Res., vol. 12, pp. 2825–2830, 2011.
  5. A. K. Singh, M. Y. Kocyigit, A. Poulton, D. Esiobu, M. Lomeli, G. Szilvasy, and D. Hupkes, “Evaluation data contamination in LLMs: How do we measure it and when does it matter?” arXiv:2411.03923, 2024.
  6. N. Kandpal, E. Wallace, and C. Raffel, “Deduplicating training data mitigates privacy risks in language models, ” in Proc. 39th Int. Conf. Machine Learning, vol. 162, pp. 10697–10707, 2022.
  7. S. Varma and R. Simon, “Bias in error estimation when using cross-validation for model selection, ” BMC Bioinformatics, vol. 7, p. 91, 2006, doi: 10.1186/1471-2105-7-91
  8. G. C. Cawley and N. L. C. Talbot, “On over-fitting in model selection and subsequent selection bias in performance evaluation, ” J. Mach. Learn. Res., vol. 11, pp. 2079–2107, 2010.
  9. C. Dwork, V. Feldman, M. Hardt, T. Pitassi, O. Reingold, and A. Roth, “The reusable holdout: Preserving validity in adaptive data analysis, ” Science, vol. 349, no. 6248, pp. 636–638, 2015, doi: 10.1126/science.aaa9375
  10. Y. Bengio and Y. Grandvalet, “No unbiased estimator of the variance of K-fold cross-validation, ” J. Mach. Learn. Res., vol. 5, pp. 1089–1105, 2004.
  11. C. Nadeau and Y. Bengio, “Inference for the generalization error, ” Machine Learning, vol. 52, pp. 239–281, 2003, doi: 10.1023/A:1024068626366
  12. G. Varoquaux, “Cross-validation failure: Small sample sizes lead to large error bars, ” NeuroImage, vol. 180, pp. 68–77, 2018, doi: 10.1016/j.neuroimage.2017.06.061
  13. R. Kohavi, “A study of cross-validation and bootstrap for accuracy estimation and model selection, ” in Proc. 14th Int. Joint Conf. Artificial Intelligence, pp. 1137–1145, 1995.
  14. T. G. Dietterich, “Approximate statistical tests for comparing supervised classification learning algorithms, ” Neural Computation, vol. 10, no. 7, pp. 1895–1923, 1998, doi: 10.1162/089976698300017197
  15. J. Demsar, “Statistical comparisons of classifiers over multiple data sets, ” J. Mach. Learn. Res., vol. 7, pp. 1–30, 2006.
  16. T. M. Cover and P. E. Hart, “Nearest neighbor pattern classification, ” IEEE Trans. Inf. Theory, vol. 13, no. 1, pp. 21–27, 1967, doi: 10.1109/TIT.1967.1053964
  17. L. Breiman, “Random forests, ” Machine Learning, vol. 45, pp. 5–32, 2001, doi: 10.1023/A:1010933404324
  18. D. Chicco and G. Jurman, “The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation, ” BMC Genomics, vol. 21, p. 6, 2020, doi: 10.1186/s12864-019-6413-7
  19. W. Wolberg, O. Mangasarian, N. Street, and W. Street, “Breast Cancer Wisconsin (Diagnostic), ” UCI Machine Learning Repository, 1993, doi: 10.24432/C5DW2B
  20. S. Aeberhard and M. Forina, “Wine, ” UCI Machine Learning Repository, 1992, doi: 10.24432/C5PC7J
  21. E. Alpaydin and C. Kaynak, “Optical Recognition of Handwritten Digits, ” UCI Machine Learning Repository, 1998, doi: 10.24432/C50P49
  22. R. A. Fisher, “The use of multiple measurements in taxonomic problems, ” Annals of Eugenics, vol. 7, no. 2, pp. 179–188, 1936, doi: 10.1111/j.1469-1809.1936.tb02137.x
134 24

Similar Articles

1-10 of 20

You may also start an advanced similarity search for this article.