Research Articles Open Access

A Comparative Analysis of Machine Learning Algorithms for Diagnostic Breast Cancer Classification: Logistic Regression, Support Vector Machine and Random Forest

Ismayilov Togrul Atabay *
* Corresponding author: Ismayilov Togrul Atabay ( togrul.ismayilov@adau.edu.az )
Available online October 25, 2026
PDF

Abstract

The choice of a classification algorithm can directly affect the usefulness of a data-driven decision system. This study compares three commonly used machine learning algorithms: Logistic Regression (LR), Support Vector Machine (SVM) with a radial basis function kernel, and Random Forest (RF). The comparison was performed using the Wisconsin Diagnostic Breast Cancer dataset, which contains 569 samples and 30 features. The study follows the CRISP-DM methodology. Model performance was evaluated using stratified 5-fold cross-validation repeated 10 times, resulting in a total of 50 folds. Differences between the models were tested using the Nadeau-Bengio corrected resampled t-test. The average accuracy was 97.89% for LR, 97.56% for SVM, and 95.92% for RF. However, the corrected statistical test did not find a significant difference between any pair of models, with the smallest p-value being 0.085. In terms of computational cost, RF took about 94 times longer to train than SVM and about 10 times longer to make predictions. The findings show that when the classes can be separated relatively well using a simple decision boundary, simpler models can perform as well as more complex ensemble models. The results also highlight the importance of using statistical tests that account for the dependence between cross-validation folds when comparing machine learning classifiers.

Keywords

Machine learning classification logistic regression support vector machine cross-validation

Introduction

Supervised machine learning is used to support decision-making across a wide range of fields, from medical diagnosis to financial risk assessment [1]. In these applications, the main practical question is often framed as “Which algorithm performs best?” However, the answer depends on the size of the dataset, the structure of the features, the evaluation protocol, and the computational costs.

Extracting quantitative measurements of cell nuclei from digitized images of fine-needle aspiration (FNA) samples of breast lesions—and predicting whether a lesion is benign or malignant based on these measurements—is a topic extensively studied in machine learning literature [2], [3]. The resulting Wisconsin Diagnostic Breast Cancer dataset is one of the most widely used test datasets for comparing algorithms, owing to its open availability and small size [4].

However, in many such comparisons, results are presented based on a single train/test split or as an average across folds; the statistical significance of differences is not assessed, and computational costs are often not reported at all. With small datasets, random variability between folds can exceed the differences between algorithms, rendering claims about the “best” algorithm unstable.

The aim of this study is to determine whether there are statistically significant differences among three well-known classifiers—LR, SVM, and RF—in terms of overall accuracy, sensitivity for the malignant class, and F1-score, under identical conditions. The study objectives are: (1) to characterize the dataset through exploratory analysis; (2) to evaluate the three models using the same repeated cross-validation protocol and test for differences using a method that accounts for correlation; and (3) to analyze error structure, computational costs, sensitivity to hyperparameters, and learning curves.

The remainder of the article is organized as follows: Section 2 reviews related work, Section 3 describes the materials and methods (dataset, data preparation, and the three classification algorithms), and Section 4 presents the experimental setup and evaluation protocol. Section 5 presents and discusses the results, while Section 6 outlines the conclusions and directions for future research.

Complete Article

The complete article, including all figures, tables, equations and algorithms, is available in the official publication PDF.

Conclusion

This study benchmarked LR, SVM, and RF on the Wisconsin Diagnostic Breast Cancer dataset using repeated cross-validation as the primary comparison protocol. LR (97.89%) and SVM (97.56%) nominally outperformed RF (95.92%) in mean accuracy, but the Nadeau–Bengio corrected tests reported in the study did not establish statistically significant pairwise accuracy differences. The separate 10-fold prediction analysis summarized error structure and ROC behavior and was not used to support the primary inferential comparison. RF had higher measured training and prediction cost in the specific single-core implementation, but these timing ratios should not be generalized as intrinsic algorithmic efficiency. Because the models were not externally validated or clinically calibrated, the findings support benchmarking conclusions only and not clinical diagnosis or deployment.

Future work plans include: (i) incorporating gradient boosting and neural networks; (ii) performing multiple comparisons across several datasets using the Friedman test; (iii) calibrating probabilities and selecting a decision threshold focused on sensitivity; (iv) validation on external data; and (v) interpretation of features using grouped importance methods.

References

  1. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed. New York, NY, USA: Springer, 2009.
  2. W. N. Street, W. H. Wolberg, and O. L. Mangasarian, “Nuclear feature extraction for breast tumor diagnosis, ” in Biomedical Image Processing and Biomedical Visualization, Proc. SPIE, vol. 1905, 1993, pp. 861–870.
  3. W. H. Wolberg and O. L. Mangasarian, “Multisurface method of pattern separation for medical diagnosis applied to breast cytology, ” Proc. Natl. Acad. Sci. U. S. A., vol. 87, no. 23, pp. 9193–9196, 1990.
  4. D. Dua and C. Graff, “UCI Machine Learning Repository, ” University of California, Irvine, School of Information and Computer Sciences, 2019. [Online]. Available: https://archive.ics.uci.edu
  5. D. W. Hosmer, S. Lemeshow, and R. X. Sturdivant, Applied Logistic Regression, 3rd ed. Hoboken, NJ, USA: Wiley, 2013.
  6. C. Cortes and V. Vapnik, “Support-vector networks, ” Machine Learning, vol. 20, no. 3, pp. 273–297, 1995.
  7. L. Breiman, “Random forests, ” Machine Learning, vol. 45, no. 1, pp. 5–32, 2001.
  8. M. Fernández-Delgado, E. Cernadas, S. Barro, and D. Amorim, “Do we need hundreds of classifiers to solve real world classification problems?, ” J. Mach. Learn. Res., vol. 15, pp. 3133–3181, 2014.
  9. R. Caruana and A. Niculescu-Mizil, “An empirical comparison of supervised learning algorithms, ” in Proc. 23rd Int. Conf. Machine Learning (ICML), 2006, pp. 161–168.
  10. D. H. Wolpert, “The lack of a priori distinctions between learning algorithms, ” Neural Computation, vol. 8, no. 7, pp. 1341–1390, 1996.
  11. J. Demšar, “Statistical comparisons of classifiers over multiple data sets, ” J. Mach. Learn. Res., vol. 7, pp. 1–30, 2006.
  12. C. Nadeau and Y. Bengio, “Inference for the generalization error, ” Machine Learning, vol. 52, no. 3, pp. 239–281, 2003.
  13. R. R. Bouckaert and E. Frank, “Evaluating the replicability of significance tests for comparing learning algorithms, ” in Advances in Knowledge Discovery and Data Mining (PAKDD 2004), LNCS, vol. 3056. Berlin, Germany: Springer, 2004, pp. 3–12.
  14. P. Chapman et al., CRISP-DM 1.0: Step-by-Step Data Mining Guide. SPSS Inc., 2000.
  15. F. Pedregosa et al., “Scikit-learn: Machine learning in Python, ” J. Mach. Learn. Res., vol. 12, pp. 2825–2830, 2011.
  16. R. Kohavi, “A study of cross-validation and bootstrap for accuracy estimation and model selection, ” in Proc. 14th Int. Joint Conf. Artificial Intelligence (IJCAI), vol. 2, 1995, pp. 1137–1143.
  17. F. Wilcoxon, “Individual comparisons by ranking methods, ” Biometrics Bulletin, vol. 1, no. 6, pp. 80–83, 1945.
  18. C. Strobl, A.-L. Boulesteix, T. Kneib, T. Augustin, and A. Zeileis, “Conditional variable importance for random forests, ” BMC Bioinformatics, vol. 9, art. 307, 2008.
  19. S. S. Das et al., “Comparative Evaluation of Machine Learning Classifiers for Breast Cancer Diagnosis: A Comprehensive Statistical Analysis, ” Engineering Proceedings, vol. 124, no. 1, art. 35, 2026, doi: 10.3390/engproc2026124035
  20. J.-J. Jung et al., “Development and External Validation of a Machine Learning Model to Predict Pathological Complete Response After Neoadjuvant Chemotherapy in Breast Cancer, ” Journal of Breast Cancer, vol. 26, no. 4, pp. 353–362, 2023, doi: 10.4048/jbc.2023.26.e14
  21. T. Saito and M. Rehmsmeier, “The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets, ” PLoS ONE, vol. 10, no. 3, p. e0118432, 2015.
34 5

Similar Articles

1-10 of 15

You may also start an advanced similarity search for this article.