Credit risk scoring is an important issue in the real business world. There are many different classifiers available to address this issue, but the determinant factor in evaluating a classifier's performance is its accuracy level. This factor becomes even more crucial when dealing with noisy data. In this paper, we aim to utilize a unique approach to construct 11 new classifiers by combining various non-parametric methods. These new classifiers complement each other's shortcomings, leading to improved performance and subsequently, higher accuracy rates. Enhancing classifier performance directly impacts the profits of banks and financial institutions that utilize them. This study also investigates the question of whether the number of classifiers is more effective or if their accuracy levels are more critical in building combined classifiers. The single classifiers used in these combinations include K-nearest neighbor, Support Vector Machine, Multi-layer Perceptron, and Decision Tree. Furthermore, all the combined classifiers are compared with two types of ensembles in terms of accuracy rate and robustness at different noise levels. We utilized a standard dataset from the UCI database for our analysis. All results have been compared using the Wilcoxon signed rank test. The findings indicate that ensembles, especially RobustBoost, outperform both multiples and single classifiers in terms of both accuracy rate and robustness. Additionally, multiple classifiers are generally better than single ones. Moreover, the results highlight that in building multiple classifiers, the accuracy rate and robustness of each building block are more important than the number of components used. Manuscript profile