Analysis of the performance of different classifiers in solving the credit risk scoring problem with noisy and clean data

Subject Areas : Mathematical Optimization

Reza Yousefi Zenouz ¹ , Fatemeh Atapour Mashhad ²

1 - Kharazmi University
2 - Information Technology, Faculty of Management, Kharazmi Univerity

Received: 2017-02-15 Accepted : 2023-07-28 Published : 2022-09-01

Keywords: multiple classifier, Credit Risk Scoring, Noisy Data, Ensembles,

Abstract :

Credit risk scoring is an important issue in the real business world. There are many different classifiers available to address this issue, but the determinant factor in evaluating a classifier's performance is its accuracy level. This factor becomes even more crucial when dealing with noisy data. In this paper, we aim to utilize a unique approach to construct 11 new classifiers by combining various non-parametric methods. These new classifiers complement each other's shortcomings, leading to improved performance and subsequently, higher accuracy rates. Enhancing classifier performance directly impacts the profits of banks and financial institutions that utilize them. This study also investigates the question of whether the number of classifiers is more effective or if their accuracy levels are more critical in building combined classifiers. The single classifiers used in these combinations include K-nearest neighbor, Support Vector Machine, Multi-layer Perceptron, and Decision Tree. Furthermore, all the combined classifiers are compared with two types of ensembles in terms of accuracy rate and robustness at different noise levels. We utilized a standard dataset from the UCI database for our analysis. All results have been compared using the Wilcoxon signed rank test. The findings indicate that ensembles, especially RobustBoost, outperform both multiples and single classifiers in terms of both accuracy rate and robustness. Additionally, multiple classifiers are generally better than single ones. Moreover, the results highlight that in building multiple classifiers, the accuracy rate and robustness of each building block are more important than the number of components used.

References:

[1] Ali, K. M., & Pazzani, M. J. (1996). Error reduction through learning multiple descriptions. Mach. Learn., 24(3), 173-202. doi: 10.1023/a:1018249309965

[2] Baesens, B., Gestel, T.V., Viaene, S., Stepanova, M., Suykens, J., Vanthienen, J. (2003a). Benchmarking state-of-the-art classification algorithms for credit scoring. . Journal of the Operational Research Society, 54, 627–635.

[3] Bekhet, H. A., & Eletter, S. F. K. (2014). Credit risk assessment model for Jordanian commercial banks: Neural scoring approach. Review of Development Finance, 4(1), 20-28. doi: http://dx.doi.org/10.1016/j.rdf.2014.03.002

[4] Bian, S., & Wang, W. (2007). On diversity and accuracy of homogeneous and heterogeneous ensembles. Int. J. Hybrid Intell. Syst., 4(2), 103-128.

[5] C. Cortes, V. V. (1995). Support vector networks. Machine Learning 20 273–297.

[6] Crook, J. N., Edelman, D.B., Thomas, L.C. (2007). Recent developments in consumer credit risk assessment. . European Journal of Operational Research 183, 1447–1465.

[7] Dietterich, T. (2000). An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization. Machine Learning, 40(2), 139-157. doi: 10.1023/A:1007607513941
[8] Durand, D. (1941). Risk elements in consumer instalment financing. National Bureau of Economic Research, New York.
[9] Finlay, S. (2011). Multiple classifier architectures and their application to credit risk assessment. European Journal of Operational Research, 210(2), 368-378. doi: http://dx.doi.org/10.1016/j.ejor.2010.09.029
[10] Freund, Y. (2009). A more robust boostingalgorithm. . arXiv:0905.2138v1.
[11] Freund.Y and Schapire.E, pages, 148ñ156, & 1996. (1996). Experiments with a new boosting algorithm. . Machine Learning: Proceedings of the Thirteenth International Conference, 148_156.
[12] Hand, D. J., Henley, W.E. (1997). Statistical classification methods in consumer credit scoring: a review. Journal of the Royal Statistical Society, Series A-Statistics in Society 160, 523–541.
[13] Harris, T. (2013). Quantitative credit risk assessment using support vector machines: Broad versus Narrow default definitions. Expert Syst. Appl., 40(11), 4404-4413. doi: 10.1016/j.eswa.2013.01.044
[14] Harris, T. (2015). Credit scoring using the clustered support vector machine. Expert Syst. Appl., 42(2), 741-750. doi:
10.1016/j.eswa.2014.08.029
[15] Hens, A. B., & Tiwari, M. K. (2012). Computational time reduction for credit scoring: An integrated approach based on support vector machine and stratified sampling method. Expert Syst. Appl., 39(8), 6774-6781. doi: 10.1016/j.eswa.2011.12.057
[16] Huang, C.-l., Chen, M.-c., & Wang, C.-j. (2007). Credit scoring with a data mining approach based on support vector machines. Expert Systems with Applications, 33(4), 847-856. doi: 10.1016/j.eswa.2006.07.007
[17] Huang, Z., Chen, H., Hsu, C.-J., Chen, W.-H., & Wu, S. (2004). Credit rating analysis with support vector machines and neural networks: a market comparative study. Decis. Support Syst., 37(4), 543-558. doi: 10.1016/s0167-9236(03)00086-1
[18] Huysmans, J., Baesens, B., Vanthienen, J. (2005). A Comprehensible SOM-Based Scoring System. . Machine Learning and Data Mining in Pattern Recognition, 3587, 80–89.
[19] Kao, L.-J., Chiu, C.-C., & Chiu, F.-Y. (2012). A Bayesian latent variable model with classification and regression tree approach for behavior and credit scoring. Knowledge-Based Systems, 36(0), 245-252. doi: http://dx.doi.org/10.1016/j.knosys.2012.07.004
[20] Lee, T.-S., Chiu, C.-C., Chou, Y.-C., & Lu, C.-J. (2006). Mining the customer credit using classification and regression tree and multivariate adaptive regression splines. Computational Statistics & Data Analysis, 50(4), 1113-1130. doi:
http://dx.doi.org/10.1016/j.csda.2004.11.006
[21] Lee, T. S., & Chen, I. F. . (2005). A two-stage hybrid credit scoring model using artificial neural networks and multivariate adaptive regression splines. . Expert Systems with Applications, 28, 743-752.
[22] Maclin, R., & Opitz, D. (1997). An empirical evaluation of bagging and boosting. Paper presented at the Proceedings of the fourteenth national conference on artificial intelligence and ninth conference on Innovative applications of artificial intelligence, Providence, Rhode Island.
[23] Maktabi, H., Tohidi, H. & Seyedaliakbar, S.M. (2011). An Application of Linear Programming for Efficient Resource Allocation Case Study of University Education. Australian Journal of Basic and Applied Sciences, 5(12): 703-706
[24] Mar, #237, Cubiles-De-La-Vega, a.-D., Blanco-Oliver, A., Pino-Mej, R., #237, . . . Lara-Rubio, J. (2013). Improving the managementof microfinance institutions by using credit scoring models based on Statistical Learning techniques. Expert Syst. Appl., 40(17), 6910-6917. doi: 10.1016/j.eswa.2013.06.031
[25] Marqu, A. I., #233, Garc, V., #237, S, J. S., #225, & nchez. (2012). Two-level classifier ensembles for credit risk assessment. Expert Syst. Appl., 39(12), 10916-10922. doi: 10.1016/j.eswa.2012.03.033
[26] Martínez Sánchez, J. F., & Pérez Lechuga, G. (2016). Assessment of a credit scoring system for popular bank savings and credit. Contaduría y Administración, 61(2), 391-417. doi: http://dx.doi.org/10.1016/j.cya.2015.11.004
[27] McLachlan, G. J. (2004). , Discriminant Analysis and Statistical Pattern Recognition. John Wiley and Sons.
[28] Myers, J. H., Forgy, E.W. (1963). The development of numerical credit evaluation systems. . Journal of the American StatisticalAssociation 50, 799–806.
[29] P. Martens , S. A., H. Maud and R. Mohsin (2010). Is globalization healthy: a statistical indicator analysis of the impacts of globalization on health. Globalization and Health.

[30] Sáez, J. A., Galar, M., Luengo, J., & Herrera, F. (2013). Tackling the problem of classification with noisy data using Multiple Classifier Systems: Analysis of the performance and robustness. Information Sciences, 247, 1-20.
[31] Serrano-Cinca, C., Bego, #241, Guti, A., #233, & Rrez-Nieto. (2013). Partial Least Square Discriminant Analysis for bankruptcyprediction. Decis. Support Syst., 54(3), 1245-1255. doi: 10.1016/j.dss.2012.11.015
[32] Tohidi, H., Jabbari, M.M., (2012). “Decision role in management to increase effectiveness of an organization”. Procedia-social and behavioral sciences, 32: 825-828.
[33] Twala, B. (2010). Multiple classifier application to credit risk assessment. Expert Syst. Appl., 37(4), 3326-3336. doi:
10.1016/j.eswa.2009.10.018
[34] Wang, G., Hao, J., Ma, J., & Jiang, H. (2011). A comparative assessment of ensemble learning for credit scoring. Expert Syst. Appl., 38(1), 223-230. doi: 10.1016/j.eswa.2010.06.048
[35] West, D. (2000). Neural network credit scoring models. Comput. Oper. Res., 27(11-12), 1131-1152. doi: 10.1016/s0305-0548(99)00149-5
[36] West, D., Dellana, S., & Qian, J. (2005). Neural network ensemble strategies for financial decision applications. Comput. Oper. Res., 32(10), 2543-2559. doi: 10.1016/j.cor.2004.03.017
[37] Wu, Z. (2007). Introduction to Network Analysis of Microwave Circuits Software VNA and Microwave Network Design and Characterisation (pp. 1-37): John Wiley & Sons, Ltd.
[38] Ziegler, J. K. n. A. A. S. n. A. G. A. n. A. A. (2013). Consumer credit risk: Individual probability estimates using machine learning. Expert Systems with Applications.