Dimension Reduction of Big Data and Deleting Noise and Its Efficiency in the Decision Tree Method and Its Use in Covid 19

Subject Areas : International Journal of Mathematical Modelling & Computations

Fazel Badakhshan Farahabadi ¹ , Kianoush Fathi Vajargah ² , Rahman Farnoosh ³

1 - Department of Statistics, Islamic Azad University, Science and Research Branch, Tehran, Iran
2 - Department of Statistics, Islamic Azad University, Tehran North Branch
3 - School of Mathematics, Iran University of Science and Technology, Tehran, 16844, Iran

Received: 2021-12-13 Accepted : 2022-02-10 Published : 2022-09-01

Keywords: C4.5, Copula Function, Decision tree, covid 19, Gaussian copula function (normal),

Abstract :

In today's world, with the advancement of science and technology, data is generated at high speeds, and with the increase in the size and volume of data, we often face a lot of extensions and redundant data and noise data that make the task of analysis difficult. Therefore, dimension reduction of the data without losing useful information in the data is very important to prepare the data for data mining and can increase the speed and even accuracy of the analysis. In this research, we present a dimensional reduction method using a copula function that reduces the dimensions of the data by identifying the relationships between the data. The copula function provides a good pattern of dependence for comparing multivariate distributions to better identify the relationship between data. In fact, by fitting the appropriate copula function to the data and estimating the copula function parameter, we measure the structural correlation of the variables and eliminate variables that are highly structurally correlated with each other. As a result, in the method presented in this study, using the copula function, we identify noise data and data with many common features and remove them from the original data.