Integrated noise reduction-data mining method for soil organic matter prediction by VNIR spectrometry

Subject Areas : Agriculture, rangeland, watershed and forestry

Elahe Akbari ¹ , Saham Mirzaei ² , Ara Toomanian ³ , Ali Darvishi Boloorani ⁴ , Hosseinali Bahrami ⁵

1 - Assistant Professor, Department of Remote Sensing and GIS, Faculty of Geography and Environmental Sciences, Hakim Sabzevari University, Sabzevar, Iran
2 - PhD. Student of Remote Sensing and Geographical Information System, Faculty of Geography, University of Tehran, Iran
3 - Associate Professor, Department of Remote Sensing and Geographical Information System, Faculty of Geography, University of Tehran, Iran
4 - Associate Professor, Department of Remote Sensing and Geographical Information System, Faculty of Geography, University of Tehran, Iran
5 - Professor, Department of Soil Science, Faculty of Agriculture, Tarbiat Modares University, Iran

Received: 2021-08-23 Accepted : 2021-09-04 Published : 2022-09-23

Keywords: BRT, Southwest of Tehran, Spectroscopy, Soil organic matter, PLSR,

Abstract :

Background and Objective Soil as a heterogeneous natural resource and the largest organic carbon storage in terrestrial ecosystems is composed of complicated processes and mechanisms. The necessity of accurately estimating soil properties on the national and regional scales for improving soil management, and understanding their influence on agriculture have resulted in attracting researchers’ attentions to this field. Soil Organic Matter (SOM) is considered as an indicator of soil quality in fertility and food production. It is also considered as a key variable in environmental and agricultural issues. Thus, using rapid and cost effective and more accuracy estimation of the SOM content in soil resources assessment and management can be helpful. In precision agriculture, the scale of soil data required for management of lands and products is very large. The scale of collecting filed data usually cannot fulfil those needs. Sampling, preparing and analyzing the large number of soil samples as well as producing the distribution map for large areas are very difficult. In addition, traditional laboratory methods of soil analysis are boring, time-consuming, and costly. In fact, they need specialized laboratory operators. The aim of the present study is to compare the performance of the two Partial Least Squares Regression (PLSR) and Boosted Regression Tree (BRT) for predicting SOM using VNIR spectrometry data. With the use of combining Wavelet transform and diagnosis of independent bands, noises existing in soil spectroscopic data has reduced. In addition, independent and effective spectra and bands in spectroscopy of SOM were selected. Consequently, in the present research, Wavelet-PCA-PLSR and Wavelet-PCA- BRT models were developed and performance were assessed.Materials and Methods 42 surface (0-30cm) soil samples in the heterogeneous areas of urban-agricultural regions in Tehran province were collected. Soil Organic Carbon (OC) measured using Walki Black method and the samples’ spectrums were measured by ASD FieldSpec-3 spectrometer. First and second derivitation of spectral reflectance and absorbance were calculated. To reduce noises and smooth the spectrum, Sym8 matrix function of wavelet transform was used, wavelet transform is conducted to show and reconstruct characteristics in the spectrum. Principal component analysis and Hotelling's T2 test with 95% confidence level were used for outlier detection. PLSR and BRT was conducted onreflectance, absorbance and their first and second derivatives, at five levels of wavelet transform. Then, by comparing the results, the appropriate model was selected via validation. For doing the PLSR in nonlinear data, Kernel functions were used. When using numerical samples, regression trees are used instead of decision trees. But their processes are the same. In regression trees, the greedy algorithm was used. Therefore, by answering the binary question through which node the maximum data about respons variable is obtained, the root node and its two children are obtained. Producing the structure of trees is recursively repeated and a typical stopping criterion is considered. The stopping criterion can be as achievement to a split which cannot be divided and provides fewer data, or when data in the node contain 5% of the total data. Moreover, the tree size should be minimized. For splitting the node, the Ginny factor, entropy factor, etc. were used for minimizing those factors. In addition, the total square error is calculated in each branches and those with minimized values are selected. In addition, in the regression tree, the pruning process is employed for over-fitting. The BRT consists of the two regression tree and boosting techniques for improving the predictability of each of them. For calibration and validation of the model, 30 and 12 soil samples were randomly selected, respectively and R2 and RMSE were used for quantify the accuracy of models. Moreover, to select the best production factor of the PLSR mode, explained variance residual values and RMSE of validation were considered. Finally, soil organic matter map was produced using Landsat OLI satellite imagery and the proofed method for the study area.Results and Discussion The SOM value acceptably, the creation of continuous mappings with more accuracy based on noise reduction and retention of suitable data have always received researchers’ attentions. The present study tried to find the better method such a more accurate quantization of SOM using soil spectroscopic data. Using wavelet transform and outlier removal based on Hotelling's T2 via the PCA, the suitable data were extracted for producing the more accurate quantization. In this method, independent and effective bands or spectra remain in the model, while Lin et al. used wavelet transform and correlation techniques for selecting appropriate bands in estimating SOM. Since the soil reflectance is more complex and affected by several factors, using correlation method in these heterogeneous areas such as the area studied in the present study does not lead to acceptable results. Considering the data values, the unsupervised PCA method calculates principle components and eigenvalues and eigenvectors. It also tries to maximize the covariance matrix based on Singular Value Decomposition (SVD). SOM estimation models were developed using the PLSR and BRT for reflectance and absurbance spectra and their first and second derivation. Based on the results, the BRT method with RMSE and R2 values as 0.58 and 0.94, respectively leads in the better results for the data of the second derivation of reflectance. Moreover, values of RMSE and R2 in the PLSR were obtained as 1.0338 and 0.938, respectively for the data related to the second derivation of reflectance. However, comparing RMSE of the BRT and PLSR shows better results of the BRT model.Conclusion In that field measurements of chemical properties of soil such as organic matters are critically time-consuming and costly. Furthermore, measuring those properties is not possible in the large samples. So, the results of the present study indicate that in heterogeneous agricultural-urban areas, potential of the developed models such as wavelet-PCA-PLSR and wavelet-PCA-BRT can be used for estimating SOM. Meanwhile, these two algorithms do not make distributional assumptions and therefore, there are no strong assumptions about normality. Using continuous functions and satellite imagery, the map of the level of SOM in large scales can be prepared in order that it can be utilized in studies such as cultivation potential, soil fertility, and sustainable development of soil.

References:

Alavipanah S.K, Damavandi A.A, Mirzaie S, Rezaie A, Matinfar H.R, Hamzeh S, Teymori H, Javad Zarrin I. 2016. Remote sensing application in evaluation of soil characteristics in desert areas. Natural Environment Change, 2(1): 1-24.

Attaeian B, Shojaeefar S, Zandieh V, Hashemi S.S. 2018. Study of soil organic carbon changes in two critical and vulnerable areas of Qahavand plain rangelands using remote sensing and GIS. RS & GIS for Natural Resources, 8(4): 76-90. (In Persian).

Dai F, Zhou Q, Lv Z, Wang X, Liu G. 2014. Spatial prediction of soil organic matter content integrating artificial neural network and ordinary kriging in Tibetan Plateau. Ecological Indicators, 45: 184-194. doi: https://doi.org/10.1016/j.ecolind.2014.04.003.

Doetterl S, Stevens A, Van Oost K, Quine T.A, Van Wesemael B. 2013. Spatially-explicit regional-scale prediction of soil organic carbon stocks in cropland using environmental variables and mixed model approaches. Geoderma, 204: 31-42. doi:https://doi.org/10.1016/j.geoderma.2013.04.007

Friedman J.H. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics: 1189-1232. doi:https://doi.org/10.1214/aos/1013203451.

Castaldi F, Palombo A, Pascucci S, Pignatti S, Santini F, Casa R. 2015. Reducing the Influence of Soil Moisture on the Estimation of Clay from Hyperspectral Data: A Case Study Using Simulated PRISMA Data. Remote Sensing, 7(11): 15561-15582. https://doi.org/10.3390/rs71115561.

Groenigen J.W, Mutters C.S, Horwath W.R, Van Kessel C. 2003. NIR and DRIFT-MIR spectrometry of soils for predicting soil and crop parameters in a flooded field. Plant and Soil, 250(1): 155-165. doi:https://doi.org/10.1023/A:1022893520315.

Hotelling H. 1933. Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24(6): 417-441. doi: 10.1037/h0071325.

Khanamani A, Jafari R, Sangoony H, Shahbazi A. 2011. Evaluation of soil status using RS and GIS technology (Case study: Segzi plain). Journal of Applied RS & GIS Techniques in Natural Resource Science, 2(3): 25-37. https://www.sid.ir/en/journal/ViewPaper.aspx?id=250690. (In Persian).

Kuang B, Tekin Y, Mouazen A.M. 2015. Comparison between artificial neural network and partial least squares for on-line visible and near infrared spectroscopy measurement of soil organic carbon, pH and clay content. Soil and Tillage Research, 146: 243-252. doi:https://doi.org/10.1016/j.still.2014.11.00.

Lacoste M, Minasny B, McBratney A, Michot D, Viaud V, Walter C. 2014. High resolution 3D mapping of soil organic carbon in a heterogeneous agricultural landscape. Geoderma, 213: 296-311. doi:https://doi.org/10.1016/j.geoderma.2013.07.002.

Liaghat S, Ehsani R, Mansor S, Shafri H.Z, Meon S, Sankaran S, Azam S.H. 2014. Early detection of basal stem rot disease (Ganoderma) in oil palms based on hyperspectral reflectance data using pattern recognition algorithms. International Journal of Remote Sensing, 35(10): 3427-3439. doi:https://doi.org/10.1080/01431161.2014.903353.

Lin L, Wang Y, Teng J, Wang X. 2016. Hyperspectral analysis of soil organic matter in coal mining regions using wavelets, correlations, and partial least squares regression. Environmental Monitoring and Assessment, 188(2): 1-11. doi:https://doi.org/10.1007/s10661-016-5107-8.

Liu L, Ji, M, Dong Y, Zhang R, Buchroithner M. 2016. Quantitative retrieval of organic soil properties from Visible Near-Infrared Shortwave Infrared (Vis-NIR-SWIR) spectroscopy using fractal-based feature extraction. Remote Sensing, 8(12): 1035. doi:https://doi.org/10.3390/rs8121035.

McCarty G.W, Reeves J.B, Reeves V.B, Follett R.F, Kimble J.M. 2002. Mid-infrared and near-infrared diffuse reflectance spectroscopy for soil carbon measurement. Soil Science Society of America Journal, 66(2): 640-646. doi:https://doi.org/10.1016/j.geoderma.2009.04.005.

Mirzaei S, Darvishi Boloorani A, Bahrami H.A, Alavipanah, S.K, Mousivand A. 2021. Moisture influence reducing on soil reflectance using EPO for organic carbon prediction. 7th International Conference on Agriculture, Environment, Urban and Rural. Tbilisi, Georgia. 16 June. https://civilica.com/doc/1256685. (In Persian).

Morellos A, Pantazi X.E, Moshou, D, Alexandridis T, Whetton R, Tziotzios G, Wiebensohn J, Bill R, Mouazen A.M. 2016. Machine learning based prediction of soil total nitrogen, organic carbon and moisture content by using VIS-NIR spectroscopy. Biosystems Engineering. doi:https://doi.org/10.1016/j.biosystemseng.2016.04.018.

Mouazen A.M, Kuang B, De Baerdemaeker J, Ramon H. 2010. Comparison among principal component, partial least squares and back propagation neural network analyses for accuracy of measurement of selected soil properties with visible and near infrared spectroscopy. Geoderma, 158(1): 23-31. doi:https://doi.org/10.1016/j.geoderma.2010.03.001.

Nawar S, Abdul Munnaf M, Mouazen A.M. 2020. Machine learning based on-line prediction of soil organic carbon after removal of soil moisture effect. Remote Sensing, 12(8): 1308. https://doi.org/10.3390/rs12081308.

Nocita M, Kooistra L, Bachmann M, Müller A, Powell M, Weel S. 2011. Predictions of soil surface and topsoil organic carbon content through the use of laboratory and field spectroscopy in the Albany Thicket Biome of Eastern Cape Province of South Africa. Geoderma, 167: 295-302. doi:https://doi.org/10.1016/j.geoderma.2011.09.018.

Ghazi M, Bahrami H.A, Darvishi Boloorani A, Mirzaei S. 2018. Estimating the measure of the soil’s lime in dust’s centers of Tehran province by using of VINR spectroscopy and satellite images of OLI. RS & GIS for Natural Resources, 8(4): 1-16, https://www.sid.ir/en/journal/ViewPaper.aspx?id=597225 (In Persian).

Steffens M, Kohlpaintner M, Buddenbaum H. 2014. Fine spatial resolution mapping of soil organic matter quality in a Histosol profile. European Journal of Soil Science, 65(6): 827-839. doi: https://doi.org/10.1111/ejss.12182.

Tekin Y, Kuang B, Mouazen A.M. 2013. Potential of on-line visible and near infrared spectroscopy for measurement of pH for deriving variable rate lime recommendations. Sensors, 13(8): 10177-10190. doi:https://doi.org/10.3390/s130810177.

Viscarra Rossel R.A, Behrens, T. 2010. Using data mining to model and interpret soil diffuse reflectance spectra. Geoderma, 158(1): 46-54. doi:https://doi.org/10.1016/j.geoderma.2009.12.025.

Viscarra Rossel R.A, Hicks W.S. 2015. Soil organic carbon and its fractions estimated by visible–near infrared transfer functions. European Journal of Soil Science, 66(3): 438-450. doi:https://doi.org/10.1111/ejss.12237.

Viscarra Rossel R.A, Cattle S.R, Ortega A, Fouad Y. 2009. In situ measurements of soil colour, mineral composition and clay content by vis–NIR spectroscopy. Geoderma, 150(3): 253-266. doi:https://doi.org/10.1016/j.geoderma.2009.01.025.

Vohland M, Besold J, Hill J, Fründ H.C. 2011. Comparing different multivariate calibration methods for the determination of soil organic carbon pools with visible to near infrared spectroscopy. Geoderma, 166(1): 198-205. doi:https://doi.org/10.1016/j.geoderma.2011.08.001.

Wang Y, Wang F, Huang J, Wang X, Liu Z. 2009. Validation of artificial neural network techniques in the estimation of nitrogen concentration in rape using canopy hyperspectral reflectance data. International Journal of Remote Sensing, 30(17): 4493-4505. doi:https://doi.org/10.1080/01431160802577998.

Yang H, Li J. 2013. Predictions of soil organic carbon using laboratory-based hyperspectral data in the northern Tianshan Mountains, China. Environmental Monitoring and Assessment, 185(5): 3897-3908. doi:https://doi.org/10.1007/s10661-012-2838-z.

Yang R.M, Zhang G.L, Liu F, Lu Y.Y, Yang F, Yang F, Yang M, Zhao Y.G, Li D.C. 2016. Comparison of boosted regression tree and random forest models for mapping topsoil organic carbon concentration in an alpine ecosystem. Ecological Indicators, 60: 870-878. doi:https://doi.org/10.1016/j.ecolind.2015.08.036.