Improving the Accuracy of Segmentation of Remote Sensing Images using Deep Learning
محورهای موضوعی : Majlesi Journal of Telecommunication DevicesAdel__Hamdy__Dhayef Adel__Hamdy__Dhayef 1 , Mehran Emadi 2
1 - Department of Computer Engineering, Isfahan (Khorasgan) branch, Islamic Azad University Isfahan, Iran
2 - Department of Electrical Engineering, Mobarakeh Branch, Mobarakeh, Isfahan, Iran
کلید واژه: Segmentation, Hyper Spectral Images, Remote Sensing, Morphological Operators, Convolution Neural Networks,
چکیده مقاله :
Image segmentation is used to exploit remote sensing images with high resolution. The purpose of segmentation is to create different segments with common features. For example, residential areas, forests, rivers and other areas are obtained in subdivision. But the random position of different areas on the ground has caused the accuracy of segmentation methods to be low. Using deep learning methods can improve segmentation accuracy. In this paper, a SegNet convolution deep neural network is proposed for segmentation of high resolution (HR) remote sensing images. The proposed strategy is to improve the semantic segmentation performance of images. The proposed SegNets strategy is carried out in two steps. The proposed method has been evaluated with accuracy criteria and F1 score. The results show that the accuracy is improved by more than 4% compared to other methods based on deep learning. Also, other evaluation criteria such as ROC have been used. The results of this criterion also show the superiority of this proposed method.
Image segmentation is used to exploit remote sensing images with high resolution. The purpose of segmentation is to create different segments with common features. For example, residential areas, forests, rivers and other areas are obtained in subdivision. But the random position of different areas on the ground has caused the accuracy of segmentation methods to be low. Using deep learning methods can improve segmentation accuracy. In this paper, a SegNet convolution deep neural network is proposed for segmentation of high resolution (HR) remote sensing images. The proposed strategy is to improve the semantic segmentation performance of images. The proposed SegNets strategy is carried out in two steps. The proposed method has been evaluated with accuracy criteria and F1 score. The results show that the accuracy is improved by more than 4% compared to other methods based on deep learning. Also, other evaluation criteria such as ROC have been used. The results of this criterion also show the superiority of this proposed method.
Improving the Accuracy of Segmentation of Remote Sensing Images using Deep Learning
Adel Hamdy Dhayef1, Mehran Emadi2
Email: adeleng975@gmail.com
2- Department of Electrical Engineering, Mobarakeh Branch, Mobarakeh, Isfahan, Iran.
Email: emadi.mehran49@gmail.com (Corresponding author)
ABSTRACT: Image segmentation is used to exploit remote sensing images with high resolution. The purpose of segmentation is to create different segments with common features. For example, residential areas, forests, rivers and other areas are obtained in subdivision. But the random position of different areas on the ground has caused the accuracy of segmentation methods to be low. Using deep learning methods can improve segmentation accuracy. In this paper, a SegNet convolution deep neural network is proposed for segmentation of high resolution (HR) remote sensing images. The proposed strategy is to improve the semantic segmentation performance of images. The proposed SegNets strategy is carried out in two steps. The proposed method has been evaluated with accuracy criteria and F1 score. The results show that the accuracy is improved by more than 4% compared to other methods based on deep learning. Also, other evaluation criteria such as ROC have been used. The results of this criterion also show the superiority of this proposed method.
KEYWORDS: Segmentation, Hyper Spectral Images, Remote Sensing, Morphological Operators, Convolution Neural Networks.
|
1. iNTRODUCTION
In general, remote sensing (RS) is the technology of collecting information and taking pictures from the surface of the earth using aviation equipment such as airplanes, balloons or space equipment such as satellites[1, 2]. In other words, remote sensing is the science and art of obtaining information about any subject under investigation by a tool that is not in physical contact with it. The superior advantage of satellite information over other information sources is their repeated coverage of certain areas with a certain time interval. In remote sensing, information transmission is done using electromagnetic radiation (EMR) [3]. The data obtained from hyper spectral images in RS systems have been available to researchers since the early 1980s, and their use indicates the maturity of the technology. For the first time, the Hyperion probe was used by NASA in November 2000 to test the capability of an airborne vehicle. Airborne and space borne hyper spectral sensors (HS) sensors have many applications as one of the powerful and advanced tools in geology, agriculture and geography studies [4-7]. The use of this technology also began in the mid-80s, and the current advantages of remote sensing data and geographic information led to the development of this technology [8]. The preparation of hyper spectral imaging (HSI) is naturally more difficult and expensive than multispectral imaging (MSI) and this is due to the current advantages of these data (its high signal-to-noise ratio indicates the spectrum High quality data, as well as its spectral coverage and large number of channels, cause a very high spectral separation power, these data are generally a combination of 100 to 200 spectral bands (channels) with a thin band width between 5-10 nm, in While the data obtained from MS sensors are in the receiver of 5 to 10 channels with a relatively wider bandwidth (between 70-400 nm)[9]. (This technology is almost new and is used by researchers and scientists for mineral identification. plants, vegetation and artificial materials are used[10]). Identifying and distinguishing the existing components and categorizing hyper spectral images in order to use these images optimally and with higher efficiency is one of the important issues for the analysis and investigation of hyper spectral images[11]. Content analysis of satellite images is important in order to use these images in urban planning, agriculture, forestry, as well as management of underground resources and other applications[12].Using satellite images, there is a greater need for automatic identification of land cover types, including roads, rivers, forests, and other natural and non-natural effects. which is one of the common methods of using segmentation methods of these images [13]. Segmentation of satellite images is a hot and active topic of remote sensing due to the complexity of complications and the spatial and spectral heterogeneity of the urban and non-urban environment (man-made and natural), which has a large variety of artificial and natural surfaces[14-16]. Segmentation is widely used for mapping purposes. Accurate mapping of urban land use is used for various applications, including information on land use, urban management, change detection, urban planning and design, and environmental monitoring. For this reason, it is very important to have accurate and timely information about the status and changes in urban areas [17]. Various segmentation techniques have been presented for satellite images [18].In the method presented in [19], the classification of hyper spectral images was done using the reduction of spatial spectral dimensions. This method is presented to overcome the challenges of a large number of spectral bands and data heterogeneity to improve classification. Classification of hyper spectral images was done with SVM and guided filter. In the method [20], considering that the combination of spatial features has been widely used, this method was used to extract spatial features in spectral-spatial using directed filter to classify hyper spectral images. has been A method for classifying hyper spectral images is presented in [21]. Researchers, in this research, presented a new spectral, spatial classification approach based on texture pattern separation for hyper spectral image classification. In [22], a method based on super pixels based on KNN nearest neighbor classifier is presented. In [23], a new method of local binary pattern (LBP) based on super pixel decision-making is proposed for the classification of hyper spectral images. In this method [24], open morphological transformations are used to separate bright (open) structures in the images, where bright means brighter than the surrounding features in the images. In [25], morphological summaries with subjective retrieval and guided MPs are first investigated for over-decision hyper spectral snapshot classification of city areas. Second, a supervised face extraction is developed to reduce the dimensionality of the morphological profiles created for prediction. In [26]proposed a joint hyper spectral and infrared image classification framework based on threshold-based local contain profile, where TLCP is a new design to suppress interference in spatial extractions. In [27], he used deep learning to quickly identify fires and predict possible spread for effective response in suppression. In [28], the most challenging problem of land use extraction in images with medium spatial resolution has been used using deep learning semantic segmentation. The random position, the completely random form of natural and human-made effects in satellite images has caused that the presented solutions cannot be effective in all applications. On the other hand, the existence of noise in these images is inevitable. In the existing methods for the analysis of hyper spectral images, uncertainty management has not been done properly. Uncertainty caused by noise, measurement error, and inaccurate content of images has a great impact on reducing the accuracy of identification and classification results of hyper spectral images [29]. It seems that it is necessary to provide useful, efficient and new methods in this field. Various methods have been proposed for image segmentation, which are mainly divided into five categories: thresholding, clustering, edge detection, region extraction, and feature extraction [30]. Since the existence of imprecise content in the images collected by remote sensing is unavoidable, it seems that the use of deep learning will be very efficient to remove the uncertainty and analyze the images [31]. Through the studies, it was found that although many studies and researches have been presented for the segmentation of satellite images, there is still a long way to go before reaching a favorable answer in segmentation with high accuracy. According to the studies conducted in the review of the research literature and the background of the research, it is clear that one of the most important challenges of this research is the similarity of the textures on the ground, the randomness of the position and shape of the textures and complications on the ground. The capabilities of fuzzy methods can be a way to overcome these challenges. Therefore, in this research, we have used a deep learning based method along with morphological operators to segment hyper spectral images. Deep learning can segment images with an acceptable accuracy by overcoming the limitations of uncertainty. In the following, this article is divided as follows. In the second part, deep learning is introduced. In the third part, the proposed method is presented with a block diagram. In the fourth part, the proposed method is evaluated. Finally, the conclusion of the article is presented in the fifth section.
2. DEEP NEURAL NETWORKS
Deep neural networks are multi-layered structures. These networks are designed to strengthen and remove the limitations of the two-layer neural network. They have two or more layered structures, which are called intermediate layers of the hidden layer. The change in the number of neurons in each layer and the number of hidden layers causes a change in the performance of the neural network. Next, a structure of a neural network is shown in figure (1).
Fig. 1.Structure of a perceptron multi-layer neural network.
Since the output of each neuron is used as the input of the next neuron, the components of the neuron labeled j with the input of pj(t) from the previous neurons include the following:
· The activation of the neuron state (aj(t)) depends on a selected location.
· If a function is constant, then a threshold is constant.
· The activation function whose goal is to activate a new one at a certain time ((t+1) of θj) with a net input pj(t).
· The activation function whose goal is to activate a new one at a certain time ((t+1) of θj) with a net input pj(t)oj(t+1)=fout(aj(t).
It uses local for effective extraction of spatial information and joint weights to significantly reduce the number of parameters. Nowadays, in order to remote sensing image analysis, an unsupervised motion network has been suggested. To formulate a deep CNN model, unsupervised layer pre-training is used in this method. In compared to the unsupervised method, the supervised CNN may extract more effective features by using of class-specific information, which can be provided by the training examples [32].
2.1. Deep convolution neural network
One of the most popular deep learning models that specialize in spatial information discovery are Convolution neural networks. In this section, the working mechanism of CNN is briefly introduced. CNNs are widely used to discover latent spatial information in applications such as image recognition, ubiquity, and object search due to their salient features such as regular structure, good localization, and translation invariance. In BCI, in particular, CNN is supposed to capture distinct dependencies among patterns associated with different brain signals. A standard CNN architecture is shown in Figure 2. CNN consists of an input layer, two convolution layers with each integration layer, a fully connected layer and an output layer. A square patch in each layer shows the processing progress of a particular set of input values. The key to CNN is to reduce the input data in a way that is easier to recognize and with as little loss of information as possible.
Fig. 2. convolution neural network.
CNN has three dense layers: convolution layer, pooled layer and fully connected layer. The convolution layer is the main block of CNN, which consists of a series of filters for de convolution of the input data, followed by a non-linear transformation to extract geographic features. In implementing deep learning, there are several key parameters that need to be set in the convolution layer, such as the number of l, the size of each l, etc. The pooling layer generally follows the convolution layer. The purpose of the fusion layer is to gradually reduce the spatial size of the features. In this way, it can help reduce the number of parameters (eg, weight and basis) and computational burden. There are three types of integration operations: maximum, minimum, average. Take max integration as an example. The merge operation outputs the maximum value of the merge region in the result. Hyper parameters in the integration layer include integration operations, integration region size, steps, etc. In a fully connected layer, such as a basic neural network, nodes are fully connected to all previous activations. To extract spectral and spatial information of hyper spectral data simultaneously, it is reasonable to formulate 3D CNN. Furthermore, to solve the problem of over fitting caused by limited training samples of hyper spectral data, we design a regularization strategy, including modified linear unit (ReLU) and dropout to achieve better generalization of the model. A complete CNN stage consists of a convolution layer and an aggregation layer. A deep CNN is built by stacking multiple convolution layers and combining the layers to create a deep architecture. First, the convolution layer is introduced. The value of neuron vxij at position x of the jth feature map in the ith layer is shown below.
(1) |
|
Convolution Layers |
|
|
|
Pooling Layer |
|
|
|
Fractional-Stride Convolution Layer |
|
|
|
Soft max Layer |
|
|
|
(a)
(b)
Fig. 4. Schematic of (a) SegNet, (b) SegNet-Basic architectures.
The steps of the proposed method are explained in the following steps:
i. In the first step, the original image is applied to the grid to divide it into two classes. In the convolution neural network section, six convolution stages and ReLu activation function, six integration stages and three sampling stages are used. In this part, the image enters the convolution neural network and the output of the specified area is black and white binary images, in other words, the output of this network is shown in number 6. Black pixels in number 7 correspond to the first class, white pixels correspond to the second class. Dividing into two classes produces sharper edges compared to dividing into six classes at once. As explained, the selection of three subclasses of the first and second class is practically investigated. Based on extensive practical testing, building, tree and vegetation pixels are first class. And so, the road, the car, and the mess are left for second class. SegNet-Basic weights were used for more accurate segmentation in the next mode.
ii. In the second step, the first class is divided into three subclasses by another network. In this step, 32 x 32 packets are sampled from the local neighborhood in the output database of satellite images in the previous step. The sample images are fed to SegNet-Basic, which has 3x3 filters and three dense layers with weights of 256, 128, and 2. There are also two Soft max because there is a fixed feature in the transfer in pool layers, which are also pool layers. used in this research. which has a batch size of 128 and cross entropy function. 1- The ReLu activity function is used to initialize the weights, which have a Gaussian distribution with zero mean. The training rate is also selected with an initial value of 0.0001. In order to avoid overtraining, this description has been increased to 0.3. The maximum number of courses is 1000. The rule with the least error is considered as the source model. This network will be trained with patch model. Also, for the final segmentation, the density layers are changed to their convolution equivalent and the loss function and regularization method are used to update the rules and weights. The output of this step, as shown in Figure 8, contains three tags: building, tree, low vegetation and some black pixels. Here, the black pixels belong to the second class, and the network does not decide which subclasses they belong to, i.e., the unknown pixels are black.
iii. The third step divides the pixels of the second class that are labeled into three subclasses.
4. evaluation
Segmentation and zoning of hyper spectral images in remote and satellite sensing has always faced many challenges. The random shape and position of land features, including man-made or natural features, the low contrast of these images has caused several methods to be presented to identify this important issue. In this research, an efficient and at the same time simple method based on two-stage SegNet deep learning is presented. In this article, the accuracy criterion is used to evaluate the proposed segmentation. Relationship 5-4 shows this important.
|
Accuracy obtained (average of cases) | Method | Reference |
83.10 | Watershed + SVM method | [33] |
62.08 | Validated segmentation method | [34] |
73.20 | Clustering method (SVM) | [35] |
97.98 | Deep Auto encoder method | [36] |
91.50 | Region expansion method | [37] |
88.00 | DBN method | [38] |
73.22 | Active Learning method | [39] |
86.33 | Recurrent Neural Networks method | [40] |
92.44 | The proposed method in this research | -- |
As can be seen in Table 1, the suggested approach does not show the best output compared to other existing methods. But the accuracy obtained shows that this method is widely accepted and has a performance close to strong methods such as deep networks and recurrent networks.
REFERENCES
[1] Chakraborty, R., R. Sushil, and M.L. Garg, Hyper-spectral image segmentation using an improved PSO aided with multilevel fuzzy entropy. Multimedia Tools and Applications, 2019. 78(23): p. 34027-34063.
[2] Emadi, M., Z. Jafarian Dehkordi, and M. Iranpour Mobarakeh, Improving the Accuracy of Brain Tumor Identification in Magnetic Resonanceaging using Super-pixel and Fast Primal Dual Algorithm. International Journal of Engineering, 2023. 36(3): p. 505-512.
[3] Sathiyamoorthi, V., et al., An effective model for predicting agricultural crop yield on remote sensing hyper‐spectral images using adaptive logistic regression classifier. Concurrency and Computation: Practice and Experience: p. e7242.
[4] Abbasi, A., et al., A meta-analysis of factors related to fertility attitudes, desires, and childbearing intentions in Iranian studies. Interdisciplinary Studies in Humanities, 2022. 14(4): p. 63-92.
[5] Harouni, M., M. Karimi, and S. Rafieipour, Precise segmentation techniques in various medical images. Artificial Intelligence and Internet of Things: Applications in Smart Healthcare, 2021. 117.
[6] Karimi, E., A. Ebrahimi, and M.R. Tavakoli, How optimal PMU placement can mitigate cascading outages blackouts? International Transactions on Electrical Energy Systems, 2019. 29(6): p. e12015.
[7] Karimi, M., et al., Automatic lung infection segmentation of covid-19 in CT scan images, in Intelligent Computing Applications for COVID-19. 2021, CRC Press. p. 235-253.
[8] Mahmudi, F., M. Soleimani, and M. Naderi, Some Properties of the Maximal Graph of a Commutative Ring. Southeast Asian Bulletin of Mathematics, 2019. 43(4).
[9] Srinivas, B. and J.R. Prasad, Enhanced Segmentation Algorithm for Hyper-spectral Imaging (HSI).
[10] Harouni, M., et al., Health monitoring methods in heart diseases based on data mining approach: A directional review, in Prognostic models in healthcare: Ai and statistical approaches. 2022, Springer. p. 115-159.
[11] .Karimi, M., et al., Improving monitoring and controlling parameters for alzheimer’s patients based on iomt, in Prognostic models in healthcare: Ai and statistical approaches. 2022, Springer. p. 213-237.
[12] Jia, J., et al., Review on active and passive remote sensing techniques for road extraction. Remote Sensing, 2021. 13(21): p. 4235.
[13] Karimi, M., M. Harouni, and S. Rafieipour, Automated medical image analysis in digital mammography, in Artificial intelligence and internet of things. 2021, CRC Press. p. 85-116.
[14] Navabifar, F. and M. Emadi, A Fusion Approach Based on HOG and Adaboost Algorithm for Face Detection under Low-Resolution Images. INTERNATIONAL ARAB JOURNAL OF INFORMATION TECHNOLOGY, 2022. 19(5): p. 728-735.
[15] Rehman, A., et al., Microscopic retinal blood vessels detection and segmentation using support vector machine and K‐nearest neighbors. Microscopy research and technique, 2022. 85(5): p. 1899-1914.
[16] Soleimani, M., M.H. Naderi, and A.R. Ashrafi, TENSOR PRODUCT OF THE POWER GRAPHS OF SOME FINITE RINGS. Facta Universitatis, Series: Mathematics and Informatics, 2019: p. 101-122.
[17] Moshayedi, A.J., et al., E-Nose design and structures from statistical analysis to application in robotic: a compressive review. EAI Endorsed Transactions on AI and Robotics, 2023. 2(1): p. e1-e1.
[18] Shafique, A., et al., Deep learning-based change detection in remote sensing images: a review. Remote Sensing, 2022. 14(4): p. 871.
[19] High-level hyperspectral image classification based on spectro-spatial dimensionality reduction. Spatial Statistics, 2016: p. 103-117.
[20] Hyperspectral image classification with SVM. Wireless Communications and Networking 2019: p. 1-9.
[21] Texture Pattern Separation for Hyperspectral. Selected Topics in Applied Earth Observations and Remote Sensing 2019: p. 3602 - 3614.
[22] KNN-Based Representation of Superpixels for Hyperspectral Image Classification. SELECTED TOPICS IN APPLIED EARTH OBSERVATIONS AND REMOTE SENSING, 2018: p. 1-16.
[23] Local Binary Pattern-Based Hyperspectral Image Classification With Superpixel Guidance. TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, 2018. 56(2): p. 749-759.
[24] Huang, W., et al., Local binary patterns and superpixel-based multiple kernels for hyperspectral image classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020. 13: p. 4550-4563.
[25] Jia, S., et al., Local binary pattern-based hyperspectral image classification with superpixel guidance. IEEE Transactions on Geoscience and Remote Sensing, 2017. 56(2): p. 749-759.
[26] Li, W., et al., Feature extraction for hyperspectral images using local contain profile. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019. 12(12): p. 5035-5046.
[27] Sun, C. Analyzing Multispectral Satellite Imagery of South American Wildfires Using Deep Learning. in 2022 International Conference on Applied Artificial Intelligence (ICAPAI). 2022. IEEE.
[28] Boonpook, W., et al., Deep Learning Semantic Segmentation for Land Use and Land Cover Types Using Landsat 8 Imagery. ISPRS International Journal of Geo-Information, 2023. 12(1): p. 14.
[29] Kaul, A., A Fundamental Review on Hyperspectral Segmentation Algorithms, in Applications of Networks, Sensors and Autonomous Systems Analytics. 2022, Springer. p. 165-185.
[30] Balafar, M.A., et al., Review of brain MRI image segmentation methods. Artificial Intelligence Review, 2010. 33(3): p. 261-274.
[31] Soleimani, M., F. Mahmudi, and M. Naderi, Some results on the maximal graph of commutative rings. Advanced Studies: Euro-Tbilisi Mathematical Journal, 2023. 16(supp1): p. 21-26.
[32] Chen, Y., et al., Deep feature extraction and classification of hyperspectral images based on convolutional neural networks. IEEE Transactions on Geoscience and Remote Sensing, 2016. 54(10): p. 6232-6251.
[33] Tarabalka, Y., J. Chanussot, and J.A. Benediktsson, Segmentation and classification of hyperspectral images using watershed transformation. Pattern Recognition, 2010. 43(7): p. 2367-2379.
[34] Nalepa, J., M. Myller, and M. Kawulok, Validating hyperspectral image segmentation. IEEE Geoscience and Remote Sensing Letters, 2019. 16(8): p. 1264-1268.
[35] Bilgin, G., S. Erturk, and T. Yildirim, Segmentation of hyperspectral images via subtractive clustering and cluster validation using one-class support vector machines. IEEE Transactions on Geoscience and Remote sensing, 2011. 49(8): p. 2936-2944.
[36] Chen, Y., et al., Deep learning-based classification of hyperspectral data. IEEE Journal of Selected topics in applied earth observations and remote sensing, 2014. 7(6): p. 2094-2107.
[37] Li, F., et al., ST-IRGS: A region-based self-training algorithm applied to hyperspectral image classification and segmentation. IEEE Transactions on Geoscience and Remote Sensing, 2017. 56(1): p. 3-16.
[38] Li, T., J. Zhang, and Y. Zhang. Classification of hyperspectral image based on deep belief networks. in 2014 IEEE international conference on image processing (ICIP). 2014. IEEE.
[39] Huck, J.J., et al., Centaur VGI: A Hybrid Human–Machine Approach to Address Global Inequalities in Map Coverage. Annals of the American Association of Geographers, 2021. 111(1): p. 231-251.
[40] Shah Heydari, S., Large Area Land Cover Mapping Using Deep Neural Networks and Landsat Time-Series Observations. 2021.
Paper type: Research paper