Enhancing Intrusion Detection Accuracy Using XGBoost, Deep Autoencoders, and LSTM
Subject Areas : Multimedia Processing, Communications Systems, Intelligent SystemsBehnam Dorostkar 1 , Zohreh Dorrani 2 , Hasan Afshar Afshien 3
1 - Department of Information and Communication Technology, Amin Police University, Tehran, Iran
2 - Department of Electrical Engineering, Payame- Noor University, Tehran, Iran
3 - Department of Information and Communication Technology, Amin Police University, Tehran, Iran
Keywords: ntrusion Detection System, Deep Autoencoder, LSTM, Feature Selection, Network Security, Anomaly Detection,
Abstract :
Intrusion detection systems face significant challenges in handling high-dimensional network data while maintaining detection accuracy. This paper proposes a novel hybrid framework integrating XGBoost, a deep autoencoder, and LSTM to address these limitations. Traditional methods often overlook the synergistic potential of feature selection, dimensionality reduction, and temporal pattern analysis, leading to suboptimal performance. Our approach begins with preprocessing raw network traffic data, including normalization and categorical encoding. XGBoost is employed for feature selection, identifying the top-k discriminative features to reduce computational overhead. A deep autoencoder then extracts compressed latent representations from the selected features, enhancing the model’s ability to capture nonlinear relationships. Finally, an LSTM network classifies sequences of these latent features, leveraging temporal dependencies for precise attack detection. Evaluated on the UNSW-NB15 and WBAN RSSI datasets, the proposed method achieves state-of-the-art accuracy of 89.25% and 84.00%, respectively, outperforming existing techniques such as standalone XGBoost (85.08–88.42%) and GRU-based models (80.52–88.13%). These results highlight the framework’s robustness in addressing high dimensionality and temporal dynamics, bridging critical gaps in IDS research. The method’s modular design ensures adaptability to diverse network environments, offering a scalable solution for real-time intrusion detection.
S. M. Kasongo, "A deep learning technique for intrusion detection system using a Recurrent Neural Networks based framework," Computer Communications, vol. 199, pp. 113-125, 2023. https://doi.org/10.1016/j.comcom.2022.12.010
[2] M. Roknaldini and E. Noroozi, "Presenting A Hybrid Method of Deep Neural Networks to Prevent Intrusion in Computer Networks," Intelligent Multimedia Processing and
Communication Systems (IMPCS), vol. 4, no. 4, pp. 57-65, 2023. https://doi.org/10.71856/impcs.2024.903654 [3] Z. Dorrani, "Optimization of Photonic Nanocrystals for Invisibility Using Artificial Intelligence," Journal of Advanced Materials in Engineering (Esteghlal), vol. 44, no. 1, pp.
55-70, 2024. https://doi.org/10.47176/jame.44.1.1088 [4] A. kaffash, S. R. K. T. Farizani, and M. Kheirabadi, "An Effective and Lightweight Intrusion Detection for IoT based on Fog and Cloud using KNN Classification," Intelligent
Multimedia Processing and Communication Systems (IMPCS), vol. 2, no. 5, pp. 55-64, 2024. https://doi.org/ 10.71856/impcs.2024.903638 [5] E. Hajian and N. Asadi, "Intrusion Detection Using Deep Learning in Wireless Body Area Networks," Intelligent Multimedia Processing and Communication Systems #(IMPCS), vol. 4, no. 5, pp. 1-13, 2024. https://doi.org/ 10.71856/impcs.2024.1195000
[6] L. Diana, P. Dini, and D. Paolini, "Overview on Intrusion Detection Systems for Computers Networking Security," Computers, vol. 14, no. 3, p. 87, 2025. https://doi.org/
[7] Z. Dorrani, H. Farsi, and S. Mohammadzadeh, "Edge Detection and Identification using Deep Learning to Identify Vehicles," Journal of Information Systems and Telecommunication (JIST), vol. 3, no. 39, p. 201, 2022. https://doi.org/ 10.52547/jist.16385.10.39.201
[8] Z. Dorrani, "Road Detection with Deep Learning in Satellite Images," Majlesi Journal of Telecommunication Devices, vol. 12, no. 1, pp. 43-47, 2023. https://doi.org/10.30486/mjtd.2023.1979006.1024
[9] W. Chimphlee and S. Chimphlee, "Hyperparameters optimization XGBoost for network intrusion detection using CSE-CICIDS 2018 dataset," IAES International Journal of Artificial Intelligence, vol. 13, no. 1, pp. 817-826, 2024. https://doi.org/ 10.11591/ijai.v13.i1.pp817-826
[10] L. Riazi, "Hybrid Intrusion Detection System: Leveraging XGBoost, Genetic Algorithms, and k-Means Clustering," Intelligent Multimedia Processing and Communication Systems (IMPCS), vol. 2, no. 5, pp. 71-80, 2024. https://doi.org/ 140305151128271
[11] Y. P. khah, M. H. Shirvani, and H. Motameni, "Metaheuristic Algorithms for Feature Selection in Intrusion Detection Systems: A Systematic Review," Intelligent Multimedia Processing and Communication Systems (IMPCS), vol. 3, no. 4, pp. 63-92, 2023. https://doi.org/ 10.71856/impcs.2023.1123897
[12] R. H. Dong, X. Y. Li, Q. Y. Zhang, and H. Yuan, "Network intrusion detection model based on multivariate correlation analysis–long short‐time memory network," IET Information Security, vol. 14, no. 2, pp. 166-174, 2020. https://doi.org/10.1049/iet-ifs.2019.0294
[13] C. Yin, Y. Zhu, J. Fei, and X. He, "A deep learning approach for intrusion detection using recurrent neural networks," Ieee Access, vol. 5, pp. 21954-21961, 2017. https://doi.org/ 10.1109/ACCESS.2017.2762418
[14] S. M. Kasongo and Y. Sun, "A deep learning method with wrapper based feature extraction for wireless intrusion detection system," Computers & Security, vol. 92, p. 101752, 2020. https://doi.org/10.1016/j.cose.2020.101752
[15] C.-M. Hsu, H.-Y. Hsieh, S. W. Prakosa, M. Z. Azhari, and J.-S. Leu, "Using long-short-term memory based convolutional neural networks for network intrusion detection," in Wireless Internet: 11th EAI International Conference, WiCON 2018, Taipei, Taiwan, October 15-16, 2018, Proceedings 11, 2019, pp. 86-94: Springer. https://doi.org/ 10.1007/978-3-030-06158-6_9
[16] S. M. Notghimoghadam, H. Farsi, and S. Mohamadzadeh, "Object Detection by a Hybrid of Feature Pyramid and Deep Neural Networks," Journal of Electrical and Computer Engineering Innovations (JECEI), vol. 11, no. 1, pp. 173-182, 2023. https://doi.org/10.22061/jecei.2022.9012.567
[17] H. Farsi, s. m. notghimoghadam, s. Mohamadzadeh, and A. Barati, "Development of a Deep Learning Model Inspired by Transformer Networks For Multi-class Skin Lesion Classification," International Journal of Engineering, pp. -, 2025. https://doi.org/ 10.5829/ije.2026.39.01a.11
[18] J. P. Barbin and M. Jalali, "Improving the Accuracy of the Intrusion Detection System in the IoT by Machine Learning and Clustering Algorithms," Intelligent Multimedia Processing and Communication Systems (IMPCS), vol. 3, no. 5, pp. 47-54, 2024. https://doi.org/ 10.71856/impcs.2024.903654
[19] https://www.cityscapes-dataset.com/examples/.
[20] Z. Dorrani, "Traffic Scene Analysis and Classification using Deep Learning," International Journal of Engineering, vol. 37, no. 3, pp. 496-502, 2024. https://doi.org/ 10.5829/ije.2024.37.03c.06
[21] Z. Dorrani, H. Farsi, and S. Mohamadzadeh, "Shadow Removal in Vehicle Detection Using ResUNet-a," Iranica Journal of Energy & Environment, vol. 14, no. 1, pp. 87-95, 2023. https://doi.org/ 10.5829/ijee.2023.14.01.11
[22] Z. Dorrani, H. Farsi, and S. Mohamadzadeh, "Deep Learning in Vehicle Detection Using ResUNet-a Architecture," Jordan Journal of Electrical Engineering. All rights reserved-Volume, vol. 8, no. 2, p. 166, 2022. https://doi.org/ 10.5455/jjee.204-1638861465
[23] Z. Dorrani, "Speech-to-text with artificial intelligence: Improving accuracy using Fuzzy ResNet and SqueezeNet," Journal of Information and Communication Technology in Policing, pp. -, 2025. https://doi.org/ 10.22034/pitc.2025.1282622.1344
[24] https://www.kaggle.com/datasets/mrwellsdavid/unsw-nb15.
[25] R. Z. Farkhady, K. Majidzadeh, M. Masdari, and A. Ghaffari, "The Integrated Three-dimensional Deep Learning Approach for an Efficient Intrusion Detection System Using Spatiol-Temporal Features," Intelligent Multimedia Processing and Communication Systems (IMPCS), vol. 3, no. 5, pp. 69-86, 2024. https://doi.org/
[26] H. SabbaghGol, H. Saadatfar, and M. Khazaiepoor, "Predicting Alzheimer's disease: A machine learning approach using advanced feature selection techniques," Journal of Modern Medical Information Sciences, vol. 10, no. 3, pp. 307-324, 2024. https://doi.org/ 10.48312/jmis.10.3.904.1
[27] M. Khodadadi, L. Riazi, and S. Yazdani, "A Novel Ensemble Deep Learning Model for Building Energy Consumption Forecast," International Journal of Engineering, vol. 37, no. 6, pp. 1067-1075, 2024. https://doi.org/ 10.5829/ije.2024.37.06c.03
[28] https://www.kaggle.com/datasets/mehdimka/ble-rssi-dataset.
[29] M. Rohani, H. Farsi, and S. Mohamadzadeh, "Deep Multi-task Convolutional Neural Networks for Efficient Classification of Face Attributes," International Journal of Engineering, vol. 36, no. 11, pp. 2102-2111, 2023. https://doi.org/ 10.5829/ije.2023.36.11b.14
