زمانبندی کارها در محاسبات ابری با استفاده از یادگیری عمیق تقویتی
محورهای موضوعی : پردازش چند رسانه ای، سیستمهای ارتباطی، سیستمهای هوشمند
بهنام سالاری
1
,
مهدی اکبری
2
,
فرامرز صافی
3
,
بهرنگ برکتین
4
1 - دانشجوی دکتری، گروه مهندسی کامپیوتر، واحد نجف‌آباد، دانشگاه آزاد اسلامی، نجف‌آباد ، ایران
2 - استادیار، گروه مهندسی کامپیوتر، واحد نجف‌آباد، دانشگاه آزاد اسلامی، نجف‌آباد ، ایران
3 - دانشیار، گروه مهندسی کامپیوتر، واحد نجف‌آباد، دانشگاه آزاد اسلامی، نجف‌آباد ، ایران
4 - دانشیار، گروه مهندسی کامپیوتر، واحد نجف‌آباد، دانشگاه آزاد اسلامی، نجف‌آباد ، ایران
کلید واژه: زمانبندی کارها, محاسبات ابری, یادگیری عمیق تقویتی, فرآیند تصمیم گیری مارکوف ,
چکیده مقاله :
زمانبندی وظایف و مصرف انرژی از جمله مسائل مهم در محاسبات ابری هستند. عدم استفاده از یک رویکرد زمانبندی مناسب در محاسبات ابری ممکن است باعث مصرف انرژی بالا و بهره‌وری پایین منابع شود. به علت پویایی و محدودیت منابع ابری برای اجرای درخواست‌های متنوع و متغیر با زمان کاربران، یک مکانیزم زمانبندی مؤثر مورد نیاز است که خود را با شرایط پویای سیستم وفق دهد. یادگیری عمیق، یک روش کاربردی برای مقابله با مشکلات مدیریت منابع رایانش ابری به عنوان یک ایده نوآورانه در سال‌های اخیر مورد توجه قرار گرفته است. از جمله الگوریتمهای یادگیری عمیق استفاده شده در این زمینه می‌توان از الگوریتمهای DQL ، ERDQL و DRL-LSTM نام برد که در دیتاستهای بزرگ به نتایج قابل قبولی رسیدهاند. ولی این الگوریتم‌ها دارای مشکلاتی از جمله تخمین غیر واقعی پاداش، آموزش طولانی مدت، نمونه برداری تصادفی از حافظه و عدم استفاده مناسب از فضای حالت می‌باشند که بر نتایج حاصل از استفاده آنها در مسائل بهینه سازی مؤثر است. در این تحقیق یک الگوریتم یادگیری عیمق تقویتی بهبود یافته برای زمانبندی کارها در محاسبات ابری ارائه شده است تا مشکلات تخمین غیر واقعی پاداش، آموزش طولانی مدت و نمونه برداری تصادفی از حافظه را بهبود دهد. همچنین ساختار فرآیند تصمیمگیری مارکوف در قسمت مدلسازی به گونهای تعریف میشود که مشکلات عدم استفاده مناسب از فضای حالت را تعدیل نماید. لذا با استفاده از الگوریتم پیشنهادی نتایج مورد انتظار این است که منجر به بهبود زمان پاسخ، زمان انتظار، برون‌ده، زمان‌کل کارها و تعداد نقض SLA گردد. در این مقاله، الگوریتم توسعه‌یافته با مجموعه داده (GOCJ (Google Cloud Jobs مورد ارزیابی قرار می‌گیرد.
Introduction: Task scheduling and energy consumption are important issues in cloud computing. Failure to use an appropriate scheduling approach in cloud computing may result in high energy consumption and low resource efficiency. Due to the dynamics and limitations of cloud resources to execute diverse and time-varying requests from users, an effective scheduling mechanism is required that adapts to dynamic system conditions. Deep learning, an applied method for dealing with cloud computing resource management problems, has been considered as an innovative idea in recent years. Among the deep learning algorithms used in this field, we can mention DQL, ERDQL, and DRL-LSTM algorithms, which have achieved acceptable results in large datasets. However, these algorithms have problems such as unrealistic reward estimation, long-term training, random sampling of memory, and lack of proper use of state space, which affect the results obtained from their use in optimization problems.
Method: In this research, an improved deep reinforcement learning algorithm for task scheduling in cloud computing is proposed to improve the problems of unrealistic reward estimation, long-term training, and random sampling from memory. Also, the Markov decision process structure in the modeling part is defined in such a way that it can mitigate the problems of not using the state space properly.
Results: The proposed algorithm has been evaluated with two scenarios including a synthetic dataset and GOCJ (Google Cloud Jobs) dataset. The results of the evaluations in the simulated environment show that the proposed algorithm has achieved better results in less time in all evaluated parameters.
Discussion: In this paper, an improved algorithm based on PERDQN is presented. Then, with a proper definition of the cloud environment based on the Markov decision process, the proposed algorithm is used to schedule tasks on virtual machines in cloud computing. The results show that using the proposed algorithm, response time, waiting time, throughput, MakeSpan and the number of SLA violations are improved, which is more significant on a larger scale.
[1] A. Keivani and J.-R. Tapamo, "Task scheduling in cloud computing: A review," in 2019 International Conference on Advances in Big Data, Computing and Data Communication Systems (icABCD), 2019: IEEE, pp. 1-6.
[2] N. K. Gondhi and A. Gupta, "Survey on machine learning based scheduling in cloud computing," in Proceedings of the 2017 International Conference on Intelligent Systems, Metaheuristics & Swarm Intelligence, 2017, pp. 57-61.
[3] A. Arunarani, D. Manjula, and V. Sugumaran, "Task scheduling techniques in cloud computing: A literature survey," Future Generation Computer Systems, vol. 91, pp. 407-415, 2019.
[4] C. Shetty, H. Sarojadevi, and S. Prabhu, "Machine learning approach to select optimal task scheduling algorithm in cloud," Turkish Journal of Computer and Mathematics Education (TURCOMAT), vol. 12, no. 6, pp. 2565-2580, 2021.
[5] J. Zhao, M. A. Rodríguez, and R. Buyya, "A deep reinforcement learning approach to resource management in hybrid clouds harnessing renewable energy and task scheduling," in 2021 IEEE 14th International Conference on Cloud Computing (CLOUD), 2021: IEEE, pp. 240-249.
[6] X. Wang et al., "Dynamic scheduling of tasks in cloud manufacturing with multi-agent reinforcement learning," Journal of Manufacturing Systems, vol. 65, pp. 130-145, 2022.
[7] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, "A brief survey of deep reinforcement learning," arXiv preprint arXiv:1708.05866, 2017.
[8] Y. Wei, L. Pan, S. Liu, L. Wu, and X. Meng, "Drl-scheduling: An intelligent qos-aware job scheduling framework for applications in clouds," IEEE Access, vol. 6, pp. 55112-55125, 2018.
[9] Z. Saadati And M. Zarei, "Solving the Multi-Objective Problem of IoT Service Placement in Fog Computing Using Reinforcement Learning Approaches," Intelligent Multimedia Processing and Communication Systems(IMPCS), pp. 29-41, 4(3) , 2023.
[10] L. Zhang, C. Yang, Y. Yan, and Y. Hu, "Distributed Real-Time Scheduling in Cloud Manufacturing by Deep Reinforcement Learning," IEEE Transactions on Industrial Informatics, vol. 18, no. 12, pp. 8999-9007, 2022.
[11] X. Wang, L. Zhang, Y. Liu, C. Zhao, and K. Wang, "Solving task scheduling problems in cloud manufacturing via attention mechanism and deep reinforcement learning," Journal of Manufacturing Systems, vol. 65, pp. 452-468, 2022.
[12] K. Siddesha, G. Jayaramaiah, and C. Singh, "A novel deep reinforcement learning scheme for task scheduling in cloud computing," Cluster Computing, vol. 25, no. 6, pp. 4171-4188, 2022.
[13] M. Naeem, S. T. H. Rizvi, and A. Coronato, "A gentle introduction to reinforcement learning and its application in different fields," IEEE access, vol. 8, pp. 209320-209344, 2020.
[14] J. Liu, F. Gao, and X. Luo, "Survey of deep reinforcement learning based on value function and policy gradient," Chinese Journal of Computers, vol. 42, no. 6, pp. 1406-1438, 2019.
[15] K. Sharma and S. Tripathy, "DEEP REINFORCEMENT LEARNING: A SURVEY."
[16] W. Yuan, Y. Li, H. Zhuang, C. Wang, and M. Yang, "Prioritized experience replay-based deep q learning: Multiple-reward architecture for highway driving decision making," IEEE Robotics & Automation Magazine, vol. 28, no. 4, pp. 21-31, 2021.
[17] W. Fedus et al., "Revisiting fundamentals of experience replay," in International Conference on Machine Learning, 2020: PMLR, pp. 3061-3071.
[18] J. Fan, Z. Wang, Y. Xie, and Z. Yang, "A theoretical analysis of deep Q-learning," in Learning for dynamics and control, 2020: PMLR, pp. 486-489.
[19] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, "Prioritized experience replay," arXiv preprint arXiv:1511.05952, 2015.
[20] H. Zhang, C. Qu, J. Zhang, and J. Li, "Self-adaptive priority correction for prioritized experience replay," Applied sciences, vol. 10, no. 19, p. 6925, 2020.
[21] Z. Tong, H. Chen, X. Deng, K. Li, and K. Li, "A scheduling scheme in the cloud computing environment using deep Q-learning," Information Sciences, vol. 512, pp. 1170-1191, 2020.
[22] T. Zheng, J. Wan, J. Zhang, and C. Jiang, "Deep reinforcement learning-based workload scheduling for edge computing," Journal of Cloud Computing, vol. 11, no. 1, p. 3, 2022.
[23] Y. Feng and F. Liu, "Resource Management in Cloud Computing Using Deep Reinforcement Learning: A Survey," in Proceedings of the 10th Chinese Society of Aeronautics and Astronautics Youth Forum, 2023: Springer, pp. 635-643.
[24] S. Swarup, E. M. Shakshuki, and A. Yasar, "Task Scheduling in Cloud Using Deep Reinforcement Learning," Procedia Computer Science, vol. 184, pp. 42-51, 2021.
[25] G. Rjoub, J. Bentahar, O. Abdel Wahab, and A. Saleh Bataineh, "Deep and reinforcement learning for automated task scheduling in large‐scale cloud computing systems," Concurrency and Computation: Practice and Experience, p. e5919, 2020.
[26] M. Roknaldini And E. Noroozi, "Presenting A Hybrid Method of Deep Neural Networks to Prevent Intrusion in Computer Networks," Intelligent Multimedia Processing and Communication Systems(IMPCS), 4(4), pp. 57-65, 2023.
[27] H. Che, Z. Bai, R. Zuo, and H. Li, "A deep reinforcement learning approach to the optimization of data center task scheduling," Complexity, vol. 2020, 2020.
[28] Z. Chen, J. Hu, G. Min, C. Luo, and T. El-Ghazawi, "Adaptive and efficient resource allocation in cloud datacenters using actor-critic deep reinforcement learning," IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 8, pp. 1911-1923, 2021.
[29] K. Li, Z. Peng, D. Cui, and Q. Li, "SLA-DQTS: SLA Constrained Adaptive Online Task Scheduling Based on DDQN in Cloud Computing," Applied Sciences, vol. 11, no. 20, p. 9360, 2021.
[30] J. Lin, D. Cui, Z. Peng, Q. Li, and J. He, "A Two-Stage Framework for the Multi-User Multi-Data Center Job Scheduling and Resource Allocation," IEEE Access, vol. 8, pp. 197863-197874, 2020.
[31] M. Cheng, J. Li, and S. Nazarian, "DRL-cloud: Deep reinforcement learning-based resource provisioning and task scheduling for cloud service providers," in 2018 23rd Asia and South pacific design automation conference (ASP-DAC), 2018: IEEE, pp. 129-134.
[32] B. Ning, F. H. T. Lin, and S. Jaimungal, "Double deep q-learning for optimal execution," Applied Mathematical Finance, vol. 28, no. 4, pp. 361-380, 2021.
[33] H. Van Hasselt, A. Guez, and D. Silver, "Deep reinforcement learning with double q-learning," in Proceedings of the AAAI conference on artificial intelligence, 2016, vol. 30, no. 1.
[34] Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas, "Dueling network architectures for deep reinforcement learning," in International conference on machine learning, 2016: PMLR, pp. 1995-2003.
[35] R. Liu and J. Zou, "The effects of memory replay in reinforcement learning," in 2018 56th annual allerton conference on communication, control, and computing (Allerton), 2018: IEEE, pp. 478-485.
[36] W. Zaremba and I. Sutskever, "Reinforcement learning neural turing machines-revised," arXiv preprint arXiv:1505.00521, 2015.
[37] Y. Li, "Deep reinforcement learning: An overview," arXiv preprint arXiv:1701.07274, 2017.
[38] D. Fährmann, N. Jorek, N. Damer, F. Kirchbuchner, and A. Kuijper, "Double deep q-learning with prioritized experience replay for anomaly detection in smart environments," IEEE Access, vol. 10, pp. 60836-60848, 2022.
[39] T. Goyal, A. Singh, and A. Agrawal, "Cloudsim: simulator for cloud computing infrastructure and modeling," Procedia Engineering, vol. 38, pp. 3566-3572, 2012.
[40] K. Ramasubramanian and A. Singh, "Deep learning using keras and tensorflow," in Machine Learning Using R: Springer, 2019, pp. 667-688.
[41] A. Hussain and M. Aleem, "GoCJ: Google cloud jobs dataset for distributed and cloud computing infrastructures," Data, vol. 3, no. 4, p. 38, 2018.
[42] P. Amini and A. Kalbasi, "An Adaptive Task Scheduling Approach for Cloud Computing Using Deep Reinforcement Learning," in 2024 Third International Conference on Distributed Computing and High Performance Computing (DCHPC), 2024: IEEE, pp. 1-9.
[43] S. Mangalampalli, G. R. Karri, M. Kumar, O. I. Khalaf, C. A. T. Romero, and G. A. Sahib, "DRLBTSA: Deep reinforcement learning based task-scheduling algorithm in cloud computing," Multimedia Tools and Applications, vol. 83, no. 3, pp. 8359-8387, 2024.
[44] S. Mangalampalli et al., "Multi Objective Prioritized workflow scheduling using Deep reinforcement based Learning in Cloud Computing," IEEE Access, 2024.
[45] S. Mangalampalli, G. R. Karri, and U. Kose, "Multi Objective Trust aware task scheduling algorithm in cloud computing using Whale Optimization," Journal of King Saud University-Computer and Information Sciences, vol. 35, no. 2, pp. 791-809, 2023.
[46] A. R. Kadhim and F. Rabee, "Deadline and Cost Aware Dynamic Task Scheduling in Cloud Computing Based on Stackelberg Game," International Journal of Intelligent Engineering & Systems, vol. 16, no. 3, 2023.
