확정적 네트워크에서의 동적 처리순위를 활용한 강화학습 기반 스케줄러 = Reinforcement Learning-Based Scheduler with Dynamic Precedence in Deterministic Networks

다국어 초록 (Multilingual Abstract)

Smart industry, metaverse, digital-twin, and military applications require deterministic data delivery in large scale networks. This paper proposes reinforcement learning-based scheduling that assigns dynamically different precedences to the flows, in addition to the flow's class or priority, and determines the scheduling algorithm according to the flow's precedence. In the proposed reinforcement learning-based scheduling algorithm with two precedence queues, the reinforcement learning agent takes two actions that assigns the precedence of flows according to a specified criterion and selects a scheduling algorithm. Depending on the purpose of the network, any factor with high importance could be a criterion for determining the precedence. In this study, the deadline required by the flow is designated as the major factor for precedence decision. By utilizing DDQN (Double Deep Q-Network), a deep learning-based reinforcement learning model, the precedence and the scheduling algorithm are determined by observing the state of the network and selecting an action at each decision period with a fixed length. In the network simulator developed for the study, it was confirmed that the DDQN agent showed better performance than various heuristic algorithms.

국문 초록 (Abstract)

스마트 인더스트리, 메타버스, 디지털 트윈, 군사용 어플리케이션 등에서 확정적 데이터 전달을 요구하고 있다.
본 논문은 일반적으로 통용되는 플로우들의 클래스 혹은 우선순위와는 별도로, 네트워크 상황과 중요도에 따라 플로우 별로 동적으로 처리순위(precedence)를 할당하고, 이에 따라 스케줄링 알고리즘을 결정하는 강화학습 기반의스케줄링 프레임워크를 제안한다. 이를 실증하기 위해서 두 개의 처리순위 큐가 존재하는 환경을 상정하여, 강화학습 에이전트가 지정된 기준에 따라 플로우들의 처리순위를 지정하며 스케줄링 알고리즘을 선택하는 두 가지의행동(action)을 취한다. 네트워크 특성에 따라 다양한 기준으로 처리순위를 결정할 수 있다. 본 연구에서는 플로우가 요구하는 마감기한(deadline)을 처리순위 결정의 중요한 기준으로 사용하였다. 딥러닝 기반의 강화학습 모델인DDQN(Double Deep Q-Network)을 활용하여, 고정된 길이의 결정 주기마다 네트워크의 상태(state)를 관측하고행동을 선택함으로써 처리순위를 결정한다. 본 연구의 환경에 맞게 개발한 네트워크 시뮬레이터를 통해 DDQN 에이전트가 여러 휴리스틱 알고리즘과 비교하여 높은 성능을 보이는 것을 확인하였다.

참고문헌 (Reference)

• 1 Y. Choi, "Ultra-high-precision network technology trend for ultra-immersive/highprecision service" 36 (36): 34-47, 2021
• 2 Xiong Xiong, "Resource Allocation Based on Deep Reinforcement Learning in IoT Edge Computing" Institute of Electrical and Electronics Engineers (IEEE) 38 (38): 1133-1146, 2020
• 3 S. Chilukuri, "RECCE : Deep reinforcement learning for joint routing and scheduling in time-constrained wireless networks" 9 : 132053-132063, 2021
• 4 V. Mnih, "Playing atari with deep reinforcement learning, arXiv preprint arXiv:1312.5602"
• 5 N. Finn, "Deterministic networking architecture, RFC 8655"
• 6 H. van Hasselt, "Deep reinforcement learning with double Q-Learning" 30 (30): 2094-2100, 2016
• 7 Tao Zheng, "Deep Reinforcement Learning-Based Workload Scheduling for Edge Computing" Springer Science and Business Media LLC 11 (11): 1-13, 2022
• 8 Chaoyun Zhang, "Deep Learning in Mobile and Wireless Networking: A Survey" Institute of Electrical and Electronics Engineers (IEEE) 21 (21): 2224-2287, 2019
• 9 류지혜 ; 권주혁 ; 정진우, "DDQN을 활용한 강화학습 기반 타임슬롯 스케줄링" 한국통신학회 47 (47): 944-952, 2022
• 10 Sandeep Chinchali, "Cellular Network Traffic Scheduling With Deep Reinforcement Learning" Association for the Advancement of Artificial Intelligence (AAAI) 32 (32): 2018
• 11 Shahram Mollahasani, "Actor-Critic Learning Based QoS-Aware Scheduler for Reconfigurable Wireless Networks" Institute of Electrical and Electronics Engineers (IEEE) 9 (9): 45-54, 2022
• 12 Dipak Ghosal, "A Reinforcement Learning Based Network Scheduler for Deadline-Driven Data Transfers" IEEE 1-6, 2019
• 13 Yong Xue, "A Framework for Military Precedence-Based Assured Services in GIG IP Networks" IEEE 2007