에피소드 랜덤화 및 액션 노이즈를 통한 강화학습 기반의 포트폴리오 최적화 성능 향상 = Improving Portfolio Optimization Performance based on Reinforcement Learning through Episode Randomization and Action Noise

국문 초록 (Abstract)

포트폴리오 최적화는 투자 관리 위험을 감소시키고 수익을 극대화하기 위해 필수적이다. 최근 인공 지능 기술이 급격히 발달하면서 다양한 분야에서 이를 활용하기 위해 연구 중이며, 특히 금융 분야에서는 강화학습을 적용하기 위한 연구가 활발히 진행되고 있다. 그러나 대부분의 연구들이 과거 금융 데이터의 반복 학습으로 인한 에이전트 과적합 문제를 해결하지 못하고 있다. 이에 본 연구에서는 강화학습 기반의 포트폴리오 최적화에서 에피소드 랜덤화 및 액션 노이즈를 통해 과적합을 완화하는 기법을 제안한다. 제안된 기법은 에피소드마다 학습 데이터 기간을 랜덤화하여 다양한 시장 상황을 경험하게 함으로써 데이터 증폭의 효과와 액션 노이즈 기법을 활용하여 에이전트가 특정 상황에 대응할 수 있게 탐색을 촉진한다. 실험 결과 제안 기법을 적용하였을 때 기존 강화학습 에이전트보다 성능이 향상되었음을 확인할 수 있었으며 비교 실험을 통해 다양한 조건에서 제안하는 기법 모두 성능 향상에 기여하였음을 확인하였다.

다국어 초록 (Multilingual Abstract)

Portfolio optimization is essential to reduce investment management risk and maximize returns. With the rapid development of artificial intelligence technology in recent years, research is being conducted to utilize it in various fields, and in particular, investigation on the application of reinforcement learning in the financial sector. However, most studies do not address the problem of agent overfitting due to iterative training on historical financial data. In this study, we propose a technique to mitigate overfitting through episode randomization and action noise in reinforcement learning-based portfolio optimization. The proposed technique randomizes the duration of the training data in each episode to experience different market conditions, thus promoting the effectiveness of data augmentation and exploration by leveraging action noise techniques to allow the agent to respond to specific situations. Experimental results show that the proposed technique improves the performance of the existing reinforcement learning agent, and comparative experiments confirm that both techniques contribute to performance improvement under various conditions.

참고문헌 (Reference)

• 1 N. Baard, "Twin-Delayed Deep Deterministic Policy Gradient Algorithm for Portfolio Selection" 2022 : 1-8, 2022
• 2 William F. Sharpe, "The Sharpe Ratio" 21 (21): 49-58, 1994
• 3 Haarnoja, T., "Soft actor-critic algorithms and applications"
• 4 Haarnoja, T., "Soft Actor-Critic : Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor" 1861-1870, 2018
• 5 Brock, W., "Simple technical trading rules and the stochastic properties of stock returns" 47 (47): 1731-1764, 1992
• 6 Jegadeesh, N., "Returns to buying winners and selling losers : Implications for stock market efficiency" 48 (48): 65-91, 1993
• 7 Schulman, J., "Proximal Policy Optimization Algorithms"
• 8 T. van Zyl, "ParDen : Surrogate assisted hyper-parameter optimisation for portfolio selection" 101-107, 2021
• 9 Fortunato, M., "Noisy Networks for Exploration" 2018
• 10 Elton, E. J, "Modern Portfolio Theory, 1950 to Date" 21 (21): 1743-1759, 1995
• 11 Malik Magdon-Ismail, "Maximum drawdown" 17 (17): 99-102, 2004
• 12 Abdolmaleki, A., "Maximum a posteriori policy optimisation"
• 13 Liu, Xiao-Yang, "FinRL : Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance" 2021
• 14 Silver, D., "Deterministic policy gradient algorithms" 2014
• 15 J. B. Heaton, "Deep learning for finance : deep portfolios" 33 (33): 3-12, 2017
• 16 Lillicrap, T. P., "Continuous control with deep reinforcement learning" 2016
• 17 Eric Benhamou, "Bridging the gap between Markowitz planning and deep reinforcement learning" Social Science Research Network 2020
• 18 Raileanu, R., "Automatic Data Augmentation for Generalization in Reinforcement Learning" 34 : 2021
• 19 Mnih, V., "Asynchronous Methods for Deep Reinforcement Learning" 48 : 1928-1937, 2016
• 20 Hollenstein, J., "Action Noise in Off-Policy Deep Reinforcement Learning : Impact on Exploration and Performance" 2022
• 21 김성문 ; 김홍선, "A study on the investment performance of Markowitz's portfolio selection model using nonlinear programming in the Korean stock market" 26 (26): 19-35, 2009
• 22 윤성식 ; 이석준, "A study on stock portfolio construction and trading simulation using reinforcement learning" Korea Society of Management Information Systems 40 (40): 185-203, 2021
• 23 Kim, K., "A study on reinforcement learning-based stock trading system" 747-749, 2020
• 24 이우식, "A study on asset allocation strategy using SAC reinforcement learning algorithm" 22 (22): 367-373, 2022
• 25 A. B. Paskaramoorthy, "A framework for online investment decisions" 49 (49): 215-231, 2020