Reinforcement learning for financial trading: algorithms, evaluations and platforms

• Main Author: Sun, Shuo
• Other Authors: Bo An
• Format: Thesis-Doctor of Philosophy
• Language: English
• Published: Nanyang Technological University 2025
• Subjects: Computer and Information Science
• Online Access: https://hdl.handle.net/10356/182059
• Institution: Nanyang Technological University

The financial markets, which involve more than $90 trillion market capitals, attract the attention of innumerable investors around the world. Due to the complexity and low signal-to-noise ratio of financial data, it is a challenging task to make profitable investment decisions. Recently, reinforcement learning (RL) has emerged as a promising direction to train agents for financial decision making. However, most existing techniques focuses on small-scale tasks and deliver a vulnerable solution without systematic evaluation. There is still a long way to go for deploying RL algorithms into real-world finance applications. This doctoral thesis is devoted to push the real-world deployment of RL in financial decision making (FinRL) from the perspectives of algorithms, evaluation benchmarks and software infrastructures, by leveraging the power of modern RL, ensemble learning and system design techniques. In particular, we tackle the following fundamental research problems and propose a solution for all of them. First, we study the problem of designing efficient FinRL algorithms under the competitive intraday trading scenarios to actively trade within the same day for profit arbitrage. We provide a formal definition and propose a risk-aware deep RL algorithm. A dueling Q-network with action branching is applied to deal with the large action space for RL optimization. Three key components, namely hindsight bonus, market embedding encoder-decoder and risk-aware auxiliary task, are introduced to further improve the performance. Extensive experiments on real-world financial datasets demonstrate the superior performance of the proposed algorithm comparing to state-of-the-art baselines. We then investigate the problem of adopting advanced ensemble learning and mixture-of-experts (MoE) techniques for robust decision making in finance. The initial idea is inspired by the real-world bottom-up trading strategy design workflow in the finance industry, where hierarchical decision-making is involved to combine the intelligence of junior traders and senior portfolio managers. We first propose a novel three-stage MoE framework for stock prediction. In stage one, an efficient ensemble learning method is designed to train multiple trading experts with tiny overhead of costs. In stage two, we build a pool of trading experts utilizing hyperparameter level and initialization level diversity of neural networks for post hoc ensemble construction. In stage three, we design three different mechanisms, namely as-needed router, with-replacement selection and integrated expert soup, to dynamically pick experts from the expert pool. Later on, we extend the idea into the FinRL setting by proposing a simple yet efficient RL algorithm. Risk-aware Bellman backup is introduced to reweight Q-values by adopting estimated uncertainty. To encourage strategy diversity, we leverage the idea of bootstrap with random initialization. Extensive experiments show that the two algorithms that leveraging the power of ensemble learning significantly outperforms other baselines with higher profit, lower risk and better diversity. Third, we focus on the systematic evaluation of FinRL algorithms. The motivation is that the evaluation of existing FinRL algorithms mostly focuses on profit-related measures and ignores many critical axes. We introduce a systematic evaluation benchmark with six axes, i.e., profitability, risk-control, universality, diversity, reliability, and explainability, and 17 measures from the literature of multiple disciplines (e.g., finance, artificial intelligence, statistics and engineering). We collect the implementation of evaluation metrics and visualization tools into an open-source library and further demonstrate its usage on four datasets with eight FinRL algorithms. Finally, we study the problem of FinRL software infrastructures. To address the engineering challenges and facilitate the development of new FinRL algorithms, we build a holistic open-source trading platform. The platform is composed of six key modules, which cover the whole pipeline including data preprocessing, market simulators implementation, algorithms design, evaluation toolkits and user interfaces. The platform offers the software infrastructures and relevant materials (e.g., documentation, tutorials) on RL based financial trading and has made great impact in the community with over 1.2k stars on Github.