EarnHFT: Efficient hierarchical reinforcement learning for high frequency trading

High-frequency trading (HFT) is using computer algorithms to make trading decisions in short time scales (e.g., second-level), which is widely used in the Cryptocurrency (Crypto) market, (e.g., Bitcoin). Reinforcement learning (RL) in financial research has shown stellar performance on many quantita...

Full description

Saved in:
Bibliographic Details
Main Authors: QIN, Molei, SUN, Shuo, ZHANG, Wentao, XIA, Haochong, WANG, Xinrun, AN, Bo
Format: text
Language:English
Published: Institutional Knowledge at Singapore Management University 2024
Subjects:
Online Access:https://ink.library.smu.edu.sg/sis_research/9128
https://ink.library.smu.edu.sg/context/sis_research/article/10131/viewcontent/29384_EarnHFT_pvoa.pdf
Tags: Add Tag
No Tags, Be the first to tag this record!
Institution: Singapore Management University
Language: English
id sg-smu-ink.sis_research-10131
record_format dspace
spelling sg-smu-ink.sis_research-101312024-08-01T09:32:45Z EarnHFT: Efficient hierarchical reinforcement learning for high frequency trading QIN, Molei SUN, Shuo ZHANG, Wentao XIA, Haochong WANG, Xinrun AN, Bo High-frequency trading (HFT) is using computer algorithms to make trading decisions in short time scales (e.g., second-level), which is widely used in the Cryptocurrency (Crypto) market, (e.g., Bitcoin). Reinforcement learning (RL) in financial research has shown stellar performance on many quantitative trading tasks. However, most methods focus on low-frequency trading, e.g., day-level, which cannot be directly applied to HFT because of two challenges. First, RL for HFT involves dealing with extremely long trajectories (e.g., 2.4 million steps per month), which is hard to optimize and evaluate. Second, the dramatic price fluctuations and market trend changes of Crypto make existing algorithms fail to maintain satisfactory performances. To tackle these challenges, we propose an Efficient hieArchical Reinforcement learNing method for High Frequency Trading (EarnHFT), a novel three-stage hierarchical RL framework for HFT. In stage I, we compute a Q-teacher, i.e., the optimal action value based on dynamic programming, for enhancing the performance and training efficiency of second level RL agents. In stage II, we construct a pool of diverse RL agents for different market trends, distinguished by return rates, where hundreds of RL agents are trained with different preferences of return rates and only a tiny fraction of them will be selected into the pool based on their profitability. In stage III, we train a minute-level router which dynamically picks a second-level agent from the pool to achieve stable performance across different markets. Through extensive experiments in various market trends on Crypto markets in a high-fidelity simulation trading environment, we demonstrate that EarnHFT significantly outperforms 6 state-of-art baselines in 6 popular financial criteria, exceeding the runner-up by 30% in profitability. 2024-02-01T08:00:00Z text application/pdf https://ink.library.smu.edu.sg/sis_research/9128 info:doi/10.1609/aaai.v38i13.29384 https://ink.library.smu.edu.sg/context/sis_research/article/10131/viewcontent/29384_EarnHFT_pvoa.pdf http://creativecommons.org/licenses/by-nc-nd/4.0/ Research Collection School Of Computing and Information Systems eng Institutional Knowledge at Singapore Management University Reinforcement Learning Time-Series/Data Streams Artificial Intelligence and Robotics Finance and Financial Management Numerical Analysis and Scientific Computing
institution Singapore Management University
building SMU Libraries
continent Asia
country Singapore
Singapore
content_provider SMU Libraries
collection InK@SMU
language English
topic Reinforcement Learning
Time-Series/Data Streams
Artificial Intelligence and Robotics
Finance and Financial Management
Numerical Analysis and Scientific Computing
spellingShingle Reinforcement Learning
Time-Series/Data Streams
Artificial Intelligence and Robotics
Finance and Financial Management
Numerical Analysis and Scientific Computing
QIN, Molei
SUN, Shuo
ZHANG, Wentao
XIA, Haochong
WANG, Xinrun
AN, Bo
EarnHFT: Efficient hierarchical reinforcement learning for high frequency trading
description High-frequency trading (HFT) is using computer algorithms to make trading decisions in short time scales (e.g., second-level), which is widely used in the Cryptocurrency (Crypto) market, (e.g., Bitcoin). Reinforcement learning (RL) in financial research has shown stellar performance on many quantitative trading tasks. However, most methods focus on low-frequency trading, e.g., day-level, which cannot be directly applied to HFT because of two challenges. First, RL for HFT involves dealing with extremely long trajectories (e.g., 2.4 million steps per month), which is hard to optimize and evaluate. Second, the dramatic price fluctuations and market trend changes of Crypto make existing algorithms fail to maintain satisfactory performances. To tackle these challenges, we propose an Efficient hieArchical Reinforcement learNing method for High Frequency Trading (EarnHFT), a novel three-stage hierarchical RL framework for HFT. In stage I, we compute a Q-teacher, i.e., the optimal action value based on dynamic programming, for enhancing the performance and training efficiency of second level RL agents. In stage II, we construct a pool of diverse RL agents for different market trends, distinguished by return rates, where hundreds of RL agents are trained with different preferences of return rates and only a tiny fraction of them will be selected into the pool based on their profitability. In stage III, we train a minute-level router which dynamically picks a second-level agent from the pool to achieve stable performance across different markets. Through extensive experiments in various market trends on Crypto markets in a high-fidelity simulation trading environment, we demonstrate that EarnHFT significantly outperforms 6 state-of-art baselines in 6 popular financial criteria, exceeding the runner-up by 30% in profitability.
format text
author QIN, Molei
SUN, Shuo
ZHANG, Wentao
XIA, Haochong
WANG, Xinrun
AN, Bo
author_facet QIN, Molei
SUN, Shuo
ZHANG, Wentao
XIA, Haochong
WANG, Xinrun
AN, Bo
author_sort QIN, Molei
title EarnHFT: Efficient hierarchical reinforcement learning for high frequency trading
title_short EarnHFT: Efficient hierarchical reinforcement learning for high frequency trading
title_full EarnHFT: Efficient hierarchical reinforcement learning for high frequency trading
title_fullStr EarnHFT: Efficient hierarchical reinforcement learning for high frequency trading
title_full_unstemmed EarnHFT: Efficient hierarchical reinforcement learning for high frequency trading
title_sort earnhft: efficient hierarchical reinforcement learning for high frequency trading
publisher Institutional Knowledge at Singapore Management University
publishDate 2024
url https://ink.library.smu.edu.sg/sis_research/9128
https://ink.library.smu.edu.sg/context/sis_research/article/10131/viewcontent/29384_EarnHFT_pvoa.pdf
_version_ 1814047750058147840