ABSTRACT:

Reinforcement learning (RL) offers remarkable versatility across applications such as robotics, industrial
control, theorem proving, and fine-tuning large language models. It does this by framing diverse tasks as
sequential decision processes without relying on pre-existing datasets. However, in long-horizon
environments with sparse rewards, RL’s efficiency is severely limited as agents may require an
exponential number of steps to locate non-zero rewards. This exploration bottleneck is a critical barrier
to the practical deployment of RL methods.

Traditional theoretically-motivated exploration strategies based on “optimism in the face of uncertainty” perform poorly and require extensive fine-tuning when integrated with neural networks, as non-stationarity and frequent policy changes tend to result in plasticity loss and model collapse. In contrast, optimization-based exploration objectives naturally align with neural network training and yields more stable, efficient exploration by transforming the exploration challenge into a stationary objective. Our project aims to develop practical, provably efficient RL algorithms for long-horizon exploration in
continuous environments. Specifically, we propose to generalize a theoretically developed long-horizon
exploration objective for use with deep neural network training by approximating and regularizing the
state occupancy measure. We focus on two research questions: (1) Can we develop state occupancy measure regularization algorithms with both provably small non-stationarity and provably efficient long-horizon exploration?, and (2) Can state occupancy measure regularization yield efficient exploration and regret bounds in continuous environments when combined with function approximation?
By addressing these questions, our work seeks to overcome the sample efficiency bottleneck that
currently limits RL’s potential, thereby enabling its application in challenging domains such as
manipulation, reasoning, planning, and navigation. The proposed theoretical framework is poised to not
only enhance RL performance but also to provide deeper insights into long-horizon exploration in deep
RL.