Exploratory State Representation Learning

Image credit: XSRL in PhD Defense Presentation


Not having access to compact and meaningful representations is known to significantly increase the complexity of reinforcement learning (RL). For this reason, it can be useful to perform state representation learning (SRL) before tackling RL tasks. However, obtaining a good state representation can only be done if a large diversity of transitions is observed, which can require a difficult exploration, especially if the environment is initially reward-free. To solve the problems of exploration and SRL in parallel, we propose a new approach called XSRL (eXploratory State Representation Learning). On one hand, it jointly learns compact state representations and a state transition estimator which is used to remove unexploitable information from the representations. On the other hand, it continuously trains an inverse model, and adds to the prediction error of this model a $k$-step learning progress bonus to form the maximization objective of a discovery policy. This results in a policy that seeks complex transitions from which the trained models can effectively learn. Our experimental results show that the approach leads to efficient exploration in challenging environments with image observations, and to state representations that significantly accelerate learning in RL tasks.

Published in Frontiers in Robotics and AI, 2022

For a complete presentation and explanation of our experiments see Paper and XSRL chapter of my PhD Defense Presentation.

XSRL Exploration Evaluation

XSRL Representations Transfer with Reinforcement Learning

Astrid Merckling
Astrid Merckling
Machine Learning Expert

Using machine learning for giving unsupervised exploratory skills to robots.