Teacher-Student aplicado a diferentes modelos de recompensa do ambiente Lunar Lander

Abstract

The techniques studied related to learning by reinforcement are becoming more and more common in real world challenges, but one challenge is to reduce the learning time. Currently, the learning time and/or amount of interactions performed by the reinforcement learning agent can result in high costs in applications, as the training of models can consume a lot of time, requiring many interactions between the agent and the task environment. This work seeks to improve learning using a new combination of techniques, the Teacher-Student technique with Reward Shaping. The Teacher-Student technique aims to choose among a set of similar tasks that train for a main task, according to the student’s learning. The Reward Shaping technique, altering the reward to try to accelerate learning, provides more frequent feedback on appropriate behaviors, that is, reports rewards more often. We adapted Teacher-Student algorithms for this combination of techniques, and used the Lunar Lander environment as a case study, using four reward models designed in (ALBUQUERQUE, 2021) for this environment. Experiments were performed running different trainings to compare this approach with training only on the original Lunar Lander (no rewards change), and with the results obtained in (ALBUQUERQUE, 2021) by adopting each of the rewards models individually. The combination of Teacher-Student techniques with Reward Shaping contributed to a new experience in the reinforcement learning area, managing to accelerate the agent’s learning, considering the duration of 600 thousand training steps, reaching the target performance in 2 out of 5 proposals, in addition to of being able to learn better than the original Lunar Lander approach with PPO algorithm.