NERL

Overview. Reinforcement learning achieves notable results in important applications from game playing to robotics. Yet, these results do not lead to real-world large-scale deployment nor easily translate to many domains where physical systems are involved. The main issues to control high-dimensional physical systems with often hybrid state-action space (e.g. due to physical contacts) stems from limited capabilities to efficiently explore, stabilize and generalize to higher dimensions while ensuring operation safety during deployment. On the other hand, excellent domain-specific models in conjunction with numerically-efficient optimal control algorithms afford easy hard constraint satisfaction for safety, local synthesis of stabilizing behaviors and easily generalize thanks to the large domain of validity of the underlying models. Yet, these approaches cannot exploit past experience, which limits them to local behaviors with unnecessary recurring computational burden. There is a strong potentiality in extending the mutual capabilities of reinforcement learning and optimal control algorithms. The overarching goal of this project is to establish a reinforcement learning framework that leverages numerical methods in optimal control to improve learning efficacy and guarantee safety for systems with hybrid modes. The project will be conducted by a French-US team of researchers with a history of successful collaborations where researchers from each country will equally contribute to the project objectives. Coordination will be ensured through student co-supervision and exchanges and a common software infrastructure. Intellectual merit. The intellectual merit of this effort stems from the formulation of numerically efficient algorithms for reinforcement learning that leverage modern optimal control methods and well-established domain-specific models. The project investigates two complementary aims: 1) the design of policy optimization methods that leverage derivatives from dynamical models and guarantee constraint satisfaction and, 2) the design of algorithms that handle hybrid non-smooth dynamics by leveraging high-fidelity differentiable simulators and strategies for planning across hybrid dynamic modes. The project aims at casting a new light on reinforcement learning and optimal control, which share strong theoretical roots but until now failed to produce unified algorithms. To demonstrate the broad applicability of the approach, the evaluation plan includes numerical experiments on problems of increasing complexity in four complex application domains: towing kites for energy supply, robots with arms and legs, musculoskeletal avatars and microscopic movement of proteins. Broader impact. This effort will generally contribute to the areas of reinforcement learning and optimal control and help building a bridge between them. The research will benefit a large range of domains where reinforcement learning can be applied, in particular high-dimensional systems with hybrid modes, typical in many physical systems. A specific open-source initiative will be created to release all the algorithms and evaluation environments and provide additional material to ensure reproducible science and to lower barriers to entry for practitioners. The project will initiate an internship program for M.Sc. students between NYU and Supaero, providing international research experience to graduate students. Educational activities further include the creation of joint events between France and the US for primary and middle school students to learn AI through a robot programming challenge. Keywords: reinforcement learning; model-predictive control; policy optimization (* abstract written following NSF recommendations)