Guest Editorial for Special Issue on Extensions of Reinforcement Learning and Adaptive Control

Frank L. Lewis Warren Dixon Zhongsheng Hou Tansel Yucelen

A DAPTIVE control is a provenmethod for learning feedback controllers for systems w ith unknown dynamic models,exogenous disturbances,nonzero setpoints,and unmodeled nonlinearities.Adaptive control has been applied for years in process control,industry,aerospace systems,vehicle systems,and elsewhere.Reinforcement learning refers to a broad class of methods for improving control policies based on observation of the performance or value of currentpolicies.Reinforcement learning allows the learning of optimal controls in real-time using datameasured along system trajectories.

Reinforcement learning is closely tied theoretically to both adaptive control and optimal control.Recentwork has shown how to use reinforcement learning to develop new forms of adaptive controllers that converge to optimal control solutions in real time and to effectively deal w ith some existing open problems in adaptive control such as handling unmatched uncertainties.Developed methods include approximate dynam ic programm ing,integral reinforcement learning,temporal difference,adaptive function approximation,and others.

However,there are many methods in adaptive control,iterative learning control,model predictive control,and elsewhere those hold out the hope of improving the design of reinforcement learning for feedback control.On the other hand,there remainmany techniques in reinforcement learning and iterative learning that have not yet been exploited for adaptive feedback control.

Thepurposeof thisspecial issue is to presentabody ofwork that shows how to more closely integrate and cross-fertilize techniques from adaptive control,from reinforcement learning and from iterative learning.Applications of reinforcement learning and from iterative learning in adaptive feedback controlhave by and largeemployed performance indicesbased on tracking errorsmeasured along system trajectories.However,learning techniques include a far broader range of methods including planning,episodic learning,learning w ith reduced information,optimal control for Markov decision processes and more that have not been fully explored for feedback control design.

Selected Papers for Issue 3“Theoretical Approaches Integrating Adaptive Control,Reinforcement Learning and Iterative learning”:

1)Off-Policy Reinforcement Learning with Gaussian Processes

2)Concurrent Learning-based Approximate Feedback-Nash Equilibrium Solution of N-player Nonzero-sum Differential Games

3)Clique-based Cooperative Multiagent Reinforcement Learning Using Factor Graphs

4)Reinforcement Learning Transfer Based on Subgoal Discovery and Subtask Similarity

5)Closed-loop P-Type Iterative Learning Control of Uncertain Linear Distributed Parameter Systems

6)Experience Replay for Least-Squares Policy Iteration

7)Event-Triggered Optimal Adaptive Control Algorithm for Continuous-Time Nonlinear Systems

8)Robust Adaptive Model Tracking for Distributed Parameter Control of Linear Infinite-dimensional Systems in Hilbert Space

9)Adaptive Iterative Learning Control for a Class of Nonlinear Time-varying Systems with Unknown Delays and Input Dead-zone

10)An Improved ResultofMultipleModel Iterative Learning Control

11)Continuous Action Reinforcement Learning for Control-Affine Systemswith Unknown Dynamics

Selected Papersfor Issue 4“Formal Learning-Based Solutions to Open ControlProblemsRanging from Reactors to Aircraft”:

1)Coordinated Adaptive Control for Coordinated Pathfollowing Surface Vesselswith a Time-invariant Orbital Velocity

2)Adaptive Iterative Learning Control Based on Unfalsified Strategy for Chylla-Haase Reactor

3)Parameters Tuning of Model Free Adaptive Control Based on Minimum Entropy

4)Near Optimal Output Feedback Control of Nonlinear Discrete-time Systems Based on Reinforcement Neural Network Learning

5)An Adaptive Obstacle Avoidance Algorithm for Unmanned Surface Vehicle in Complicated Marine Environments

6)Adaptive Pinpoint and Fuel Efficient Mars Landing Using Reinforcement Learning

7)Online Adaptive Approximate Optimal Tracking Control with Simplified Dual Approximation Structure for Continuous-time Unknown Nonlinear Systems

8)Bi-Objective Optimal Control Modification Adaptive Control for Systems with Input Uncertainty

9)Reinforcement Learning Based Controller Synthesis for Flexible AircraftWings

Finally,we take this opportunity to thank all the authors for their subm issions and all the reviewerswho took their time in rigorously going through these papers.Deputy Editor-in-Chief Derong Liu provided enormousassistance and help during the entire preparation time of this special issue and we are glad to have a chance to work w ith him.