Abstract
In this paper, a stochastic linear quadratic optimal tracking scheme is proposed for unknown linear discrete-time (DT) systems based on adaptive dynamic programming (ADP) algorithm. First, an augmented system composed of the original system and the command generator is constructed and then an augmented stochastic algebraic equation is derived based on the augmented system. Next, to obtain the optimal control strategy, the stochastic case is converted into the deterministic one by system transformation, and then an ADP algorithm is proposed with convergence analysis. For the purpose of realizing the ADP algorithm, three back propagation neural networks including model network, critic network and action network are devised to guarantee unknown system model, optimal value function and optimal control strategy, respectively. Finally, the obtained optimal control strategy is applied to the original stochastic system, and two simulations are provided to demonstrate the effectiveness of the proposed algorithm.
Similar content being viewed by others
References
Injoong, H., & Gilbert, E. (1987). Robust tracking in nonlinear systems. IEEE Transactions on Automatic Control, 32(9), 763–771.
Devasia, S., Chen, D., & Paden, B. (1996). Nonlinear inversion based output tracking. IEEE Transactions on Automatic Control, 41(7), 930–942.
Wang, D., Liu, D., & Wei, Q. (2012). Finite-horizon optimal tracking control for a class of discrete-time nonlinear systems using adaptive dynamic programming approach. Neurocomputing, 78(1), 14–22.
Kiumarsi, B., Lewis, F. L., Modares, H., Karimpour, A., & Naghibi-Sistani, M. B. (2014). Reinforcement Q-learning for optimal tracking control of linear discrete-time systems with unknown dynamics. Automatica, 50(4), 1167–1175.
Zhao, B., & Li, Y. (2018). Model-free adaptive dynamic programming based near-optimal decentralized tracking control of reconfigurable manipulators. International Journal of Control, Automation and Systems, 16(2), 478–490.
Wei, Q., & Liu, D. (2013). Adaptive dynamic programming for optimal tracking control of unknown nonlinear systems with application to coal gasification. IEEE Transactions on Automation Science and Engineering, 11(4), 1020–1036.
Mu, C., Ni, Z., Sun, C., & He, H. (2016). Air-breathing hypersonic vehicle tracking control based on adaptive dynamic programming. IEEE Transactions on Neural Networks and Learning Systems, 28(3), 584–598.
Seierstad, A., & Sydsaeter, K. (1986). Optimal control theory with economic applications. New York: Elsevier North-Holland.
Lewis, F. L., Vrabie, D., & Syrmos, V. L. (2012). Optimal control. Hoboken: Wiley.
Al-Tamimi, A., Lewis, F. .L., & Abu-Khalaf, M. (2008). Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 38(4), 943–949.
Murray, J. J., Cox, C. J., Lendaris, G. G., & Saeks, R. (2002). Adaptive dynamic programming. IEEE Transactions on Systems, Man, and Cybernetics-Part C, 32(2), 140–153.
Lee, J. M., & Lee, J. H. (2005). Approximate dynamic programming-based approaches for input-output data-driven control of nonlinear processes. Automatica, 41(7), 1281–1288.
Huang, Y., & Liu, D. (2014). Neural-network-based optimal tracking control scheme for a class of unknown discrete-time nonlinear systems using iterative ADP algorithm. Neurocomputing, 125, 46–56.
Bhasin, S., Kamalapurkar, R., Johnson, M., Vamvoudakis, K. G., Lewis, F. L., & Dixon, W. E. (2013). A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems. Automatica, 49(1), 82–92.
Zhang, H., Cui, L., Zhang, X., & Luo, Y. (2011). Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method. IEEE Transactions on Neural Networks, 22(12), 2226–2236.
Yang, X., He, H., & Zhong, X. (2017). Adaptive dynamic programming for robust regulation and its application to power systems. IEEE Transactions on Industrial Electronics, 65(7), 5722–5732.
Yang, X., He, H., & Zhong, X. (2019). Approximate dynamic programming for nonlinear-constrained optimizations. IEEE Transactions on Cybernetics. https://doi.org/10.1109/TCYB.2019.2926248.
Kiumarsi, B., Lewis, F. L., Naghibi-Sistani, M. B., & Karimpour, A. (2015). Optimal tracking control of unknown discrete-time linear systems using input-output measured data. IEEE Transactions on Cybernetics, 45(12), 2770–2779.
Tao, G. (2014). Multivariable adaptive control: A survey. Automatica, 50(11), 2737–2764.
Rami, M. A., Moore, J. B., & Zhou, X. Y. (2002). Indefinite stochastic linear quadratic control and generalized differential Riccati equation. SIAM Journal on Control and Optimization, 40(4), 1296–1311.
Rami, M. A., & Zhou, X. Y. (2000). Linear matrix inequalities, Riccati equations, and indefinite stochastic linear quadratic controls. IEEE Transactions on Automatic Control, 45(6), 1131–1143.
Yao, D. D., Zhang, S., & Zhou, X. Y. (2001). Stochastic linear-quadratic control via semidefinite programming. SIAM Journal on Control and Optimization, 40(3), 801–823.
Liu, X., Li, Y., & Zhang, W. (2014). Stochastic linear quadratic optimal control with constraint for discrete-time systems. Applied Mathematics and Computation, 228, 264–270.
Wang, T., Zhang, H., & Luo, Y. (2016). Infinite-time stochastic linear quadratic optimal control for unknown discrete-time systems using adaptive dynamic programming approach. Neurocomputing, 171, 379–386.
Wang, T., Zhang, H., & Luo, Y. (2018). Stochastic linear quadratic optimal control for model-free discrete-time systems based on Q-learning algorithm. Neurocomputing, 312, 1–8.
Bian, T., Jiang, Y., & Jiang, Z. P. (2016). Adaptive dynamic programming for stochastic systems with state and control dependent noise. IEEE Transactions on Automatic Control, 61(12), 4170–4175.
Long, Y., & Su, W. (2016). Optimal tracking for linear systems with multiplicative noises. In Proceedings of 14th International Conference on Control, Automation, Robotics and Vision (ICARCV), Phukhet, Thailand. https://doi.org/10.1109/ICARCV.2016.7838792.
Han, C., & Liu, Y. (2018). Optimal tracking control and stabilization for stochastic systems with multi-step input delay. In Proceedings of 37th Chinese Control Conference (CCC) (pp. 2185–2190). Wuhan, China.
Acknowledgements
This work was supported by the National Natural Science Foundation of China (No. 61873248), the Hubei Provincial Natural Science Foundation of China (Nos. 2017CFA030, 2015CFA010), and the 111 project (No. B17040).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Chen, X., Wang, F. Neural-network-based stochastic linear quadratic optimal tracking control scheme for unknown discrete-time systems using adaptive dynamic programming. Control Theory Technol. 19, 315–327 (2021). https://doi.org/10.1007/s11768-021-00046-y
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11768-021-00046-y