Skip to main content
SpringerLink
Log in

Neural-network-based stochastic linear quadratic optimal tracking control scheme for unknown discrete-time systems using adaptive dynamic programming

  • Research Article
  • Published:
Control Theory and Technology Aims and scope Submit manuscript

Abstract

In this paper, a stochastic linear quadratic optimal tracking scheme is proposed for unknown linear discrete-time (DT) systems based on adaptive dynamic programming (ADP) algorithm. First, an augmented system composed of the original system and the command generator is constructed and then an augmented stochastic algebraic equation is derived based on the augmented system. Next, to obtain the optimal control strategy, the stochastic case is converted into the deterministic one by system transformation, and then an ADP algorithm is proposed with convergence analysis. For the purpose of realizing the ADP algorithm, three back propagation neural networks including model network, critic network and action network are devised to guarantee unknown system model, optimal value function and optimal control strategy, respectively. Finally, the obtained optimal control strategy is applied to the original stochastic system, and two simulations are provided to demonstrate the effectiveness of the proposed algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Injoong, H., & Gilbert, E. (1987). Robust tracking in nonlinear systems. IEEE Transactions on Automatic Control, 32(9), 763–771.

    Article  MathSciNet  Google Scholar 

  2. Devasia, S., Chen, D., & Paden, B. (1996). Nonlinear inversion based output tracking. IEEE Transactions on Automatic Control, 41(7), 930–942.

    Article  MathSciNet  Google Scholar 

  3. Wang, D., Liu, D., & Wei, Q. (2012). Finite-horizon optimal tracking control for a class of discrete-time nonlinear systems using adaptive dynamic programming approach. Neurocomputing, 78(1), 14–22.

    Article  Google Scholar 

  4. Kiumarsi, B., Lewis, F. L., Modares, H., Karimpour, A., & Naghibi-Sistani, M. B. (2014). Reinforcement Q-learning for optimal tracking control of linear discrete-time systems with unknown dynamics. Automatica, 50(4), 1167–1175.

    Article  MathSciNet  Google Scholar 

  5. Zhao, B., & Li, Y. (2018). Model-free adaptive dynamic programming based near-optimal decentralized tracking control of reconfigurable manipulators. International Journal of Control, Automation and Systems, 16(2), 478–490.

    Article  Google Scholar 

  6. Wei, Q., & Liu, D. (2013). Adaptive dynamic programming for optimal tracking control of unknown nonlinear systems with application to coal gasification. IEEE Transactions on Automation Science and Engineering, 11(4), 1020–1036.

    Article  Google Scholar 

  7. Mu, C., Ni, Z., Sun, C., & He, H. (2016). Air-breathing hypersonic vehicle tracking control based on adaptive dynamic programming. IEEE Transactions on Neural Networks and Learning Systems, 28(3), 584–598.

    Article  MathSciNet  Google Scholar 

  8. Seierstad, A., & Sydsaeter, K. (1986). Optimal control theory with economic applications. New York: Elsevier North-Holland.

    MATH  Google Scholar 

  9. Lewis, F. L., Vrabie, D., & Syrmos, V. L. (2012). Optimal control. Hoboken: Wiley.

    Book  Google Scholar 

  10. Al-Tamimi, A., Lewis, F. .L., & Abu-Khalaf, M. (2008). Discrete-time nonlinear HJB solution using approximate dynamic programming: Convergence proof. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 38(4), 943–949.

    Article  Google Scholar 

  11. Murray, J. J., Cox, C. J., Lendaris, G. G., & Saeks, R. (2002). Adaptive dynamic programming. IEEE Transactions on Systems, Man, and Cybernetics-Part C, 32(2), 140–153.

    Article  Google Scholar 

  12. Lee, J. M., & Lee, J. H. (2005). Approximate dynamic programming-based approaches for input-output data-driven control of nonlinear processes. Automatica, 41(7), 1281–1288.

    Article  MathSciNet  Google Scholar 

  13. Huang, Y., & Liu, D. (2014). Neural-network-based optimal tracking control scheme for a class of unknown discrete-time nonlinear systems using iterative ADP algorithm. Neurocomputing, 125, 46–56.

    Article  Google Scholar 

  14. Bhasin, S., Kamalapurkar, R., Johnson, M., Vamvoudakis, K. G., Lewis, F. L., & Dixon, W. E. (2013). A novel actor-critic-identifier architecture for approximate optimal control of uncertain nonlinear systems. Automatica, 49(1), 82–92.

    Article  MathSciNet  Google Scholar 

  15. Zhang, H., Cui, L., Zhang, X., & Luo, Y. (2011). Data-driven robust approximate optimal tracking control for unknown general nonlinear systems using adaptive dynamic programming method. IEEE Transactions on Neural Networks, 22(12), 2226–2236.

    Article  Google Scholar 

  16. Yang, X., He, H., & Zhong, X. (2017). Adaptive dynamic programming for robust regulation and its application to power systems. IEEE Transactions on Industrial Electronics, 65(7), 5722–5732.

    Article  Google Scholar 

  17. Yang, X., He, H., & Zhong, X. (2019). Approximate dynamic programming for nonlinear-constrained optimizations. IEEE Transactions on Cybernetics. https://doi.org/10.1109/TCYB.2019.2926248.

    Article  Google Scholar 

  18. Kiumarsi, B., Lewis, F. L., Naghibi-Sistani, M. B., & Karimpour, A. (2015). Optimal tracking control of unknown discrete-time linear systems using input-output measured data. IEEE Transactions on Cybernetics, 45(12), 2770–2779.

    Article  Google Scholar 

  19. Tao, G. (2014). Multivariable adaptive control: A survey. Automatica, 50(11), 2737–2764.

    Article  MathSciNet  Google Scholar 

  20. Rami, M. A., Moore, J. B., & Zhou, X. Y. (2002). Indefinite stochastic linear quadratic control and generalized differential Riccati equation. SIAM Journal on Control and Optimization, 40(4), 1296–1311.

    Article  MathSciNet  Google Scholar 

  21. Rami, M. A., & Zhou, X. Y. (2000). Linear matrix inequalities, Riccati equations, and indefinite stochastic linear quadratic controls. IEEE Transactions on Automatic Control, 45(6), 1131–1143.

    Article  MathSciNet  Google Scholar 

  22. Yao, D. D., Zhang, S., & Zhou, X. Y. (2001). Stochastic linear-quadratic control via semidefinite programming. SIAM Journal on Control and Optimization, 40(3), 801–823.

    Article  MathSciNet  Google Scholar 

  23. Liu, X., Li, Y., & Zhang, W. (2014). Stochastic linear quadratic optimal control with constraint for discrete-time systems. Applied Mathematics and Computation, 228, 264–270.

    Article  MathSciNet  Google Scholar 

  24. Wang, T., Zhang, H., & Luo, Y. (2016). Infinite-time stochastic linear quadratic optimal control for unknown discrete-time systems using adaptive dynamic programming approach. Neurocomputing, 171, 379–386.

    Article  Google Scholar 

  25. Wang, T., Zhang, H., & Luo, Y. (2018). Stochastic linear quadratic optimal control for model-free discrete-time systems based on Q-learning algorithm. Neurocomputing, 312, 1–8.

    Article  Google Scholar 

  26. Bian, T., Jiang, Y., & Jiang, Z. P. (2016). Adaptive dynamic programming for stochastic systems with state and control dependent noise. IEEE Transactions on Automatic Control, 61(12), 4170–4175.

    Article  MathSciNet  Google Scholar 

  27. Long, Y., & Su, W. (2016). Optimal tracking for linear systems with multiplicative noises. In Proceedings of 14th International Conference on Control, Automation, Robotics and Vision (ICARCV), Phukhet, Thailand. https://doi.org/10.1109/ICARCV.2016.7838792.

  28. Han, C., & Liu, Y. (2018). Optimal tracking control and stabilization for stochastic systems with multi-step input delay. In Proceedings of 37th Chinese Control Conference (CCC) (pp. 2185–2190). Wuhan, China.

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No. 61873248), the Hubei Provincial Natural Science Foundation of China (Nos. 2017CFA030, 2015CFA010), and the 111 project (No. B17040).

Author information

Authors and Affiliations

  1. School of Automation, China University of Geosciences, Wuhan, 430074, Hubei, China

    Xin Chen & Fang Wang

  2. Hubei Key Laboratory of Advanced Control and Intelligent Automation for Complex Systems, Wuhan, 430074, Hubei, China

    Xin Chen & Fang Wang

Authors
  1. Xin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Chen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, X., Wang, F. Neural-network-based stochastic linear quadratic optimal tracking control scheme for unknown discrete-time systems using adaptive dynamic programming. Control Theory Technol. 19, 315–327 (2021). https://doi.org/10.1007/s11768-021-00046-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11768-021-00046-y

Keywords

Search

Navigation