Learning feedback Nash strategies for nonlinear port-Hamiltonian systems

Authors

Lukas Kölsch, Pol Jané Soneira, Albertus Johannes Malan, Sören Hohmann

Abstract

This paper presents an adaptive control strategy for solving multi-player noncooperative differential games with dynamics modelled as general nonlinear input-state-output port-Hamiltonian systems. The proposed controller is obtained by extending an existing single-player feedback Nash strategy to N players and by using the Hamiltonian of the port-Hamiltonian system as an admissible control-Lyapunov function for each player. Necessary and sufficient conditions for the stability of the resulting controlled system are provided by employing Lyapunov stability theory. Furthermore, the N player feedback strategy is extended by adaptively weighting the individual value functions to ensure convergence to the Nash solution. Finally, numerical simulations demonstrate the effectiveness of the proposed explicit control laws.

Citation

• Journal: International Journal of Control
• Year: 2023
• Volume: 96
• Issue: 1
• Pages: 201–213
• Publisher: Informa UK Limited
• DOI: 10.1080/00207179.2021.1986233

BibTeX

@article{K_lsch_2021,
  title={{Learning feedback Nash strategies for nonlinear port-Hamiltonian systems}},
  volume={96},
  ISSN={1366-5820},
  DOI={10.1080/00207179.2021.1986233},
  number={1},
  journal={International Journal of Control},
  publisher={Informa UK Limited},
  author={Kölsch, Lukas and Jané Soneira, Pol and Malan, Albertus Johannes and Hohmann, Sören},
  year={2021},
  pages={201--213}
}

Download the bib file

References

• Abouheaf MI, Lewis FL (2013) Multi-agent differential graphical games: Nash online adaptive learning solutions. In: 52nd IEEE Conference on Decision and Control. IEEE, pp 5803–5809 – 10.1109/cdc.2013.6760804
• Altmann R, Schulze P (2017) A port-Hamiltonian formulation of the Navier–Stokes equations for reactive flows. Systems & Control Letters 100:51–55. https://doi.org/10.1016/j.sysconle.2016.12.005 – 10.1016/j.sysconle.2016.12.005
• Aoues S, Cardoso-Ribeiro FL, Matignon D, Alazard D (2019) Modeling and Control of a Rotating Flexible Spacecraft: A Port-Hamiltonian Approach. IEEE Trans Contr Syst Technol 27(1):355–362. https://doi.org/10.1109/tcst.2017.2771244 – 10.1109/tcst.2017.2771244
• Arrow, K. J., Hurwicz, L., & Uzawa, H. (1958). Studies in linear and non-linear programming. Stanford University Press.
• Avila-Becerril S, Espinosa-Pérez G (2020) Control of islanded microgrids considering power converter dynamics. International Journal of Control 94(9):2520–2530. https://doi.org/10.1080/00207179.2020.1713402 – 10.1080/00207179.2020.1713402
• Başar, T., & Olsder, G. J. (1999). Dynamic noncooperative game theory (2nd ed., Vol. 23). SIAM.
• Bressan A (2011) Noncooperative Differential Games. Milan J Math 79(2):357–427. https://doi.org/10.1007/s00032-011-0163-6 – 10.1007/s00032-011-0163-6
• Engwerda, J. C. (2006). Lq dynamic optimization and differential games. John Wiley & Sons, Ltd.
• Engwerda JC, Salmah (2012) Necessary and Sufficient Conditions for Feedback Nash Equilibria for the Affine-Quadratic Differential Game. J Optim Theory Appl 157(2):552–563. https://doi.org/10.1007/s10957-012-0188-1 – 10.1007/s10957-012-0188-1
• Fiaz S, Zonetti D, Ortega R, Scherpen JMA, van der Schaft AJ (2013) A port-Hamiltonian approach to power network modeling and analysis. European Journal of Control 19(6):477–485. https://doi.org/10.1016/j.ejcon.2013.09.002 – 10.1016/j.ejcon.2013.09.002
• Gheibi A, Ghiasi AR, Ghaemi S, Badamchizadeh MA (2018) Designing of robust adaptive passivity-based controller based on reinforcement learning for nonlinear port-Hamiltonian model with disturbance. International Journal of Control 93(8):1754–1764. https://doi.org/10.1080/00207179.2018.1532607 – 10.1080/00207179.2018.1532607
• Golub G, Kahan W (1965) Calculating the Singular Values and Pseudo-Inverse of a Matrix. Journal of the Society for Industrial and Applied Mathematics Series B Numerical Analysis 2(2):205–224. https://doi.org/10.1137/0702016 – 10.1137/0702016
• Horn RA, Johnson CR (2012) Matrix Analysis. Cambridge University Press – 10.1017/cbo9781139020411
• Kölsch L, Jané Soneira P, Strehle F, Hohmann S (2021) Optimal control of port-Hamiltonian systems: A continuous-time learning approach. Automatica 130:109725. https://doi.org/10.1016/j.automatica.2021.109725 – 10.1016/j.automatica.2021.109725
• Krstic M, Kokotovic PV (1995) Adaptive nonlinear design with controller-identifier separation and swapping. IEEE Trans Automat Contr 40(3):426–440. https://doi.org/10.1109/9.376055 – 10.1109/9.376055
• Lewis FL, Vrabie DL, Syrmos VL (2012) Optimal Control. Wiley – 10.1002/9781118122631
• Liu D, Li H, Wang D (2014) Online Synchronous Approximate Optimal Learning Algorithm for Multi-Player Non-Zero-Sum Games With Unknown Dynamics. IEEE Trans Syst Man Cybern, Syst 44(8):1015–1027. https://doi.org/10.1109/tsmc.2013.2295351 – 10.1109/tsmc.2013.2295351
• Macchelli A, Melchiorri C, Stramigioli S (2009) Port-Based Modeling and Simulation of Mechanical Systems With Rigid and Flexible Links. IEEE Trans Robot 25(5):1016–1029. https://doi.org/10.1109/tro.2009.2026504 – 10.1109/tro.2009.2026504
• Mazouchi M, Naghibi-Sistani MB, Sani SKH (2018) A novel distributed optimal adaptive control algorithm for nonlinear multi-agent differential graphical games. IEEE/CAA J Autom Sinica 5(1):331–341. https://doi.org/10.1109/jas.2017.7510784 – 10.1109/jas.2017.7510784
• Milano F (2009) Continuous Newton’s Method for Power Flow Analysis. IEEE Trans Power Syst 24(1):50–57. https://doi.org/10.1109/tpwrs.2008.2004820 – 10.1109/tpwrs.2008.2004820
• Modares H, Lewis FL, Sistani MN (2012) Online solution of nonquadratic two‐player zero‐sum games arising in the \(H_{\infty}\) control of constrained input systems. Adaptive Control & Signal 28(3-5):232–254. https://doi.org/10.1002/acs.2348 – 10.1002/acs.2348
• Mullins SH, Charlesworth WW, Anderson DC (1995) A New Method for Solving Mixed Sets of Equality and Inequality Constraints. Journal of Mechanical Design 117(2A):322–328. https://doi.org/10.1115/1.2826142 – 10.1115/1.2826142
• Nevistić V. & Primbs J. A. (1996). Constrained nonlinear optimal control: A converse HJB approach. Pasadena CA. https://authors.library.caltech.edu/28127/1/CDS96-021.pdf
• Oliveira TR, Rodrigues VHP, Krstić M, Başar T (2020) Nash Equilibrium Seeking in Quadratic Noncooperative Games Under Two Delayed Information-Sharing Schemes. J Optim Theory Appl 191(2-3):700–735. https://doi.org/10.1007/s10957-020-01757-z – 10.1007/s10957-020-01757-z
• Oliveira TR, Hugo Pereira Rodrigues V, Krstic M, Basar T (2020) Nash Equilibrium Seeking with Arbitrarily Delayed Player Actions. In: 2020 59th IEEE Conference on Decision and Control (CDC). IEEE, pp 150–155 – 10.1109/cdc42340.2020.9303894
• Oliveira TR, Rodrigues VHP, Krstic M, Basar T (2021) Nash Equilibrium Seeking with Players Acting Through Heat PDE Dynamics. In: 2021 American Control Conference (ACC). IEEE, pp 684–689 – 10.23919/acc50511.2021.9483114
• Ortega R, Liu Z, Su H (2012) Control via interconnection and damping assignment of linear time-invariant systems: a tutorial. International Journal of Control 85(5):603–611. https://doi.org/10.1080/00207179.2012.660734 – 10.1080/00207179.2012.660734
• Papavassilopoulos GP, Medanic JV, Cruz JB Jr (1979) On the existence of Nash strategies and solutions to coupled riccati equations in linear-quadratic games. J Optim Theory Appl 28(1):49–76. https://doi.org/10.1007/bf00933600 – 10.1007/bf00933600
• Pfeifer M, Caspart S, Hampel S, Muller C, Krebs S, Hohmann S (2020) Explicit port-Hamiltonian formulation of multi-bond graphs for an automated model generation. Automatica 120:109121. https://doi.org/10.1016/j.automatica.2020.109121 – 10.1016/j.automatica.2020.109121
• Polyak RA (2007) Regularized Newton method for unconstrained convex optimization. Math Program 120(1):125–145. https://doi.org/10.1007/s10107-007-0143-3 – 10.1007/s10107-007-0143-3
• Sackmann MS, Krebs VG (2000) Modified Optimal Control: Global Asymptotic Stabilization of Nonlinear Systems. IFAC Proceedings Volumes 33(13):199–204. https://doi.org/10.1016/s1474-6670(17)37190-2 – 10.1016/s1474-6670(17)37190-2
• Sontag ED (1989) A ‘universal’ construction of Artstein’s theorem on nonlinear stabilization. Systems & Control Letters 13(2):117–123. https://doi.org/10.1016/0167-6911(89)90028-5 – 10.1016/0167-6911(89)90028-5
• Starr AW, Ho YC (1969) Nonzero-sum differential games. J Optim Theory Appl 3(3):184–206. https://doi.org/10.1007/bf00929443 – 10.1007/bf00929443
• Tang D, Chen L, Tian ZF, Hu E (2019) Modified value-function-approximation for synchronous policy iteration with single-critic configuration for nonlinear optimal control. International Journal of Control 94(5):1321–1333. https://doi.org/10.1080/00207179.2019.1648874 – 10.1080/00207179.2019.1648874
• Vamvoudakis KG, Lewis FL (2010) Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica 46(5):878–888. https://doi.org/10.1016/j.automatica.2010.02.018 – 10.1016/j.automatica.2010.02.018
• Vamvoudakis KG, Lewis FL (2011) Multi-player non-zero-sum games: Online adaptive learning solution of coupled Hamilton–Jacobi equations. Automatica 47(8):1556–1569. https://doi.org/10.1016/j.automatica.2011.03.005 – 10.1016/j.automatica.2011.03.005
• Vamvoudakis KG, Miranda MF, Hespanha JP (2016) Asymptotically Stable Adaptive–Optimal Control Algorithm With Saturating Actuators and Relaxed Persistence of Excitation. IEEE Trans Neural Netw Learning Syst 27(11):2386–2398. https://doi.org/10.1109/tnnls.2015.2487972 – 10.1109/tnnls.2015.2487972
• van der Schaft A (2017) L2-Gain and Passivity Techniques in Nonlinear Control. Springer International Publishing, Cham – 10.1007/978-3-319-49992-5
• van der Schaft A, Jeltsema D (2014) Port-Hamiltonian Systems Theory: An Introductory Overview. Foundations and Trends® in Systems and Control 1(2-3):173–378. https://doi.org/10.1561/2600000002 – 10.1561/2600000002
• van der Schaft AJ, Rao S, Jayawardhana B (2015) A network dynamics approach to chemical reaction networks. International Journal of Control 89(4):731–745. https://doi.org/10.1080/00207179.2015.1095353 – 10.1080/00207179.2015.1095353
• Weeren AJTM, Schumacher JM, Engwerda JC (1999) Asymptotic Analysis of Linear Feedback Nash Equilibria in Nonzero-Sum Linear-Quadratic Differential Games. Journal of Optimization Theory and Applications 101(3):693–722. https://doi.org/10.1023/a:1021798322597 – 10.1023/a:1021798322597
• Wu Y, Hamroun B, Le Gorrec Y, Maschke B (2018) Reduced order LQG control design for port Hamiltonian systems. Automatica 95:86–92. https://doi.org/10.1016/j.automatica.2018.05.003 – 10.1016/j.automatica.2018.05.003
• Huaguang Zhang, Lili Cui, Yanhong Luo (2013) Near-Optimal Control for Nonzero-Sum Differential Games of Continuous-Time Nonlinear Systems Using Single-Network ADP. IEEE Trans Cybern 43(1):206–216. https://doi.org/10.1109/tsmcb.2012.2203336 – 10.1109/tsmcb.2012.2203336