Optimal control of port-Hamiltonian systems: A continuous-time learning approach

Authors

Lukas Kölsch, Pol Jané Soneira, Felix Strehle, Sören Hohmann

Abstract

In this paper, we propose a continuous-time adaptive feedback controller for the optimal control of input-state-output port-Hamiltonian systems with respect to general Lagrangian performance indices. The proposed control law implements an online learning procedure which uses the Hamiltonian of the system as an initial value function candidate. The continuous-time learning of the value function is achieved by means of a certain Lagrange multiplier that allows to evaluate the optimality of the current solution. In particular, constructive conditions for stabilizing initial value function candidates are stated and asymptotic stability of the closed-loop equilibrium is proven. Simulations of an exemplary nonlinear optimal control problem demonstrate the performance of the controller resulting from the proposed online learning procedure.

Keywords

Port-Hamiltonian systems; Optimization-based controller synthesis; Adaptive control; Dynamic optimization problem

Citation

• Journal: Automatica
• Year: 2021
• Volume: 130
• Issue:
• Pages: 109725
• Publisher: Elsevier BV
• DOI: 10.1016/j.automatica.2021.109725

BibTeX

@article{K_lsch_2021,
  title={{Optimal control of port-Hamiltonian systems: A continuous-time learning approach}},
  volume={130},
  ISSN={0005-1098},
  DOI={10.1016/j.automatica.2021.109725},
  journal={Automatica},
  publisher={Elsevier BV},
  author={Kölsch, Lukas and Jané Soneira, Pol and Strehle, Felix and Hohmann, Sören},
  year={2021},
  pages={109725}
}

Download the bib file

References

• Aoues, S., Cardoso-Ribeiro, F. L., Matignon, D. & Alazard, D. Modeling and Control of a Rotating Flexible Spacecraft: A Port-Hamiltonian Approach. IEEE Transactions on Control Systems Technology vol. 27 355–362 (2019) – 10.1109/tcst.2017.2771244
• Bergna-Diaz, G., Sanchez, S. & Tedeschi, E. Port-Hamiltonian modelling of Modular Multilevel Converters with fixed equilibrium point. 2017 Twelfth International Conference on Ecological Vehicles and Renewable Energies (EVER) 1–12 (2017) doi:10.1109/ever.2017.7935911 – 10.1109/ever.2017.7935911
• Bosch, (2014)
• Crossley, (2005)
• Deo, (2018)
• Falaize, A. & Hélie, T. Passive Guaranteed Simulation of Analog Audio Circuits: A Port-Hamiltonian Approach. Applied Sciences vol. 6 273 (2016) – 10.3390/app6100273
• Fiaz, S., Zonetti, D., Ortega, R., Scherpen, J. M. A. & van der Schaft, A. J. A port-Hamiltonian approach to power network modeling and analysis. European Journal of Control vol. 19 477–485 (2013) – 10.1016/j.ejcon.2013.09.002
• Freeman, (2008)
• Fujimoto, Optimal control of hamiltonian systems via iterative learning. (2003)
• Fujimoto, Optimal control of hamiltonian systems with input constraints via iterative learning. (2003)
• Fujimoto, K. & Koyama, I. Iterative Feedback Tuning for Hamiltonian Systems. IFAC Proceedings Volumes vol. 41 15678–15683 (2008) – 10.3182/20080706-5-kr-1001.02651
• Gao, W. & Jiang, Z.-P. Adaptive Dynamic Programming and Adaptive Optimal Output Regulation of Linear Systems. IEEE Transactions on Automatic Control vol. 61 4164–4169 (2016) – 10.1109/tac.2016.2548662
• Groothuis, S. S., Stramigioli, S. & Carloni, R. Modeling Robotic Manipulators Powered by Variable Stiffness Actuators: A Graph-Theoretic and Port-Hamiltonian Formalism. IEEE Transactions on Robotics vol. 33 807–818 (2017) – 10.1109/tro.2017.2668385
• Horn, (2012)
• Jiang, Z.-P. & Jiang, Y. Robust adaptive dynamic programming for linear and nonlinear systems: An overview. European Journal of Control vol. 19 417–425 (2013) – 10.1016/j.ejcon.2013.05.017
• Kamalapurkar, R., Walters, P. & Dixon, W. E. Model-based reinforcement learning for approximate optimal regulation. Automatica vol. 64 94–104 (2016) – 10.1016/j.automatica.2015.10.039
• Kokotović, P. & Arcak, M. Constructive nonlinear control: a historical perspective. Automatica vol. 37 637–662 (2001) – 10.1016/s0005-1098(01)00002-4
• Kölsch, L., Wieninger, K., Krebs, S. & Hohmann, S. Distributed Frequency and Voltage Control for AC Microgrids based on Primal-Dual Gradient Dynamics. IFAC-PapersOnLine vol. 53 12229–12236 (2020) – 10.1016/j.ifacol.2020.12.1110
• Liu, D., Li, H. & Wang, D. Online Synchronous Approximate Optimal Learning Algorithm for Multi-Player Non-Zero-Sum Games With Unknown Dynamics. IEEE Transactions on Systems, Man, and Cybernetics: Systems vol. 44 1015–1027 (2014) – 10.1109/tsmc.2013.2295351
• Liu, D., Liu, L. & Lu, Y. LQ-Optimal Control of Boundary Control Systems. Iranian Journal of Science and Technology, Transactions of Electrical Engineering vol. 44 403–412 (2019) – 10.1007/s40998-019-00222-6
• Monaco, S. & Normand-Cyrot, D. On optimality of passivity based controllers in discrete-time. Systems & Control Letters vol. 75 117–123 (2015) – 10.1016/j.sysconle.2014.10.002
• Mouhib, O., Jardin, A., Marquis-Favre, W., Bideaux, E. & Thomasset, D. Optimal control problem in bond graph formalism. Simulation Modelling Practice and Theory vol. 17 240–256 (2009) – 10.1016/j.simpat.2008.04.011
• Murray, J. J., Cox, C. J., Lendaris, G. G. & Saeks, R. Adaptive dynamic programming. IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews) vol. 32 140–153 (2002) – 10.1109/tsmcc.2002.801727
• Nageshrao, S. P., Lopes, G. A. D., Jeltsema, D. & Babuška, R. Passivity-based reinforcement learning control of a 2-DOF manipulator arm. Mechatronics vol. 24 1001–1007 (2014) – 10.1016/j.mechatronics.2014.10.005
• Nageshrao, S. P., Lopes, G. A. D., Jeltsema, D. & Babuska, R. Port-Hamiltonian Systems in Adaptive and Learning Control: A Survey. IEEE Transactions on Automatic Control vol. 61 1223–1238 (2016) – 10.1109/tac.2015.2458491
• Okura, Y., Fujimoto, K., Maruta, I., Saito, A. & Ikeda, H. Bayesian Inference for Path Following Control of Port-Hamiltonian Systems with Training Trajectory Data. SICE Journal of Control, Measurement, and System Integration vol. 13 40–46 (2020) – 10.9746/jcmsi.13.40
• Ortega, R. & García-Canseco, E. Interconnection and Damping Assignment Passivity-Based Control: A Survey. European Journal of Control vol. 10 432–450 (2004) – 10.3166/ejc.10.432-450
• Ortega, R., Praly, L., Astolfi, A., Lee, J. & Nam, K. Estimation of Rotor Position and Speed of Permanent Magnet Synchronous Motors With Guaranteed Stability. IEEE Transactions on Control Systems Technology vol. 19 601–614 (2011) – 10.1109/tcst.2010.2047396
• Ortega, R., van der Schaft, A., Castanos, F. & Astolfi, A. Control by Interconnection and Standard Passivity-Based Control of Port-Hamiltonian Systems. IEEE Transactions on Automatic Control vol. 53 2527–2542 (2008) – 10.1109/tac.2008.2006930
• Sackmann, M. S. & Krebs, V. G. Modified Optimal Control: Global Asymptotic Stabilization of Nonlinear Systems. IFAC Proceedings Volumes vol. 33 199–204 (2000) – 10.1016/s1474-6670(17)37190-2
• van der Schaft, (2017)
• Sepulchre, (1997)
• Singh, (2019)
• Sontag, E. D. A ‘universal’ construction of Artstein’s theorem on nonlinear stabilization. Systems & Control Letters vol. 13 117–123 (1989) – 10.1016/0167-6911(89)90028-5
• Sprangers, O., Babuska, R., Nageshrao, S. P. & Lopes, G. A. D. Reinforcement Learning for Port-Hamiltonian Systems. IEEE Transactions on Cybernetics vol. 45 1017–1027 (2015) – 10.1109/tcyb.2014.2343194
• Stegink, T. W., Persis, C. D. & van der Schaft, A. J. Port-Hamiltonian Formulation of the Gradient Method Applied to Smart Grids. IFAC-PapersOnLine vol. 48 13–18 (2015) – 10.1016/j.ifacol.2015.10.207
• Stegink, T., De Persis, C. & van der Schaft, A. A Unifying Energy-Based Approach to Stability of Power Grids With Market Dynamics. IEEE Transactions on Automatic Control vol. 62 2612–2622 (2017) – 10.1109/tac.2016.2613901
• Strehle, F. et al. Towards Port-Hamiltonian Modeling of Multi-Carrier Energy Systems: A Case Study for a Coupled Electricity and Gas Distribution System. IFAC-PapersOnLine vol. 51 463–468 (2018) – 10.1016/j.ifacol.2018.03.078
• Vamvoudakis, K. G. & Lewis, F. L. Online actor–critic algorithm to solve the continuous-time infinite horizon optimal control problem. Automatica vol. 46 878–888 (2010) – 10.1016/j.automatica.2010.02.018
• Wang, D., He, H. & Liu, D. Adaptive Critic Nonlinear Robust Control: A Survey. IEEE Transactions on Cybernetics vol. 47 3429–3451 (2017) – 10.1109/tcyb.2017.2712188
• Wu, Y., Hamroun, B., Le Gorrec, Y. & Maschke, B. Reduced order LQG control design for port Hamiltonian systems. Automatica vol. 95 86–92 (2018) – 10.1016/j.automatica.2018.05.003