Passivity-based reinforcement learning control of a 2-DOF manipulator arm

Authors

S.P. Nageshrao, G.A.D. Lopes, D. Jeltsema, R. Babuška

Abstract

Passivity-based control (PBC) is commonly used for the stabilization of port-Hamiltonian (PH) systems. The PH framework is suitable for multi-domain systems, for example mechatronic devices or micro-electro-mechanical systems. Passivity-based control synthesis for PH systems involves solving partial differential equations, which can be cumbersome. Rather than explicitly solving these equations, in our approach the control law is parameterized and the unknown parameter vector is learned using an actor–critic reinforcement learning algorithm. The key advantages of combining learning with PBC are: (i) the complexity of the control design procedure is reduced, (ii) prior knowledge about the system, given in the form of a PH model, speeds up the learning process, (iii) physical meaning can be attributed to the learned control law. In this paper we extended the learning-based PBC method to a regulation problem and present the experimental results for a two-degree-of-freedom manipulator. We show that the learning algorithm is capable of achieving feedback regulation in the presence of model uncertainties.

Keywords

Passivity-based control; Port-Hamiltonian systems; Reinforcement learning; Robotics

Citation

• Journal: Mechatronics
• Year: 2014
• Volume: 24
• Issue: 8
• Pages: 1001–1007
• Publisher: Elsevier BV
• DOI: 10.1016/j.mechatronics.2014.10.005

BibTeX

@article{Nageshrao_2014,
  title={{Passivity-based reinforcement learning control of a 2-DOF manipulator arm}},
  volume={24},
  ISSN={0957-4158},
  DOI={10.1016/j.mechatronics.2014.10.005},
  number={8},
  journal={Mechatronics},
  publisher={Elsevier BV},
  author={Nageshrao, S.P. and Lopes, G.A.D. and Jeltsema, D. and Babuška, R.},
  year={2014},
  pages={1001--1007}
}

Download the bib file

References

• Schaft, (2000)
• Duindam, (2009)
• Putting energy back in control. IEEE Control Systems vol. 21 18–33 (2001) – 10.1109/37.915398
• Ortega, R. & Spong, M. W. Adaptive motion control of rigid robots: A tutorial. Automatica vol. 25 877–888 (1989) – 10.1016/0005-1098(89)90054-x
• Ortega, R., van der Schaft, A., Castanos, F. & Astolfi, A. Control by Interconnection and Standard Passivity-Based Control of Port-Hamiltonian Systems. IEEE Transactions on Automatic Control vol. 53 2527–2542 (2008) – 10.1109/tac.2008.2006930
• Sutton, (1998)
• Busoniu, (2010)
• Grondman, I., Busoniu, L., Lopes, G. A. D. & Babuska, R. A Survey of Actor-Critic Reinforcement Learning: Standard and Natural Policy Gradients. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) vol. 42 1291–1307 (2012) – 10.1109/tsmcc.2012.2218595
• Reinforcement learning is direct adaptive optimal control. IEEE Control Systems vol. 12 19–22 (1992) – 10.1109/37.126844
• Peters, Policy gradient methods for robotics. (2006)
• Grondman, I., Vaandrager, M., Busoniu, L., Babuska, R. & Schuitema, E. Efficient Model Learning Methods for Actor–Critic Control. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) vol. 42 591–602 (2012) – 10.1109/tsmcb.2011.2170565
• Sprangers, O., Babuska, R., Nageshrao, S. P. & Lopes, G. A. D. Reinforcement Learning for Port-Hamiltonian Systems. IEEE Transactions on Cybernetics vol. 45 1017–1027 (2015) – 10.1109/tcyb.2014.2343194
• Dirksz, D. A. & Scherpen, J. M. A. Structure Preserving Adaptive Control of Port-Hamiltonian Systems. IEEE Transactions on Automatic Control vol. 57 2880–2885 (2012) – 10.1109/tac.2012.2192359