Authors

Shuichi FUKUNAGA, Yuki IWAMOTO

Abstract

In this paper, we accelerated a reinforcement learning algorithm for port-Hamiltonian systems using a natural gradient method. The proposed algorithm consists of an actor-critic structure wherein the actor generates a control input according to a policy and learns the policy using a temporal difference (TD) error, and the critic computes the TD error and learns a state-value function. Furthermore, the reinforcement learning algorithm for port-Hamiltonian systems has two types of the policy parameters which the proposed algorithm learns using the natural gradient method. Additionally, the proposed method was applied to the problem of swing-up control for an inverted pendulum through numerical simulation. The simulation result showed that the learning speed of the proposed method was higher than that of the existing method.

Citation

  • Journal: Transactions of the Society of Instrument and Control Engineers
  • Year: 2023
  • Volume: 59
  • Issue: 2
  • Pages: 70–76
  • Publisher: The Society of Instrument and Control Engineers
  • DOI: 10.9746/sicetr.59.70

BibTeX

@article{FUKUNAGA_2023,
  title={{Acceleration of Reinforcement Learning for Port-Hamiltonian Systems Using Natural Gradient}},
  volume={59},
  ISSN={1883-8189},
  DOI={10.9746/sicetr.59.70},
  number={2},
  journal={Transactions of the Society of Instrument and Control Engineers},
  publisher={The Society of Instrument and Control Engineers},
  author={FUKUNAGA, Shuichi and IWAMOTO, Yuki},
  year={2023},
  pages={70--76}
}

Download the bib file

References

  • Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015) – 10.1038/nature14236
  • Silver, D. et al. Mastering the game of Go without human knowledge. Nature 550, 354–359 (2017) – 10.1038/nature24270
  • Viharos, Z. J. & Jakab, R. Reinforcement Learning for Statistical Process Control in Manufacturing. Measurement 182, 109616 (2021) – 10.1016/j.measurement.2021.109616
  • Kober, J., Bagnell, J. A. & Peters, J. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research 32, 1238–1274 (2013) – 10.1177/0278364913495721
  • Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J. & Quillen, D. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research 37, 421–436 (2017) – 10.1177/0278364917710318
  • 6) 小林泰介：《第7回》機械学習と制御：連続行動空間における強化学習，計測と制御，58-10, 806/810 (2019)
  • van der Schaft, A. L2-Gain and Passivity Techniques in Nonlinear Control. Communications and Control Engineering (Springer International Publishing, 2017). doi:10.1007/978-3-319-49992-5 – 10.1007/978-3-319-49992-5
  • 8) 藤本健治：ポート・ハミルトン系モデルを用いた機械システム制御の最前線，システム/制御/情報，64-12, 483/490 (2020)
  • Sprangers, O., Babuska, R., Nageshrao, S. P. & Lopes, G. A. D. Reinforcement Learning for Port-Hamiltonian Systems. IEEE Trans. Cybern. 45, 1017–1027 (2015) – 10.1109/tcyb.2014.2343194
  • 10) R.S. Sutton, D. McAllester, S. Singh and Y. Mansour: Policy Gradient Methods for Reinforcement Learning with Function Approximation, Proceedings of the 12th International Conference on Neural Information Processing Systems, 1057/1063 (1999)
  • 11) S. Kakade: A Natural Policy Gradient, Proceedings of the 14th International Conference on Neural Information Processing Systems, 1531/1538 (2001)
  • Peters, J. & Schaal, S. Natural Actor-Critic. Neurocomputing 71, 1180–1190 (2008) – 10.1016/j.neucom.2007.11.026
  • Amari, S. Natural Gradient Works Efficiently in Learning. Neural Computation 10, 251–276 (1998) – 10.1162/089976698300017746
  • 14) T. Morimura, E. Uchibe and K. Doya: Utilizing the natural gradient in temporal difference reinforcement learning with eligibility traces, Proceedings of the 2nd International Symposium on Information Geometry and its Applications, 256/263 (2005)
  • Bhatnagar, S., Sutton, R. S., Ghavamzadeh, M. & Lee, M. Natural actor–critic algorithms. Automatica 45, 2471–2482 (2009) – 10.1016/j.automatica.2009.07.008
  • 16) P.S. Thomas: Bias in Natural Actor-Critic Algorithms, Proceedings of the 31st International Conference on Machine Learning, 441/448 (2014)
  • Iwaki, R. & Asada, M. Implicit incremental natural actor critic algorithm. Neural Networks 109, 103–112 (2019) – 10.1016/j.neunet.2018.10.007
  • 18) 金森，鈴木，竹内，佐藤：機械学習のための連続最適化，講談社 (2016)
  • 19) R.S. Sutton and A.G. Barto: Reinforcement Learning: An Introduction, 2nd edition, MIT Press (2018)
  • 20) M. Zhao: META-Learning Eligibility Traces for More Sample Efficient Temporal Difference Learning, arXiv: 2006.08906 (2020)
  • Sutton, R. S. Learning to predict by the methods of temporal differences. Mach Learn 3, 9–44 (1988) – 10.1007/bf00115009
  • Akiba, T., Sano, S., Yanase, T., Ohta, T. & Koyama, M. Optuna. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining 2623–2631 (2019) doi:10.1145/3292500.3330701 – 10.1145/3292500.3330701