Plenary talk by Prof. R Srikant (University of Illinois at Urbana-Champaign)
Title: Learning and Control in Countable State Spaces
Time & Venue: December 10, 2024 (Morning Plenary); VH Auditorium
Abstract:
We will consider policy optimization methods in reinforcement learning where the state space is countably infinite. The motivation arises from control problems in communication networks and matching markets. Specifically, we consider the popular Natural Policy Gradient (NPG) algorithm, which has been studied in the past only under the assumptions that the cost is bounded and the state space is finite, neither of which holds for the aforementioned control problems. Assuming a Lyapunov drift condition, which is naturally satisfied in some cases and can be satisfied in other cases at a small cost in performance, we design a state-dependent step-size rule which dramatically improves the performance of NPG for our intended applications. In addition to experimentally verifying the performance improvement, we also theoretically show that the iteration complexity of NPG can be made independent of the size of the state space. The key analytical tool we use is the connection between NPG stepsizes and the solution to Poisson’s equation. In particular, we provide policy-independent bounds on the solution to Poisson’s equation, which are then used to guide the choice of NPG stepsizes.
Speaker Bio:
Prof. Rayadurgam Srikant received his B.Tech. from the Indian Institute of Technology, Madras in 1985, his M.S. and Ph.D. from the University of Illinois in 1988 and 1991, respectively, all in Electrical Engineering. He was a Member of Technical Staff at AT&T Bell Laboratories from 1991 to 1995. He is currently with the University of Illinois at Urbana-Champaign, where he is the Fredric G. and Elizabeth H. Nearing Professor in the Department of Electrical and Computer Engineering, and a Professor in the Coordinated Science Lab. He is also the co-director of the C3.ai Digital Transformation Institute.
He is the recipient of the 2015 INFOCOM Achievement Award and the 2019 IEEE Koji Kobayashi Computers and Communications Award, and has received several Best Paper awards including the 2015 INFOCOM Best Paper Award and the 2017 Applied Probability Society Best Publication Award. He was the Editor-in-Chief of the IEEE/ACM Transactions on Networking from 2013 to 2017. His research interests include communication networks, machine learning, and applied probability. He is a Fellow of the IEEE.