Dynamic Programming And Optimal Control, Vol. 1

by Dimitri P. Bertsekas

This 4th edition is a major revision of Vol. I of the leading two-volume dynamic programming textbook by Bertsekas, and contains a substantial amount of new material, particularly on approximate DP in Chapter 6. This chapter was thoroughly reorganized and rewritten, to bring it in line, both with the contents of Vol. II, whose latest edition appeared in 2012, and with recent developments, which have propelled approximate DP to the forefront of attention. Some of the highlights of the revision of Chapter 6 are an increased emphasis on one-step and multistep lookahead methods, parametric approximation architectures, neural networks, rollout, and Monte Carlo tree search. Among other applications, these methods have been instrumental in the recent spectacular success of computer Go programs. The material on approximate DP also provides an introduction and some perspective for the more analytically oriented treatment of Vol. II.The book includes a substantial number of examples, and exercises, detailed solutions of many of which are posted on the internet. It was developed through teaching graduate courses at M.I.T., and is supported by a large amount of educational material, such as slides and videos, posted at the MIT Open Courseware, the author's, and the publisher's web sites.Contents: 1. The Dynamic Programming Algorithm. 2. Deterministic Systems and the Shortest PathProblem. 3. Problems with Perfect State Information. 4. Problems with Imperfect State Information. 5. Introduction to Infinite Horizon Problems. 6. Approximate Dynamic Programming. 7. Deterministic Continuous-Time Optimal Control.

Genres: Mathematics, Computer Science, Algorithms, Research, Textbooks, Academic

555 pages, Hardcover

First published January 1, 1995

Reviews

[Hyokun Yun · Rating 4 out of 5]

This book provides a very gentle introduction to basics of dynamic programming. I have never seen a book in mathematics or engineering which is more reader-friendly with respect to the presentation of theorems and examples. Most proofs cover almost every case without notorious "this is left to the reader as an exercise", and every example is accompanied with detailed steps for computation. Although this book is targeted for first-year graduate students, undergraduate students would not have much difficulty understanding most of the material.

I can't say much about the coverage of the breadth since I am not an expert on dynamic programming, but the book seems to cover a good range of topics, from basic discrete finite-horizon problems to infinite-horizon problems, continuous-time problems, and approximate control. Unfortunately, the chapter on approximate control, which is the most fashionable topic today and a matter of fact the primary motivation for me to read this book, is focused mostly on delivering very basic intuitions and defers most of serious discussions to Volume II.

I would've loved this book more if it contained numerical exercises as well. Although computational considerations are discussed time to time, most of the examples and exercise problems are analytical ones. This is unfortunate, because implementing an algorithm is often a very good way of understanding it. Since that dynamic programming has a lot of fascinating applications, implementing algorithms for such problems and seeing them work would help students gain interest on this topic. Sutton and Barto's reinforcement learning book certainly does a very good job on this aspect.