Dynamic Programming and Markov Processes

by Ronald A. Howard

The book presents an analytic structure for a decision-making system that is at the same time both general enough to be descriptive and yet computationally feasible. It is based on the Markov process as a system model, and uses and iterative technique like dynamic programming as its optimization method.

136 pages, Hardcover

First published January 1, 1970

Reviews

[Wangdo Kim · Rating 5 out of 5]

The basic concepts of the Markov process are those of "state" of a system and state "transition." Ronald Howard said that a graphical example of a Markov process is presented by a frog in a lily pond. As time goes by, the frog jumps from one lily pad to another according to his whim of moment. A particular important feature of this book, compared with Richard Bellman's original work of dynamic programming, is that he would much rather have a method that direct itself to the problem of analyzing processes of indefinite duration, processes that will make many transitions before termination. He used z-Transform analysis of Markov processes in order to demonstrate a limiting state probability in a completely ergodic process. I guess his approach (book) has a huge potential to understand how animals make its decision as a function of a signal from the environment called the environment's state.