Markov Chains and Decision Processes for Engineers and Managers (Hardcover)

Markov Chain Structure and Models
Historical Note
States and Transitions
Model of the Weather
Random Walks
Estimating Transition Probabilities
Multiple-Step Transition Probabilities
State Probabilities after Multiple Steps
Classification of States
Markov Chain Structure
Markov Chain Models
Problems
References

Regular Markov Chains
Steady State Probabilities
First Passage to a Target State
Problems
References

Reducible Markov Chains
Canonical Form of the Transition Matrix
The Fundamental Matrix
Passage to a Target State
Eventual Passage to a Closed Set Within a Reducible Multichain
Limiting Transition Probability Matrix
Problems
References

A Markov Chain with Rewards (MCR)
Rewards
Undiscounted Rewards
Discounted Rewards
Problems
References

A Markov Decision Process (MDP)
An Undiscounted MDP
A Discounted MDP
Problems
References

Special Topics: State Reduction and Hidden Markov Chains
State Reduction
An Introduction to Hidden Markov
Problems
References
Index