CODEWITHSUNDEEP
reinforcement-learningmarkov
Abhishek Bhatia
Abhishek Bhatia

Reputation: 9806

Why is the environment state markov?

I had a question related to reinforcement learning, why is the environment state markov? I read it somewhere it is by definition but I can't understand how the definition of environment state implies that is markov.

Upvotes: 0

Views: 317

Answers (1)

Nick Walker
Nick Walker

Reputation: 798

It isn't necessarily, but in general, reinforcement learning algorithms assume that you provide Markov states.

From chapter 3.5 of Reinforcement Learning: An Introduction:

What we would like, ideally, is a state signal that summarizes past sensations compactly, yet in such a way that all relevant information is retained. This normally requires more than the immediate sensations, but never more than the complete history of all past sensations. A state signal that succeeds in retaining all relevant information is said to be Markov, or to have the Markov property (we define this formally below).

Of course, it's unlikely that you'll ever be able to provide a perfect Markov state representation and actually learn.

The Markov property is important in reinforcement learning because decisions and values are assumed to be a function only of the current state. In order for these to be effective and informative, the state representation must be informative. This means that not all the theory strictly applies to cases in which the Markov property does not strictly apply. However, the theory developed for the Markov case still helps us to understand the behavior of the algorithms, and the algorithms can be successfully applied to many tasks with states that are not strictly Markov.

Upvotes: 3

Related Questions