Partially Observable Markov Decision Process

A partially observable Markov decision process (POMDP) is a generalization of a Markov decision process in which the agent cannot directly observe the underlying state of the environment and must act on the basis of uncertain observations. A POMDP is formally a tuple ⟨S, A, T, R, Ω, O⟩ with states S, actions A, a transition model T, a reward function R, an observation set Ω, and an observation model O, and the agent summarizes its action–observation history in a belief state, a probability distribution over states that serves as a sufficient statistic. Variants add a finite time horizon, a discount factor, and an initial belief, giving the extended tuple ⟨h, S, A, Ω, T, O, R, γ, b₀⟩. The figure illustrates this interaction loop: at each step the policy selects an action based on the current belief, and a Bayes filter update incorporates that action and the new observation to produce the next belief.

One interaction step of a POMDP: the policy π acts on the belief b, and a Bayes filter update τ incorporates the action and the new observation.