The algorithm using Monte Carlo method

• Since we have too many states, lets take only subset of the states - $\tilde{S}$.
• $\forall s\in \tilde{S}$, there are M(s) runs : c( s,1 ) ... c( s,M(s)).
• We look for r, which minimizes,
  
  $$\sum_{s\in \tilde{s}} \sum_{i=1}^{M(s)} (\tilde{V}^{\pi}(s,r) - c(s,i))^{2}$$
  
  

  

Figure: Diagram for a mechanism that produces Approximate Policy Iteration