🦾

Reinforcement Learning

생성일

2024/07/16 06:17

최종 수정일

2025/11/11 07:15

태그

강화학습

작성자

•

Dynamic Programming

◦

Model-based

◦

A method designed to efficiently perform complex computations when updating values by considering all possible next states.

•

Reinforcement Learning (RL)

◦

Model-free

◦

Updates values approximately based on sampled experiences.

◦

Finds an approximate solution to the Bellman optimality equation using the updated values.

•

Tabular Updation Method

◦

Commonly used in both Dynamic Programming (DP) and Reinforcement Learning (RL).

◦

Aims to find the optimal policy by creating a table to store value function estimates and updating them iteratively.

◦

In DP: all values in the table are updated at each iteration.

▪

Typically uses the state-value function for computational efficiency.

▪

Value function is updated greedily

◦

In RL: only the values corresponding to sampled transitions are updated at each iteration.

▪

Transition probabilities are unknown.

▪

The policy is updated using the action-value function.

▪

Since updates are sample-based, the method is not affected by the dimensionality of the state space.

▪

Thus, RL can avoid the curse of dimensionality inherent in DP.

•

Generalized Policy Iteration (GPI) performs Policy Evaluation (PE) and Policy Improvement (PI) based on sampled experiences.

•

The value function is approximated using samples, and the policy is updated greedily based on this approximation.

•

In the policy evaluation phase, value is updated only from sampled data, so full iteration until convergence is not possible.

◦

Transition probabilities and rewards cannot be computed exactly.

•

If the policy becomes stabilized, the values change very little in subsequent iterations.

◦

At this point, the Q-function satisfies the Bellman optimality equation, indicating an optimal policy.