Reinforcement Learning

AI with Python – Reinforcement Learning

Reinforcement Learning (RL) is a paradigm of machine learning where an agent learns to make decisions by interacting with an environment. The agent receives rewards or penalties for its actions, aiming to maximize its cumulative reward over time.

Core Concepts of Reinforcement Learning

RL involves an agent learning optimal behavior through trial and error. Unlike supervised learning, there’s no explicit ‘correct answer’ for each action. Instead, the agent receives feedback (rewards or punishments) from the environment, which ‘reinforces’ good actions and discourages bad ones. This process allows the agent to develop a strategy, or policy, to maximize its long-term reward.

[Image description: A diagram illustrating the reinforcement learning loop. An ‘Agent’ takes an ‘Action’ in an ‘Environment’. The Environment transitions to a new ‘State’ and provides a ‘Reward’ (and observation) back to the Agent. This cycle repeats.]

Key Components: Agent and Environment

The two main components in RL are the agent and the environment.

Agent

The agent is the learner or decision-maker. It perceives the environment’s state, selects actions, and receives rewards.

• A human agent uses senses (eyes, ears) as sensors and physical actions (hands, legs) as actuators.
• A robotic agent might use cameras and Lidar as sensors, and motors as actuators.
• A software agent (e.g., playing a game) processes game data as input and outputs game commands.

Agent Terminology

• State (S): A representation of the environment at a particular time.
• Action (A): A choice made by the agent from a set of possible actions.
• Reward (R): Feedback from the environment indicating the immediate desirability of an action taken in a state.
• Policy (π): The agent’s strategy, mapping states to actions. It defines how the agent behaves.
• Value Function (V or Q): Estimates the expected long-term return (cumulative reward) from a state, or state-action pair, under a given policy.
• Observation: What the agent actually perceives from the environment. This might be the full state or a partial view.
• Episode: A sequence of states, actions, and rewards from a starting state to a terminal state.

Environment

The environment is everything outside the agent. It defines the world in which the agent operates, including the rules for state transitions and reward generation.

Environments can range from simple, simulated worlds (like a game) to complex, real-world systems (like a robot navigating a room or managing financial portfolios).

Properties of Environments

Environments can be characterized by several properties:

• Discrete/Continuous: State and action spaces can be discrete (finite set of possibilities) or continuous (values in a range).
• Observable/Partially Observable: If the agent can perceive the complete state, it’s observable (MDP). If only partial information is available, it’s partially observable (POMDP).
• Deterministic/Stochastic: If the next state and reward are uniquely determined by the current state and action, it’s deterministic. Otherwise, it’s stochastic.
• Episodic/Continuing: Episodic tasks have a clear start and end (e.g., a game). Continuing tasks go on indefinitely (e.g., a robot maintaining balance).
• Single-agent/Multi-agent: Involves one agent or multiple interacting agents.

[Image description: A conceptual table or list outlining properties of environments such as Discrete/Continuous, Observable/Partially Observable, Static/Dynamic, etc., with brief explanations or examples for each.]

Constructing an Environment with Gymnasium (formerly OpenAI Gym)

Gymnasium is a widely used toolkit for developing and comparing RL algorithms. It provides a collection of standardized environments. Install it using pip:

pip install gymnasium

Gymnasium offers various environments like CartPole-v1 (balancing a pole on a cart), MountainCar-v0, and Atari game environments (requiring pip install gymnasium[atari]). Documentation: https://gymnasium.farama.org/

Example: Interacting with the CartPole-v1 environment by taking random actions.

import gymnasium as gym

# Create the CartPole environment
# Use render_mode='human' to see the visualization
env = gym.make('CartPole-v1', render_mode='rgb_array') # or 'human' for a pop-up window

# Reset the environment to get an initial observation
observation, info = env.reset(seed=42) # seed for reproducibility

for _ in range(100): # Run for 100 timesteps
    # Render the environment (optional, creates an array for later display)
    # For 'human' mode, env.render() is called implicitly by step usually or not needed
    # frame = env.render()

    # Take a random action from the action space
    action = env.action_space.sample()

    # Perform the action and get the next state, reward, and other info
    # The step function returns: observation, reward, terminated, truncated, info
    observation, reward, terminated, truncated, info = env.step(action)

    done = terminated or truncated

    if done:
        print(f"Episode finished after {_ + 1} timesteps")
        observation, info = env.reset(seed=42) # Reset for next episode if needed
        # break # If you only want to run one episode

env.close() # Close the environment window and resources
print("Simulation finished.")

[Image description: A conceptual animation or series of frames showing the CartPole environment. A cart moves left or right on a track, with a pole hinged on top. The goal is to keep the pole balanced upright by applying forces to the cart.]

This code demonstrates the basic interaction loop: reset, sample action, step. A real RL agent would use a policy to choose actions instead of env.action_space.sample().

Elements of a Learning Agent

A learning agent typically includes one or more of these components:

• Policy: Decides which action to take in a given state.
• Value function: Predicts future rewards for being in a state or taking an action in a state.
• Model (optional): A representation of the environment’s dynamics (how states transition and rewards are generated).

The example below shows the agent-environment interaction structure. A complete learning agent would involve an algorithm (like Q-learning or Policy Gradients) to update its policy or value function based on experience.

import gymnasium as gym

# env = gym.make('CartPole-v1', render_mode='human')
env = gym.make('CartPole-v1') # No rendering for faster loops if not needed for demo

num_episodes = 5
max_steps_per_episode = 100

for episode in range(num_episodes):
    observation, info = env.reset()
    print(f"Episode {episode + 1} started.")

    for step in range(max_steps_per_episode):
        # In a real agent, action would be chosen by a policy
        action = env.action_space.sample()

        new_observation, reward, terminated, truncated, info = env.step(action)

        # Here, a learning algorithm would update the policy/value function
        # e.g., store (observation, action, reward, new_observation)

        observation = new_observation
        done = terminated or truncated

        if done:
            print(f"  Episode finished after {step + 1} timesteps.")
            break

env.close()

[Image description: Similar to the CartPole visualization, perhaps showing a few successful steps where the pole remains balanced, emphasizing the agent’s goal is to prolong this state.]

This structure forms the basis for implementing various RL algorithms. The agent’s goal is to learn a policy that maximizes the sum of rewards (return) over an episode or indefinitely for continuing tasks.