CarRacing — Proximal Policy Optimization (PPO)
Overview
Developed as the final project for the Reinforcement Learning course at University of San Andrés. We implemented the Proximal Policy Optimization (PPO) clipped surrogate objective algorithm entirely from scratch in Python and PyTorch. The goal was to master continuous control, policy gradient dynamics, Generalized Advantage Estimation (GAE), and entropy regularization, benchmarking our implementation directly against industry-standard baselines such as Stable-Baselines3 (SB3).
Key Features
- From-Scratch Actor-Critic Architecture: Implemented dual neural networks for policy π_θ(a|s) and value estimation V_ϕ(s) with orthogonal weight initialization and gradient clipping.
- Generalized Advantage Estimation (GAE): Implemented GAE(λ) to balance the bias-variance trade-off in multi-step advantage calculation across parallelized rollouts.
- Benchmark Validation on CartPole-v1: Tested on discrete control, consistently solving the environment at the optimal maximum return of 500.
- Autonomous Driving on Highway-Env (racetrack-v0): Trained the agent to navigate multi-lane racing tracks with continuous velocity and steering controls.
- Comparative Benchmarking vs. SB3: Evaluated learning curves, wall-clock sample efficiency, and policy stability against Stable-Baselines3.
Architecture & Implementation
The pipeline consists of an environment wrapper layer handling observation normalization and action bounds, a trajectory rollout collector with Generalized Advantage Estimation (γ = 0.99, λ = 0.95), and mini-batch policy gradient optimization with clipped ratio r_t(θ) ∈ [1 - ε, 1 + ε] (ε = 0.2). Value loss clipping and entropy bonuses prevent early premature policy collapse.
Results & Performance
On CartPole-v1, the agent consistently reached the 500/500 maximum return score. On the continuous racetrack-v0 environment, our custom PPO agent achieved an average return of 348.84 (score = 336.09) with 0.85 average collisions per episode, demonstrating stable lane-following, smooth steering transitions, and robust collision avoidance.