What does RL change inside a model?

Reinforcement fine-tuning makes a model better — but at what, and where? I train sparse autoencoders across a chain of PPO checkpoints and measure how the internal feature geometry drifts as training proceeds.

The pipeline

base model → PPO fine-tune (verl · FSDP · vLLM)
   → cache activations: 8 checkpoints × 4 residual layers
   → train 32 warm-start TopK SAEs → align features
   → decoder cosine drift = how the geometry moved

Warm-starting each SAE from the last keeps feature indices aligned across checkpoints, so decoder cosine-similarity becomes a well-defined drift metric.

Per-feature scorecard tracking representational drift across PPO checkpoints — Feature-level drift tracked across the checkpoint chain.

Reconstruction accuracy of the trained sparse autoencoders — SAE reconstruction holding up across layers and checkpoints.

GitHub ↗