In-Class Project · Deep Learning

Behavioral Cloning
with PilotNet

A convolutional neural network that learns to steer a car from camera images alone — no lane-detection rules, no hand-tuned logic. It watches a human drive, then drives the simulator itself.

▶ Watch it drive ↗ PilotNet paper ↗ ● Complete

The idea

This is end-to-end learning for self-driving: instead of writing code that detects lane lines and computes a steering correction, you show the network thousands of frames of a human driving and let it learn the mapping from pixels straight to steering angle on its own. I built the full pipeline in TensorFlow — data exploration, a balancing and augmentation layer, the NVIDIA PilotNet CNN, and a real-time server that drives the Udacity simulator over a socket connection.

It reproduces the core result from NVIDIA’s 2016 paper “End to End Learning for Self-Driving Cars” on the Udacity simulator dataset, and it’s the class version of a project I’m later extending to real camera-plus-CAN data from my own car.

Best val MSE

0.0156

mean squared error, epoch 20

Val MAE

~0.096

≈4.8% of the full steering range

Training images

~24k

8k frames × 3 cameras, balanced

Conv + FC layers

5 + 4

PilotNet regression head

How it works

Simulator3 cameras + steering log, 160×320 frames

→

Preprocesscrop → resize 66×200 → YUV

→

PilotNet CNN5 conv + 4 dense, 1 output neuron

→

Steeringpredicted angle back to the sim, ~15×/s

During training the network learns from recorded frames. At drive time, drive.py runs a Flask + SocketIO server: the simulator streams live camera frames, the model predicts a steering angle for each one, and a small proportional controller sets the throttle — the exact same preprocessing runs on both sides so the model sees what it was trained on.

The data problem

The hardest part of behavioral cloning isn’t the network — it’s the data. Most driving is straight, so the raw dataset is dominated by a steering angle of zero. Train on it as-is and the model learns the laziest possible policy: go straight, always. It looks great until the first curve.

Steering angle distribution histograms showing a large spike at zero — **The bias, visualized.** 54.3% of frames have a steering angle of exactly zero, and 60.7% are near-zero. I profiled this in explore_data.py before touching the model, then corrected for it by randomly dropping a share of the straight-driving samples so turns aren’t drowned out.

A grid of preprocessed training images with their steering labels — **A live training batch, after augmentation.** Three fixes work together: all **three cameras** (left/right frames get a steering correction, teaching recovery from drift), **horizontal flips** that mirror the frame and negate the angle to balance left vs. right turns, plus random **brightness and translation**. Roughly 8k recorded frames expand into ~24k+ training samples.

What the model sees

Raw camera frame beside its cropped, resized, YUV-converted version — **Preprocessing pipeline.** The raw 320×160 camera frame (left) is cropped to drop the sky and the car’s hood, resized to 66×200, and converted to YUV — the same input size and color space NVIDIA used. The network only ever sees the road (right, brightness channel), which trains faster and generalizes better across lighting.

The network

A faithful build of NVIDIA’s PilotNet: five convolutional layers extract road features at increasing scales, then four fully-connected layers collapse them into a single steering decision. Because steering is a continuous value, the output neuron is linear — this is regression, not classification — trained on mean-squared-error with the Adam optimizer, ELU activations, batch-norm, and dropout for regularization.

Layer	Detail	Output	Role
Input	YUV, normalized 0–1	66 × 200 × 3	preprocessed frame
Conv 1	24 × 5×5, stride 2	31 × 98 × 24	edges, lines
Conv 2	36 × 5×5, stride 2	14 × 47 × 36
Conv 3	48 × 5×5, stride 2	5 × 22 × 48	road shapes
Conv 4	64 × 3×3, stride 1	3 × 20 × 64
Conv 5	64 × 3×3, stride 1	1 × 18 × 64	fine features
Flatten + Dropout	0.5 drop rate	1152	regularize
Dense 1–4	1164 → 100 → 50 → 10, ELU	—	combine features
Output	1 neuron, linear	1	steering angle

Results

Training used early stopping, model checkpointing on validation loss, and learning-rate reduction on plateau — all on an RTX 4060. Validation loss tracked training loss closely the whole way down, so no meaningful overfitting.

Training and validation loss curves over 20 epochs — **20 epochs, best validation MSE 0.0156.** Final validation MAE landed around **0.096 steering units** — roughly 4.8% of the full −1…+1 range, comfortably inside “stays in lane reliably.” The tight gap between the training and validation curves is the sign the model generalized rather than memorized.

What it demonstrates

● MLEnd-to-end deep learning. A CNN trained for regression that maps raw pixels directly to a control output — built, trained, and run, not just described.
● DataData engineering that decides the outcome. Diagnosing dataset bias and fixing it with resampling, multi-camera labeling, and augmentation — the part that actually made the model drive.
● SystemsReal-time inference. A live socket server doing ~15 predictions per second with identical train/inference preprocessing — the discipline that keeps a model working outside the notebook.
● CraftA clean, reproducible project. Config-driven, seeded for reproducibility, and split into readable modules — the way production ML is structured.