Self-Driving Car Simulation - Deep Learning CNN
End-to-end deep learning system using NVIDIA CNN architecture to autonomously drive a car in Udacity simulator. Features real-time steering prediction from camera images with comprehensive data augmentation and preprocessing pipeline.
Technologies Used
Project Overview
🚗 End-to-End Deep Learning for Autonomous Driving
Implemented an autonomous driving system using the NVIDIA CNN architecture that learns to predict steering angles directly from raw camera images, enabling the car to navigate a track without human intervention in the Udacity self-driving car simulator.
🧠 Deep Learning Architecture
- NVIDIA-inspired CNN with 5 convolutional + 4 fully connected layers
- ~1.6 million trainable parameters for steering prediction
- Input: 66×200×3 YUV images preprocessed from simulator
- Output: Single steering angle value (-1 to 1 range)
- End-to-end learning from pixels to steering commands
- Real-time inference at 30+ FPS in autonomous mode
📊 Data Collection & Preprocessing
- Manual driving data collection (15,000+ training images)
- 5-stage preprocessing pipeline: crop, YUV conversion, blur, resize, normalize
- Data balancing to prevent straight-driving bias (40-50% turns)
- 80/20 train-validation split with stratified sampling
- Batch generator for efficient memory usage during training
- Comprehensive visualization tools for data analysis
🎨 Advanced Data Augmentation
- Horizontal flip (50%) - mirrors image and negates steering
- Random brightness adjustment (50%) - simulates lighting conditions
- Shadow augmentation (30%) - adds random shadows to images
- Horizontal pan (30%) - shifts image ±50 pixels with steering correction
- Zoom augmentation (30%) - scales image ±20% to simulate distance
- Prevents overfitting and improves generalization
🔧 Training Pipeline
- TensorFlow 2.19+ with Keras 3.11+ for model training
- Adam optimizer with 0.0003 learning rate
- Mean Squared Error (MSE) loss function
- Early stopping with patience=5 to prevent overfitting
- Model checkpointing to save best validation performance
- Training visualization with loss and MAE metrics
🚀 Real-Time Autonomous Driving
- Socket.IO communication with Udacity simulator
- Real-time telemetry: speed, steering angle, throttle
- Adaptive throttle control based on steering angle
- Configurable max speed for safety and stability
- Live performance monitoring and debugging
- Successful autonomous navigation on trained track
📈 Model Performance
- Validation Loss (MSE): 0.0323
- Validation MAE: 0.1392 degrees
- Training samples: 4,312 images
- Validation samples: 1,078 images
- Training time: 2-5 minutes on GPU, 5-15 minutes on CPU
- Successfully completes multiple laps autonomously
🏆 Technical Achievements
- End-to-End Learning: Implemented complete autonomous driving pipeline from raw camera images to steering predictions
- NVIDIA Architecture: Successfully adapted and trained production-grade CNN architecture for behavioral cloning
- Data Engineering: Built comprehensive data collection, balancing, and augmentation pipeline for robust model training
- Real-Time Performance: Achieved 30+ FPS inference with stable autonomous driving in simulator environment
- Computer Vision: Mastered image preprocessing techniques including color space conversion, cropping, and normalization
- Production System: Developed complete training and deployment pipeline with visualization and monitoring tools
💡 Key Features
🔬 Technical Deep Dive
The project demonstrates mastery of deep learning for autonomous systems through implementation of the NVIDIA architecture. The preprocessing pipeline converts BGR images to YUV color space (preferred for CNNs), crops unnecessary regions, applies Gaussian blur for noise reduction, and resizes to the standard 66×200 input size. The data augmentation strategy prevents overfitting by artificially expanding the training set with realistic variations in lighting, position, and orientation. The trained model achieves low validation loss (0.03) and successfully generalizes to unseen track sections, demonstrating effective learning of steering behavior from visual input alone.
Related Projects
Production-ready AI assistant with reasoning agent, voice input/output (OpenAI Whisper & TTS), RAG document Q&A, cloud storage, and 7 integrated tools. Features conversational context awareness, natural voice interaction with 6 voice personalities, lazy loading, and ChatGPT-style session management.
A Python-based motion detection and viewport tracking system that simulates a "virtual camera" for sports video analysis using computer vision techniques.