Multimodal Object Detection with YOLOv11
Overview
Developed as the final project for the Artificial Vision course at University of San Andrés. We built a reproducible multimodal object detection pipeline leveraging YOLOv11, comparing detection robustness across RGB visual cameras, thermal infrared sensors, and multi-spectral fusion strategies in adverse visual conditions.
Key Features
- Multi-Spectral Sensor Fusion: Evaluated RGB, thermal infrared, GST, HST, and late-fusion ensemble architectures for robust target detection under low lighting and smoke.
- High-Performance TensorRT Acceleration: Exported PyTorch YOLOv11 models via ONNX and compiled optimized TensorRT engines with FP16 half-precision kernels.
- Hardware Deployment on RTX 3090: Benchmarked execution on an NVIDIA GeForce RTX 3090, slashing memory footprint and enabling ultra-low-latency real-time inference.
- Comprehensive Evaluation Metrics: Evaluated confusion matrices, Precision-Recall (PR) curves, and F1-score curves across IoU thresholds to balance false positives and recall.
Architecture & Implementation
Dual-stream data loaders synchronize RGB and thermal frames. YOLOv11 heads process fused feature representations. The trained PyTorch model (.pt) is converted into ONNX format (.onnx) with dynamic axes, and compiled with NVIDIA TensorRT (trtexec) into an optimized .engine serialized plan.
Results & Performance
The fine-tuned TensorRT FP16 engine delivered high-throughput real-time inference on the RTX 3090 while exhibiting significant qualitative and quantitative robustness over baseline RGB detectors in low-visibility scenarios.