Complete2026
Evaluating detectors on driving scenes
Changing one post-processing setting raised YOLO26m KITTI AP from 26.20 to 27.80 and reversed its comparison with the NMS-free head. A CMPE 249 study of evaluation settings, domain shift, and inference latency.
Completed CMPE 249 course study, October 2026.
Tools and project details
- Status
- Completed CMPE 249 course study, October 2026.
- Stack
- Python
- PyTorch
- Ultralytics
- pycocotools
- TensorRT
- Jetson Orin Nano
- Role
- Evaluation harness, experiments, and analysis
- Inference hardware
- Jetson Orin Nano 8 GB
Overview
- Problem
- How much of a detector’s apparent advantage comes from the evaluation protocol?
- Approach
- I built a common evaluator with dataset mappings and ignored-region handling, then compared predictions on the same images and measured inference latency on a Jetson Orin Nano.
Results
| Metric | Value | Note |
|---|---|---|
| Multi-label vs single-label NMS | +1.60 AP points | YOLO26m, KITTI val, 640 px, four mapped classes; paired bootstrap 95% CI +1.29 to +1.93, 300 resamples. |
| Multi-label NMS vs NMS-free | +0.90 AP points | Same KITTI protocol; paired bootstrap 95% CI +0.57 to +1.15, 300 resamples. Speed is evaluated separately. |
Case study
What I changed
The single-label predictor keeps only one class for each box. Enabling multi-label NMS changed the KITTI result enough to reverse the AP comparison with the NMS-free head. I held the YOLO26m checkpoint and 640 px input fixed and used paired image-level resampling to estimate the difference.
The wider study examines resolution, fine-tuning, annotation gaps, and transfer from COCO to KITTI and nuImages. The contribution is the evaluation protocol and analysis; the detector and VLM architectures are existing models.
Scope of the result
These AP differences apply to four mapped KITTI classes and this configuration. They do not establish a general architecture ranking. Fine-tuning comparisons include single runs; model size and pretraining differ between the VLM and detectors.
The PDF is the October 23 course report. Jetson results measure inference accuracy and latency. The GX10 was used for VLM training and accuracy evaluation. The repository credits the course’s ngdet tutorial and contains the separate evaluation harness.