A robotics perception engineering project focused on building 3D spatial understanding, multi-object tracking, and temporal perception using synchronized cameras, LiDAR, radar, IMU, and GNSS.
Built using CARLA, ROS2, OpenCV, YOLO, LiDAR-based 3D detection, and a C++ perception core, the project progresses from camera–LiDAR integration and multi-camera perception to sensor fusion, unified spatial perception, cross-camera object association, 2D/3D multi-object tracking, ego-motion estimation, and surrounding-object motion estimation. The latest phase ports the computational perception pipeline to C++.
The project culminates in a unified multi-sensor perception stack for 360° spatial perception and dynamic-scene understanding, with its computational core implemented in C++ and exposed to ROS2 through pybind11.
This project builds upon the 2D perception stack developed in the companion repository:
ROS2 Autonomous Perception Stack — 2D Perception
Consolidated multi-camera RGB perception, LiDAR-based 3D localization, radar-based motion estimation, GNSS/IMU ego-motion, and temporal tracking into a unified spatial perception stack with BEV visualization and real-time monitoring.
| Category | Technologies |
|---|---|
| Simulation | CARLA 0.9.15 |
| Robotics Middleware | ROS2 Humble |
| Computer Vision | OpenCV |
| Object Detection | YOLOv8m-seg INT8 / YOLO26m-seg FP16 |
| 3D Detection | PointPillars / OpenPCDet |
| 2D Tracking | ByteTrack |
| 3D Tracking | AB3DMOT |
| 3D Sensor | LiDAR |
| Motion Sensors | Radar / IMU / GNSS |
| Programming Languages | Python / C++ |
| C++ Core | Eigen / OpenCV (C++) / pybind11 / CMake |
| Communication | CycloneDDS |
| Environment | Windows 11 + WSL2 Ubuntu 22.04 |
RGB Cameras + LiDAR + Radar + GNSS + IMU
│
▼
Sensor Synchronization [Python · ROS2]
│
▼
Multi-Camera Perception [Python · TensorRT]
Detection + Segmentation
│
▼
ByteTrack [Python]
2D Object Tracking
│
▼
Camera–LiDAR Fusion [C++]
3D Object Localization
│
▼
360° Spatial Perception
│
▼
Cross-Camera Object Association
│
▼
Unified World Representation [C++]
world_objects
│
┌─────────┴─────────┐
▼ ▼
Radar Object Motion GNSS/IMU Ego Motion [C++]
Range + Velocity Position + Heading
Motion State Speed + Yaw Rate
│ │
└─────────┬─────────┘
▼
Temporal Scene Perception
│
▼
2D / 3D Multi-Object Tracking
│
▼
BEV + Dashboard Visualization [C++ BEV prep · Python dashboard]
│
▼
Dynamic Scene Understanding
Stages without a language tag are not part of the C++ core.
Python / ROS2
Sensor sync → YOLO/TensorRT → ByteTrack
│
▼ one pybind11 call per frame
C++ PerceptionOrchestrator
LiDAR projection → object clouds → distance/position
→ track-ID attachment → radar fusion (front)
→ counts · nearest objects · BEV · GNSS · IMU
│
▼ single result dictionary
Python
Dashboard payload · metrics · WebSocket · CSV recording
Established the foundation for spatial perception by integrating LiDAR with RGB cameras, calibrating multi-sensor geometry, associating 2D detections with 3D point clouds, and preparing the perception pipeline for object-level distance estimation.
-
✅ 7A — LiDAR Integration
- Integrated a 32-channel LiDAR sensor with the RGB perception pipeline.
- Published synchronized
PointCloud2data and validated real-time visualization. - 📁 View Phase 7A
-
✅ 7B — Camera–LiDAR Calibration
- Calibrated RGB camera and LiDAR sensors using intrinsic and extrinsic parameters.
- Implemented point cloud projection and verified exact timestamp synchronization.
- 📁 View Phase 7B
-
✅ 7C — 2D–3D Association
- Associated projected LiDAR points with YOLOv8 segmentation masks to generate object-specific point clouds.
- Developed deterministic recording and offline replay pipelines for repeatable perception experiments.
- 📁 View Phase 7C
-
✅ 7D — Object Distance Estimation
- LiDAR-based object distance estimation
- Monocular distance estimation
- Camera–LiDAR distance fusion
- 📁 View Phase 7D
Expanded the perception pipeline from a single forward-facing camera to a synchronized 360° multi-sensor perception system by integrating multiple RGB cameras and LiDAR. This phase establishes the foundation for surround perception, multi-camera sensor fusion, and unified environmental understanding.
-
✅ 8A — 360° Camera–LiDAR Perception
- Developed a synchronized 360° perception pipeline using four RGB cameras and a 32-channel LiDAR sensor.
- Implemented deterministic recording, offline replay, multi-camera perception, and LiDAR projection for repeatable perception experiments.
- 📁 View Phase 8A
-
✅ 8B — 360° Panoramic Distance Estimation
- Extended the 360° perception pipeline with panoramic image generation and LiDAR-based object distance estimation.
- Implemented multi-camera object detection, camera–LiDAR projection, object-level point association, cylindrical panoramic stitching, and distance-aware surround visualization.
- 📁 View Phase 8B
Developed a unified spatial perception framework by transforming synchronized multi-camera detections and LiDAR observations into a common ego-centric world representation. This phase establishes the foundation for spatial scene understanding through object localization, Bird's-Eye View generation, and cross-camera object reasoning.
-
✅ 9A — Unified Spatial Perception
- Localized detected objects from four synchronized RGB cameras into a unified ego coordinate frame using Camera–LiDAR fusion.
- Generated a unified Bird's-Eye View (BEV) and validated coordinate transformations through camera-specific yaw correction.
- 📁 View Phase 9A
-
✅ 9B — Cross-Camera Object Merging
- Associated duplicate object detections across overlapping camera views using bearing-based overlap filtering and Hungarian assignment.
- Merged duplicate observations into a unified object representation for consistent 360° spatial perception.
- 📁 View Phase 9B
Extending the perception pipeline with temporal multi-object tracking using YOLO26m TensorRT FP16 and ByteTrack.
-
✅ 10A — 2D Multi-Object Tracking
- Integrated YOLO26m TensorRT FP16 with the official ByteTrack implementation for persistent 2D object identities across consecutive frames.
- Implemented track lifecycle management and tracking-focused visualization using synchronized offline replay data.
- 📁 View Phase 10A
-
✅ 10B — 3D Multi-Object Tracking
- Integrated geometry-based and PointPillars-based 3D detection with AB3DMOT for persistent 3D object tracking.
- Enabled modular comparison of 3D detection approaches using a common tracking and visualization framework.
- 📁 View Phase 10B
-
✅ 10C — Multi-Camera 3D Multi-Object Tracking
-
Extended 3D tracking to four synchronized RGB cameras and LiDAR with geometric cross-camera association and duplicate object merging.
-
Integrated AB3DMOT to maintain persistent 3D Track IDs across the unified 360° ego-centric scene.
-
Extending the perception pipeline with ego and surrounding-object motion estimation using synchronized GNSS, IMU, camera, and radar measurements.
-
✅ 11A — Ego Motion Estimation
- Estimated ego position, speed, acceleration, heading, and yaw rate using synchronized GNSS and IMU measurements.
- Implemented ego motion-state classification and trajectory visualization using deterministic offline replay data.
- 📁 View Phase 11A
-
✅ 11B — Surrounding Object Motion Estimation
- Estimated surrounding-object range, bearing, radial velocity, and motion state by geometrically fusing synchronized Front Camera and Radar measurements.
- Implemented camera–radar association and motion-state classification for detected objects using deterministic offline replay data.
- 📁 View Phase 11B
Consolidating the complete perception pipeline into a unified multi-sensor stack that combines multi-camera RGB, LiDAR, radar, GNSS, and IMU data for spatial understanding, object tracking, motion estimation, and real-time visualization.
- Integrated multi-camera object detection and segmentation.
- Combined LiDAR-based 3D object localization with camera perception.
- Incorporated radar-based range, bearing, radial velocity, and motion-state estimation.
- Integrated GNSS/IMU-based ego-motion estimation.
- Unified perception outputs through the central
world_objectsrepresentation. - Added BEV visualization, object statistics, ego-motion information, and system monitoring through a perception dashboard. 📁 View Phase 12
Porting the computational core of the unified perception stack from Python to C++ and exposing it through a single pybind11 call per frame, while ROS2, YOLO/TensorRT inference, and ByteTrack stay in Python.
- Ported LiDAR projection, mask-based object cloud extraction, and distance/position estimation to C++ (Eigen, OpenCV).
- Moved track-ID attachment, front-camera radar fusion, object counting, nearest-object and BEV preparation to C++.
- Reimplemented GNSS (geodetic → ENU) and IMU ego-motion estimation in C++.
- Unified all C++ stages behind a single
PerceptionOrchestrator, reducing 23 Python ↔ C++ calls per frame to one. - Kept the Phase 12 output schema, so the dashboard payload and CSV recorder work unchanged.
- Ported incrementally, checking each stage against the Python reference before moving on. 📁 View Phase 13
Integrated a 32-channel LiDAR sensor into the ROS2 perception pipeline and validated synchronized PointCloud2 visualization in RViz2.
Projected LiDAR points onto synchronized RGB images through camera calibration, coordinate transformation, and perspective projection.
Associated projected LiDAR points with YOLOv8 segmentation masks to generate object-specific point clouds using a deterministic offline replay pipeline.
Estimated object distances using monocular camera geometry, LiDAR point clouds, and camera–LiDAR sensor fusion.
Established a synchronized 360° perception pipeline using four RGB cameras and LiDAR with deterministic recording, offline replay, and unified surround-view perception.
Extended the 360° Camera–LiDAR perception pipeline with panoramic image generation, LiDAR-based object distance estimation, and distance-aware surround visualization using synchronized multi-camera perception.
Unified spatial perception with ego-coordinate object localization and Bird's-Eye View generation from synchronized multi-camera and LiDAR observations.
Bird's-Eye View visualization used to validate ego-coordinate transformations and correct left and right camera yaw, resulting in consistent object localization across all synchronized camera views.
Associated duplicate object detections across overlapping camera views using bearing-based overlap filtering and Hungarian assignment.
Persistent 2D object identities maintained across consecutive frames using YOLO26m TensorRT FP16 and ByteTrack on synchronized offline replay data.
Persistent 3D object identities maintained across consecutive LiDAR frames using PointPillars 3D detection and AB3DMOT tracking with synchronized front-camera visualization.
Unified 360° 3D object tracking using four synchronized RGB cameras, LiDAR, geometric cross-camera association, and AB3DMOT with persistent 3D Track IDs.
Ego position, speed, acceleration, heading, yaw rate, and motion state estimated from synchronized GNSS and IMU data with front-camera trajectory visualization.
Object range, bearing, radial velocity, and motion state estimated from synchronized camera and radar data with camera–radar association and radar-beam visualization.
Detailed planning, development notes, experiments, and engineering decisions are maintained separately:
roadmap/README.md