Software-Hardware Co-Design of Multi Task Deep Neural Networks for Camera Vision System

Abstract

Real-time camera vision on edge devices must run multiple perception tasks simultaneously while staying within tight power and memory constraints. However, single-task detection networks like YOLO series are unable to do so efficiently. In this thesis, we present a software hardware co-design multi-task deep neural network architecture which performs seven visual perception tasks: detection, classification, semantic & instance segmentation, pose, oriented detection, and age-gender estimation, with a single feature backbone and lightweight task specific heads. An Intelligent Preprocessing Module measures scene complexity and runs only the tasks a given frame actuallyneeds, reducingredundantcomputationandpowerconsumption. The architecture was first developed in Python, trained on Kaggle, and demonstrated on an NVIDIA H100 GPU.Its computational core was then translated into synthesizable Verilog and integrated with the Zynq UltraScale+MPSoC.Thisprovideshardware–softwareco-design, with the Processing System handling control while the custom accelerator runs in the programmable logic. On the H100, the model sustains 177–196 FPS at 5.0–5.6 ms per frame with a 4.7 MB backbone. On hardware, it uses 36,472 LUTs, 71,429 registers, and 59 DSP slices at 100 MHz. It meets all timing constraints (WNS +0.358 ns) and consumes 3.532 W total on-chip power, independently verified as 3.590 W in AMD Power Design Manager.