mmWave Radar and Low Resolution IR Sensor Fusion for Human Parsing

This is a new multi-modal privacy preserving system for human parsing. To address the issues of sparse semantics and multipath reflections in mmWave radar, low-resolution infrared sensors are utilized to supplement thermal information for achieving more robust human semantic segmentation. A unique deep neural network, MIRaSeg, was designed for the fusion of multimodalities. The feature of the two modalities was extracted through a customized feature extraction module. The NLN network was utilized to achieve the fusion and alignment of features, and finally semantic segmentation was realized through joint training.