Exploring Fusion Techniques in Multimodal AI-Based Recruitment: Insights from FairCVdb

Despite the large body of work on fairness-aware learning for individual modalities like tabular data, images, and text, less work has been done on multimodal data, which fuses various modalities for a comprehensive analysis. In this work, we investigate the fairness and bias implications of multimodal fusion techniques in the context of multimodal AI-based recruitment systems using the FairCVdb dataset. Our results show that early-fusion closely matches the ground truth for both demographics, achieving the lowest MAEs by integrating each modality’s unique characteristics. In contrast, late-fusion leads to highly generalized mean scores and higher MAEs. Our findings emphasise the significant potential of early-fusion for accurate and fair applications, even in the presence of demographic biases, compared to late-fusion. Future research could explore alternative fusion strategies and incorporate modality-related fairness constraints to improve fairness….

Swati Swati, Arjun Roy and Eirini Ntoutsi. Exploring Fusion Techniques in Multimodal AI-Based Recruitment: Insights from FairCVdb. In the Proceedings of the third European Workshop on Algorithmic Fairness (EWAF’24). 2024.