AISA-BS: A Multimodal Employment Screening Framework Integrating Transformer-Based Semantic Analysis and IoT-Driven Behavioral Sensing

Abstract

With the rise of digital hiring platforms, it has become increasingly challenging to evaluate job candidates using only resumes and interviews accurately. Traditional screening methods often overlook important behavioral and contextual cues, which can lead to poor hiring decisions. To overcome these limitations, there is a growing need for more comprehensive screening systems. This paper proposes AISA-BS (Artificial Intelligence Semantic Analysis and Behavioral Sensing), a multimodal employment screening framework designed to improve candidate evaluation by combining language analysis with behavioral data collected through IoT devices. AISA-BS leverages Transformer-based NLP models (e.g., BERT) to analyze unstructured text inputs like resumes and interview transcripts. It uses IoT-enabled sensors to capture behavioral data—such as gaze, posture, and stress—from candidates in simulated job environments. These multimodal signals are fused through tensor decomposition and cross-modal attention, and interpreted using a BiLSTM-based behavioral engine for temporal analysis. Experiments were conducted using the DAiSEE dataset, which includes video-based affective state annotations. The proposed model achieved a classification F1-score of 93.2%, reducing the Mean Absolute Error (MAE) to 3.1%, outperforming BERT+MLP and MM-DNN baselines. In conclusion, AISA-BS sets a new benchmark for intelligent, fair, and context-aware employment screening by combining deep semantic insight with behavioral interpretation.