Adaptive multimodal fusion of behavioural signals for objective mental health assessment

Introduction

Mental health disorders, including Attention-Deficit/Hyperactivity Disorder (ADHD), are among the most prevalent neurodevelopmental conditions worldwide and are commonly assessed using subjective clinical methods such as rating scales and structured interviews. These approaches may limit reliability, scalability, and timely access to care [1,2]. Recent advances in Artificial Intelligence (AI) enable the extraction of behavioural markers from multimodal data, including facial dynamics, speech characteristics, and linguistic features, which reflect underlying cognitive and emotional processes. Two key challenges remain: how to effectively represent and integrate heterogeneous multimodal signals, and how to adaptively weight modality contributions to account for variability across individuals.

Aims

This study evaluates whether multimodal behavioural signal representation learning combined with adaptive weighted fusion can improve objective ADHD classification compared with conventional fusion methods.

Methods

Multimodal behavioural features were extracted from three modalities: facial dynamics from video, acoustic speech characteristics from audio, and textual features derived from structured clinical responses. These modalities were selected because they capture complementary behavioural, cognitive, and emotional processes relevant to mental health assessment [3,4]. In this framework, video, audio, and text features were first encoded independently to account for differences in feature type, dimensionality, and scale. Facial, acoustic, and textual features were mapped into a shared latent representation space using modality-specific encoders, allowing the signals to be aligned before multimodal integration.

To combine the modality-specific information, a Joint Weighted Average (JWA) method adaptively combines prediction scores from video, audio, and text at the participant level. In contrast to Linear Weighted Average (LWA), which applies fixed global weights, and Ordered Weighted Average (OWA), which assigns weights according to ranked prediction scores, JWA dynamically learns the relative importance of each modality based on the participant’s multimodal representation. This adaptive weighting enables the model to prioritize the most informative modality for each participant while reducing the influence of noisier or less relevant signals. In the context of ADHD assessment, this is clinically meaningful because symptom expression varies across individuals and may be reflected differently in facial behaviour, speech patterns, and linguistic responses. The fused representation was evaluated using neural network classifiers. Performance was assessed using stratified cross-validation and standard metrics, including accuracy, precision, sensitivity, specificity, F1-score, and ROC-AUC.

Results

This study extends previous video-based ADHD detection work [5]. As shown in the table, the method outperformed others across all reported metrics, suggesting that adaptive participant-level weighting improves multimodal behavioural signal integration.

Fusion Method	Weighting Approach	Video Weight	Audio Weight	Text Weight	Accuracy (%)	Precision**(%)**	Recall**(%)**	F1-Score**(%)**	ROC-AUC
LWA	Fixed modality	0.50	0.25	0.25	78.0	79.2	75.0	77.0	0.85
OWA	Rank-based	0.50	0.30	0.20	80.5	82.0	77.0	79.4	0.87
JWA	Adaptive participant-level	0.55 ± 0.12	0.27 ± 0.10	0.18 ± 0.08	86.2	88.0	82.0	84.7	0.92

Conclusion

Multimodal behavioural signal representation combined with adaptive JWA fusion improved objective ADHD classification compared with conventional fusion approaches. By dynamically weighting audio, video, and text modalities at the participant level, the method supports more flexible and interpretable integration of heterogeneous behavioural signals. These results provide preliminary evidence that adaptive weighted fusion of heterogeneous multimodal representations enhances classification performance.

References

1. P. Song, M. Zha, Q. Yang, Y. Zhang, X. Li, and I. Rudan, “The prevalence of adult attention-deficit hyperactivity disorder: A global systematic review and meta-analysis,” Journal of Global Health, vol. 11, Art. no. 04009, 2021.
2. J. A. Ramos-Quiroga, V. Nasillo, V. Richarte, M. Corrales, F. Palma, P. Ibáñez, M. Michelsen, G. van de Glind, M. Casas, and J. J. S. Kooij, “Criteria and concurrent validity of DIVA 2.0: A semi-structured diagnostic interview for adult ADHD,” Journal of Attention Disorders, vol. 23, no. 10, pp. 1126–1135, 2019.
3. T. Baltrušaitis, A. Zadeh, Y. C. Lim, and L. P. Morency, “OpenFace 2.0: Facial behavior analysis toolkit,” in Proc. 13th IEEE Int. Conf. Autom. Face Gesture Recognit. (FG), 2018, pp. 59–66.
4. F. Eyben, M. Wöllmer, and B. Schuller, “openSMILE: The Munich versatile and fast open-source audio feature extractor,” in Proc. 18th ACM Int. Conf. Multimedia, 2010, pp. 1459–1462.
5. Y. Li, S. M. Naqvi, and R. Nair, “ADHD diagnosis based on action characteristics recorded in videos using machine learning,” Neuroscience Applied, vol. 2, Art. no. 102835, 2023.