2026 ECNP Congress

PS01-1030

A small open language model captures linguistic signatures of mental health conditions in social media posts

Gianluca De Franceschi — G. Torino, P. Brambilla, E. Maggioni · Politecnico di Miano, Department of Electronics- Information and Bioengineering, Milano, Italy

2026 ECNP Congress

Introduction

Small open language models (≤1.5 B parameters) enable privacy-preserving, locally deployable mental-health text analysis of social media posts, yet systematic evaluation of their embedding-space structure and decision interpretability is lacking. Most explainability work targets opaque embedding dimensions rather than patients’ actual words. A further gap is that performance is typically measured on text in which users self-disclose their diagnosis, inflating accuracy and obscuring the latent linguistic signal.

Aims

We evaluate whether a small open language model produces clinically meaningful embedding-space structure and interpretable linguistic markers when explicit condition terms are removed.

Methods

We retrieved the test split of the SWMH corpus [1] (n = 10,883 Reddit posts; five classes: depression, SuicideWatch, anxiety, bipolar, offmychest) and defined two binary tasks: bipolar-vs-depression and anxiety-vs-depression. Posts containing explicit condition terms (“bipolar”, “depression”, “depressed”, “anxiety”, “anxious”) were excluded, and classes were balanced by random undersampling, yielding n = 1,428 posts for bipolar-vs-depression and n = 1,108 for anxiety-vs-depression. We extracted mean-pooled embeddings from Qwen2.5-1.5B-Instruct. Three analyses were performed: (i) embedding-space geometry via UMAP, with class separability quantified by silhouette coefficient (SC) and Davies–Bouldin (DB) index; (ii) supervised classification with XGBoost using stratified 5-fold cross-validation and a held-out 20% test split (accuracy, ROC-AUC); (iii) word-level explainability via occlusion-based attribution [2,3], computing importance from the L2 change in predicted probability after leave-one-out re-embedding, filtered at ≥5% document frequency [4]. We also computed directional importance, signed scores indicating which class each word favoured.

Results

Embedding spaces showed weak-to-moderate separability, stronger for bipolar-vs-depression (SC 0.158, DB 1.889) than anxiety-vs-depression (SC 0.050, DB 4.155); UMAP confirmed visually separable regions despite modest silhouette values. Classification was substantially above the 50% baseline: bipolar-vs-depression reached hold-out accuracy 78.3% (CV 79.7% ± 2.2%) and AUC 0.89 (CV 0.88 ± 0.02); anxiety-vs-depression reached 72.5% (CV 74.3% ± 2.6%) and AUC 0.79 (CV 0.82 ± 0.02). Directional occlusion recovered clinically meaningful markers without condition labels. Bipolar-favouring words pointed to treatment and somatic/circadian symptomatology (“symptoms”, “taking”, “sleep”, “manic”, “sick”, “head”); depression-favouring words captured low mood, hopelessness and isolation (“alone”, “tired”, “hate”, “life”, “never”, “world”). Anxiety-favouring words emphasised acute affective–cognitive symptoms and situational triggers (“panic”, “thoughts”, “hours”, “school”, “friend”), while depression-favouring words again reflected absolutist, hopeless language (“never”, “only”, “won’t”, “kill”), consistent with established markers of depressive phenomenology [5].

Conclusions

A 1.5 B-parameter open language model produces clinically meaningful embedding-space structure even when explicit condition terms are removed, supporting AUC up to 0.89. Occlusion attribution identifies symptom, treatment and circadian markers aligned with clinical phenomenology, showing interpretation is feasible at the level of patients’ actual words. Geometry–classification co-variation (SC 0.158–0.050 tracking AUC 0.79–0.89) suggests unsupervised separability metrics may serve as efficient proxies for classifier performance. Limitations include self-reported diagnosis, modest sample sizes, single-language analysis, and the correlational nature of occlusion importance. These methods provide a framework for evaluating small open language models in privacy-sensitive mental-health applications and support transparent, locally deployable decision-support tools.

References

  1. Ji S, Li X, Huang Z, Cambria E. Suicidal ideation and mental disorder detection with attentive relation networks. Neural Computing and Applications. 2022;34:10309–10319.
  2. Zeiler MD, Fergus R. Visualizing and understanding convolutional networks. European Conference on Computer Vision (ECCV). 2014:818–833.
  3. Li J, Monroe W, Jurafsky D. Understanding neural networks through representation erasure. arXiv:1612.08220. 2016.
  4. Ribeiro MT, Singh S, Guestrin C. “Why should I trust you?” Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016:1135–1144.
  5. Al-Mosaiwi M, Johnstone T. In an absolute state: Elevated use of absolutist words is a marker specific to anxiety, depression, and suicidal ideation. Clinical Psychological Science. 2018;6(4):529–542.