Large language model-assisted screening of positive and negative syndrome scale items in first-person accounts of schizophrenia spectrum disorders

Background

Schizophrenia spectrum disorders (SSDs) are characterised by marked heterogeneity in symptom expression, spanning positive, negative, cognitive, affective, and consciousness-related disturbances [1]. Although the Positive and Negative Syndrome Scale (PANSS) is one of the most widely used instruments for the assessment of psychotic symptoms [2], it was developed as a clinician-rated scale and may not fully capture the phenomenological nuance of first-person symptom narratives. Large language models (LLMs) may offer a scalable approach to the preliminary screening of symptom content in patient-reported text, but their ability to identify PANSS-relevant symptoms in first-person accounts of SSDs remains unclear.

Aims

This study aimed to evaluate whether a LLM could support the preliminary screening of PANSS items in first-person accounts from people with SSDs by comparing model-assigned primary items with expert human coding. A secondary aim was to identify and quantify symptom content reported by patients that was not adequately represented by PANSS categories.

Methods

This methodological study was nested within a systematic review of lived experience in SSDs (PROSPERO: CRD420251069446). Qualitative data were derived from the 52 studies included in that review and consisted of first-person quotations from people with SSDs. Before analysis, the dataset was cleaned by removing quotations that did not directly describe symptoms, such as excerpts referring more broadly to illness experience or to non-symptom aspects. The final cleaned dataset comprised 673 quotations. A structured LLM-assisted coding pipeline was developed using a reference file including all 30 PANSS items and a not applicable category. For each quotation, the model was instructed to assign one primary PANSS item or return not applicable when no item was sufficiently supported. Agreement analyses in this preliminary phase were conducted using ratings from one expert human coder compared with model output. Agreement on primary PANSS item assignment was assessed using raw agreement, unweighted Cohen’s kappa with 95% confidence intervals (CI), and Gwet’s first-order agreement coefficient as a sensitivity analysis. A second validation phase comparing two independent human coders versus the LLM is planned.

Results

Agreement between the human coder and the LLM was very high. Raw agreement was 89.7%. Cohen’s kappa was 0.876 (95% CI 0.848–0.903), indicating excellent agreement. This was confirmed by Gwet’s first-order agreement coefficient (0.894, 95% CI 0.869–0.917). A subset of quotations contained symptom content judged to fall outside the PANSS framework. The most frequent additional categories were derealization (n = 7), disturbances of memory (n = 6), suicidal behaviour (n = 5), expanded consciousness (n = 4), affective lability (n = 3), depersonalization (n = 3), and self-harm (n = 3), with less frequent categories including compulsive actions, disturbances of sleep, pressured thinking, and narrowed consciousness.

Conclusions

These findings suggest that LLMs may be useful as first-pass screening tools for identifying PANSS-relevant symptom content in first-person accounts of SSDs. At the same time, the results indicate that patient-reported psychopathology is not fully exhausted by PANSS categories. LLM-assisted screening may therefore support structured symptom extraction while also helping to highlight clinically meaningful experiential content beyond conventional rating scales.

References

1. Leucht, S., Siafis, S., McGrath, J.J., McGorry, P., Howes, O.D., Tamminga, C., Carr, R., Bighelli, I., Schneider-Thoma, J., Priller, J., Davis, J.M., 2025. Schizophrenia. Nat. Rev. Dis. Primers 11 (1), 83.
2. Kay, S.R., Fiszbein, A., Opler, L.A., 1987. The positive and negative syndrome scale (PANSS) for schizophrenia. Schizophr. Bull. 13 (2), 261-276.