Performance of an MRI-based psychosis risk score varies by symptom burden and brain maturation

Clinical high-risk states for psychosis are heterogeneous, and transition rates vary substantially across cohorts, age ranges, symptom profiles and ascertainment strategies. Prognostic brain imaging models may therefore show acceptable average performance while failing in clinically relevant subgroups. We externally validated a structural magnetic resonance imaging risk score for transition to psychosis [1] and tested whether its discriminative utility varied across developmental, clinical and biological contexts.

We analysed individual participant data from five prospective clinical high-risk for psychosis cohorts, including the European PRONIA cohort, the North American NAPLS-3 cohort, the Munich MUC-FRUE cohort, the Zurich ZInEP cohort and the Basel FePsy cohort. The pooled sample included 1015 participants aged 12–42 years, of whom 132 transitioned to psychosis during follow-up. The structural neuroimaging model trained in a sample of the PRONIA study was evaluated in the discovery cohort and in four independent external cohorts. Discrimination was assessed using area under the receiver operating characteristic curve (AUC) from continuous imaging risk scores, and threshold-dependent performance was assessed using balanced accuracy. To move beyond pooled performance, we mapped local discrimination across age, cognitive basic symptoms, positive symptoms, negative symptoms, psychosocial functioning and BrainAGE using a fixed-range sliding windows approach. Local above-chance performance was assessed using within-cohort label permutation, and landscape-level heterogeneity was summarised using the maximum AUC range. We further tested individual-level moderation using logistic interaction models and non-linear generalised additive model sensitivity analyses.

In the PRONIA discovery cohort, the model showed moderate discrimination (AUC = 0.742, p < 0.001) and balanced accuracy of 0.723. In pooled external validation, discrimination was weaker but above chance (AUC = 0.563, p = 0.011; balanced accuracy = 0.558, p = 0.008), with degraded calibration (Brier score = 0.245; calibration slope = 0.166; intercept = −1.73). External cohort performance was heterogeneous, with moderate discrimination in ZInEP (AUC = 0.650, p = 0.012) and FePsy (AUC = 0.690, p = 0.032), but not in NAPLS-3 or MUC-FRUE. Sliding-window analyses showed structured variation in discrimination, with strongest evidence for heterogeneity along BrainAGE (maximum AUC range = 0.275, p = 0.035) and nominal evidence along age, cognitive basic symptoms and negative symptoms. The bivariate negative symptoms × BrainAGE surface showed significant performance heterogeneity (maximum AUC range = 0.493, p = 0.031). Component decomposition linked higher discrimination in BrainAGE and negative symptom performance landscapes to distinct neuroanatomical mechanisms, particularly temporoparietal converter–non-converter separation and posterior-cerebellar/motor-executive shifts. In logistic moderation models, the imaging risk score was associated with transition (β = 0.271, SE = 0.133, p = 0.042), and the imaging risk score × negative symptoms interaction indicated stronger score–outcome coupling at higher negative symptom burden (β = 0.259, SE = 0.131, p = 0.048).

These findings indicate that structural imaging-based psychosis risk prediction is transportable only to a limited degree on average and that its utility is context-dependent. Heterogeneity-aware validation may help identify where brain imaging risk scores retain clinically meaningful signals and where recalibration or model updating is required before clinical use.

References

1. Koutsouleris N, Dwyer DB, Degenhardt F, Maj C, Urquijo-Castro MF, Sanfelici R, Popovic D, Oeztuerk O, Haas SS, Weiske J, Ruef A, Kambeitz-Ilankovic L, Antonucci LA, Neufang S, Schmidt-Kraepelin C, Ruhrmann S, Penzel N, Kambeitz J, Haidl TK, Rosen M, Chisholm K, Riecher-Rössler A, Egloff L, Schmidt A, Andreou C, Hietala J, Schirmer T, Romer G, Walger P, Franscini M, Traber-Walker N, Schimmelmann BG, Flückiger R, Michel C, Rössler W, Borisov O, Krawitz PM, Heekeren K, Buechler R, Pantelis C, Falkai P, Salokangas RKR, Lencer R, Bertolino A, Borgwardt S, Noethen M, Brambilla P, Wood SJ, Upthegrove R, Schultze-Lutter F, Theodoridou A, Meisenzahl E; PRONIA Consortium. Multimodal Machine Learning Workflows for Prediction of Psychosis in Patients With Clinical High-Risk Syndromes and Recent-Onset Depression. JAMA Psychiatry. 2021 Feb 1;78(2):195-209. doi: 10.1001/jamapsychiatry.2020.3604. PMID: 33263726; PMCID: PMC7711566.