Uncovering clinical utility and model behaviour of large language models in attention deficit hyperactivity disorder diagnosis

Background

Large language models (LLMs) have demonstrated strong performance across a range of natural language processing tasks, including text generation and classification. Their scalability and domain-agnostic training enable promising applications in medical contexts. However, clinical decision-making imposes substantially higher requirements on reliability, objectivity, and transparency. LLM performance is known to vary as a function of model size, temperature, prompting strategy, and task formulation, limiting generalizability—particularly in domain-specific applications such as mental health diagnostics. Prior work has predominantly relied on simplified task paradigms (e.g., binary classification or multiple-choice formats), which insufficiently capture the complexity of real-world clinical assessments. Furthermore, LLMs exhibit systematic overconfidence, raising concerns regarding the interpretability and trustworthiness of their outputs. Uncertainty quantification (UQ) has been proposed as a potential mechanism to mitigate these limitations by enabling self-assessment of model outputs. However, systematic evaluations of UQ in clinically realistic, non-deterministic settings remain scarce.

Methods

This study investigates the diagnostic performance, reliability, and uncertainty estimation of LLMs in ADHD assessment using n = 158 transcripts of the structured Wender-Reimherr Interview. A three-level evaluation framework was applied to assess clinical utility comprehensively. We implemented a single-turn chain-of-thought prompting strategy to minimize inference variability. First, diagnostic accuracy (balanced accuracy, sensitivity, specificity) was examined across seven ADHD symptom criteria and the overall diagnosis, comparing multiple LLM configurations against expert consensus. Second, the best-performing configuration was evaluated for agreement with human clinicians at the item and diagnosis level, including assessment of systematic rating biases. Third, model-generated uncertainty estimates were tested for their ability to identify unreliable outputs and flag cases warranting clinician review.

Results

Preliminary exploratory analyses using a locally deployed open-source LLM (Qwen3-32B) indicate substantial agreement with expert ratings at the diagnostic level (Cohen’s κ = 0.61; accuracy = 81%). Analyses at the level of the overall assessment of the criteria show moderate-to-strong associations with human ratings (Spearman ρ ranging from 0.63 to 0.75, all p < .001), with the highest correlation observed for impulsivity. Criterion-level sum scores demonstrated higher correlations (mean ρ = .78, SD = .06), indicating a stronger association compared to aggregated global ratings. Performance was sensitive to temperature settings, with optimal results observed at low temperature (T = 0) and notable degradation at higher values (κ = 0.61 at T = 0, κ = 0.49 at T = 1.2).

Further Analyses: Final analyses will extend these findings by systematically comparing multiple model sizes and configurations, and by quantifying the discriminative capacity of uncertainty estimates (e.g., ROC-AUC) to identify clinically unreliable outputs. We hypothesize that uncertainty scores may lead to useful thresholds for determining whether human supervision is required.

Conclusions

This study provides a clinically grounded evaluation of LLMs as diagnostic support tools in mental health. By integrating performance, reliability, and uncertainty analyses, it aims to identify conditions under which LLM outputs may be considered sufficiently trustworthy for clinical assistance, and where human oversight remains necessary. The findings further highlight the importance of combining diagnostic accuracy with uncertainty-aware mechanisms to ensure safe and interpretable deployment in clinical workflows.

References

No references