2026 ECNP Congress

PS03-3170

Can we trust algorithms in sleep research? Evaluating automated sleep spindle detection in simultaneous EEG-fMRI recordings

Vivienne Matev — G. Dörl, K. Al Barede, P. Falb, S. Tüchler, P. Stöhrmann, M. Techen, J. Lenz, M. Klöbl, R. Lanzenberger · Medical University of Vienna, Department of Psychiatry and Psychotherapy, Vienna, Austria

2026 ECNP Congress

Introduction

Decreased sleep spindle activity has been associated with certain psychiatric conditions, such as schizophrenia [1]. The detection of these sleep spindles is traditionally done by visual inspection of Electroencephalography (EEG) recordings by trained experts. However, the manual detection of spindles is highly subjective and time consuming, which is why automated detection methods have been gaining popularity in recent years. Nevertheless, no standardized algorithm has been established to accurately detect sleep spindles in EEG data that has been acquired during MRI measurements. Thus, the aim of this analysis is to compare three spindle detection algorithms against manual scoring using simultaneously acquired EEG-fMRI data, in order to facilitate future data analysis in sleep research.

Methods

The EEG data of 23 healthy volunteers (mean age 25 ± 3.7 years, 11 females), who reached at least N2 sleep during a 60-minute EEG-fMRI measurement, was used in this analysis. Scans were acquired using a 3T MRI scanner and an MR-compatible EEG system. EEG preprocessing, sleep staging and manual spindle detection was carried out using BrainVision Analyzer 2.3 software. Sleep stages and sleep spindles were scored according to the AASM Scoring Manual. Automated spindle detection was carried out in MATLAB R2023a using three different methods: time-frequency (TF) peak detection [2] based on the Wamsley detector (“A1”) with a threshold set at 80%, TF peak detection using k-means clustering (“A2”) [2], and a method using complex demodulation (“A3”) [3]. Sleep stages N2 and N3 were included in the analysis and C3/C4 channels were analyzed. Duplicate spindles scored in both, C3 and C4 channels, were counted as one event. Events from manual scoring were used to define true positive and false positive spindle events. Performance of detection methods was assessed by computing the F1 scores – the harmonic mean of precision (the proportion of automatically detected events that were also hand scored) and recall (the proportion of hand scored events that were also automatically detected). F1 scores were then compared to the expert scoring consensus (F1 = 0.61 ± 0.06) established in a study evaluating intra-expert reliability in sleep spindle scoring [4].

Results

In total, 2277 spindles were detected using manual scoring, 1852 with algorithm A1, 4640 using algorithm A2 and 1210 with algorithm A3. Algorithm A1 showed the best performance with an average F1 score of 0.5 and highest precision value of 0.54. Algorithm A2 reached the highest recall value of 0.84 and an F1 score of 0.47. A3 showed the lowest performance with an F1 score of 0.37. Comparison of the algorithm performance to the reference expert scoring consensus indicated superiority of manual expert scoring compared to automated spindle detection.

Conclusions

Our results suggest that manual scoring of spindle events likely outperforms the automated methods discussed here. However, the expert scoring consensus used in this analysis is based on EEG data recorded outside the MRI environment. Thus, to improve the interpretative value and potentially the performance of automated detection methods, a manual scoring consensus of simultaneous EEG-MRI recordings will be conducted in further analyses.

References

  1. Ferrarelli F, Peterson MJ, Sarasso S, Riedner BA, Murphy MJ, Benca RM, and others. 2010. Thalamic dysfunction in schizophrenia suggested by whole-night deficits in slow and fast spindles. Am J Psychiatry 167(11):1339–48.
  2. Dimitrov T, He M, Stickgold R, Prerau MJ. Sleep spindles comprise a subset of a broader class of electroencephalogram events. Sleep. 2021 Sep 13;44(9):zsab099. doi: 10.1093/sleep/zsab099. PMID: 33857311; PMCID: PMC8436142.
  3. Ray LB, Sockeel S, Soon M, Bore A, Myhr A, Stojanoski B, Cusack R, Owen AM, Doyon J and Fogel SM (2015) Expert and crowd-sourced validation of an individualized sleep spindle detection method employing complex demodulation and individualized normalization. Front. Hum. Neurosci. 9:507. doi: 10.3389/fnhum.2015.00507
  4. Wendt SL, Welinder P, Sorensen HB, Peppard PE, Jennum P, Perona P, Mignot E, Warby SC. Inter-expert and intra-expert reliability in sleep spindle scoring. Clin Neurophysiol. 2015 Aug;126(8):1548-56. doi: 10.1016/j.clinph.2014.10.158