Validation of UNISI-CAT, a cognitive test battery for early phase drug development trials and biomarker research

Introduction

The development of digital tests incorporating artificial intelligence applications by design to monitor minimal cognitive changes is a viable strategy to increase the cost-efficiency of therapeutic benefits or safety risk signal detection.

Methods

University of Siena Cognitive Assessment Tool (UNISI-CAT) is a digital test battery currently under development. It consists of three modules. 1) Waves is based on a repeated copy design paradigm and informed by automated image analysis for scoring. It involves a “paper and pencil” reproduction of three target objects (bi-dimensional waveform patterns) by different modalities: open eyes and dominant/preferred hand, open-eyes and contra-lateral hand, and closed eyes with dominant/preferred hand. A Wave assessment protocol produces a gallery of twelve reproductions. For each reproduction, three measures of similarity - Hausdorff’s distance (HD), modified HD (mHD) and HD95% - and spatial dimensions of specific areas are automatically calculated by a customized scoring software; a summary measure of accuracy of reproduction (AoR) based on clinical expert scoring of a predetermined set of twelve graphic markers is also obtained. Aggregate measures of AoR, visuo-graphic motor constructional processing (VGMCP) and spatial dimensions are derived as average score from twelve reproductions. 2) Mirror Verbal Fluency (MVF) is a speeded handwriting task with automated analysis and scoring of word production organization. The MVF score (range 0-56) is the number of words (animals or concrete objects) correctly written in forward and backward (reverse) modality in 120 seconds. 3) Digit Symbol Coding (DSC) is a speeded visuomotor processing task measuring the accuracy (range 0-90) of digit and symbol matching. A field trial was carried out to evaluate UNISI-CAT feasibility of use, administration procedures and psychometric properties of a set of behavioral outcomes. Retest was performed after 6 weeks. Reliability, as measured by Cronbach’s α or Intraclass Coefficient of Correlation (ICC), was considered acceptable for values >.70.

Results

The sample consisted of 44 participants, male (68.18%), aged 26.9(9.0), in apparent good health status, Trail Making Test A mean (SD) 25.5(8.4) and Trail Making Test B mean(SD) 55.4(22.4). Twenty-six participants completed retesting. Waves AoR [score range: 0-120, mean(SD) 16.9(6.3), Cronbach’s α=.72, ICC=.83 (Two-Way Effects Model)] showed no significant association (pairwise correlation, Spearman’s rho and Pearson’s r) with other variables of interest for construct validity, i.e., age, Waves Spatial Extension, DSC, MVF, TMT-A and B; MVF [mean(SD) 27.6(6.2), ICC=.82 ] correlated with TMT-B (rho=-.30, p=.0501) only; DSC [mean(SD) 49.4(9.4), ICC=.76] with age (r =-.347, p=.0201), TMT-A (r =-. 33, p=.027) and TMT-B (r =-.33, p=.030). VGMCP-HD, VGMCP-mHD, and VGMCP-HD95% showed ICC=0.0 and no correlation with AoR. UNISI-CAT administration feasibility was excellent.

Conclusions

UNISI-CAT measures, assessing diverse domains of cognition with known utility in drug research, showed valuable psychometric properties. Of note, Hausdorff’s VGMCP similarity measures did not show valuable reliability properties, suggesting the incorporation of these algorithms did not result in more cost-efficient cognitive assessment. These results can inform the design of future studies on cognitive assessment methodology.

References

None.