Ground truth construction and parameter tuning for the detection of sleep spindle timing in rodents

Background: The precise detection of cortical sleep spindles is critical to basic research on memory consolidation in rodents. Previous research using automatic spindle detection algorithms often lacks systematic parameter variations and validations. New Method: We present a method to systematically tune and validate algorithm parameters in automatic spindle detection algorithms using a moderate number of human raters. Results: Comparing a Hilbert transform-based algorithm to a ground truth constructed by six human raters, this method produced a parameter set yielding an F1 score of 0.82 at 10 ms resolution. The algorithm performance fell within the range of human agreement with the ground truth. Both human and algorithm failures arose largely from disagreement in spindle boundaries rather than spindle occurrence. With no additional tuning, the algorithm performed similarly in recordings from different days or rats. Comparison with existing methods: Most spindle detection algorithms do not perform systematic parameter variations and validation using a ground truth. To our knowledge, our study is the first in which rodent spindle data is scored by humans, and in which an automatic spindle detection algorithm is evaluated with respect to this ground truth. The rodent data from this study make it possible to compare our algorithm with others previously tested on human data. Conclusions: We present a general ground truth based approach for the tuning and validation of spindle extraction algorithms and suggest that algorithms aimed at extracting precise spindle timing in rats should use a systematic approach for parameter tuning.