Hierarchical, model-based merging of multiple fragments for improved three-dimensional segmentation of nuclei

Background: Automated segmentation of fluorescently labeled cell nuclei in three-dimensional confocal images is essential for numerous studies, e.g., spatiotemporal fluorescence in situ hybridization quantification of immediate early gene transcription. High accuracy and automation levels are required in high-throughput and large-scale studies. Common sources of segmentation error include tight clustering and fragmentation of nuclei. Previous region-based methods are limited because they perform merging of two nuclear fragments at a time. To achieve higher accuracy without sacrificing scale, more sophisticated yet computationally efficient algorithms are needed. Methods: A recursive tree-based algorithm that can consider multiple object fragments simultaneously is described. Starting with oversegmented data, it searches efficiently for the optimal merging pattern guided by a quantitative scoring criterion based on object modeling. Computation is bounded by limiting the depth of the merging tree. Results: The proposed method was found to perform consistently better, achieving merging accuracy in the range of 92% to 100% compared with our previous algorithm, which varied in the range of 75% to 97%, even with a modest merging tree depth of 3. The overall average accuracy improved from 90% to 96%, with roughly the same computational cost for a set of representative images drawn from the CA1, CA3, and parietal cortex regions of the rat hippocampus. Conclusion: Hierarchical tree model-based algorithms significantly improve the accuracy of automated nuclear segmentation without sacrificing speed.