A Hybrid 3D Watershed Algorithm Incorporating Gradient Cues and Object Models for Automatic Segmentation of Nuclei in Confocal Image Stacks

Background: Automated segmentation of fluorescently-labeled cell nuclei in 3D confocal microscope images is essential to many studies involving morphological and functional analysis. A common source of segmentation error is tight clustering of nuclei. There is a compelling need to minimize these errors for constructing highly automated scoring systems. Methods: A combination of two approaches is presented. First, an improved distance transform combining intensity gradients and geometric distance is used for the water-shed step. Second, an explicit mathematical model for the anatomic characteristics of cell nuclei such as size and shape measures is incorporated. This model is constructed automatically from the data. Deliberate initial over-segmentation of the image data is performed, followed by statistical model-based merging. A confidence score is computed for each detected nucleus, measuring how well the nucleus fits the model. This is used in combination with the intensity gradient to control the merge decisions. Results: Experimental validation on a set of rodent brain cell images showed 97% concordance with the human observer and significant improvement over prior methods. Conclusions: Combining a gradient-weighted distance transform with a richer morphometric model significantly improves the accuracy of automated segmentation and FISH analysis.