Optical phenotyping using label-free microscopy and deep learning

Significance Tissue phenotyping plays a critical role in biomedical research and clinical applications by providing insight into the structural and functional characteristics of tissues that can characterize clinical behavior and identify therapeutic targets. However, conventional phenotyping techniques are destructive, time-intensive, and expensive, posing challenges for both efficiency and widespread use. Aim We aim to develop an optical phenotyping approach in pancreatic cancer specimens using label-free multiphoton microscopy combined with spatial transcriptomics and deep learning. Approach We measure and co-register a dataset comprised of spatial transcriptomics, autofluorescence, and second harmonic generation microscopy. We then cluster tissue subregions into meaningful phenotypes using transcriptomic signatures. We evaluate three different classification models to predict phenotype based on label-free imaging data, and we assess generalizability and prediction accuracy. Result Our deep-learning classification model achieves over 89% accuracy in classifying six tissue types using label-free microscopy images. The one-versus-rest area under the curve (AUC) values for all classes approach 1, confirming the robustness of our model. Conclusion We demonstrate the feasibility of optical phenotyping in distinguishing the structural and functional characteristics of pancreatic cancer specimens. Integrating additional gene-expression data or complementary label-free imaging modalities, such as fluorescence lifetime imaging microscopy, holds the potential to further enhance its accuracy and expand its applications in clinical research and diagnostics.