Attn-NucleiSeg: Cross-Attention Fusion with Adaptive Scale Selection for Nuclei Segmentation in HE Images

Abstract

Nuclei instance segmentation in histopathology images is an important yet challenging task for cancer diagnosis and treatment planning. Existing methods often rely on single-encoder architectures that may have difficulty capturing both local morphological details and global contextual information, while fixed receptive fields may not adequately represent nuclei with different sizes and appearances. To address these challenges, we propose Attn-NucleiSeg, an integrated framework that combines three key components: a dual-encoder architecture based on ResNet-50+FPN and Swin Transformer for learning complementary local and global features, a Cross-Modal Attention Fusion (CAFM) module for interaction between CNN and Transformer features, and a Dynamic Receptive Field Selection (DRFS) module that adapts the receptive field using dilation rates of 1, 3, and 5. Experiments on the PanNuke dataset show that Attn-NucleiSeg achieves the highest detection performance, with a precision of 0.86, recall of 0.83, and F₁,d of 0.84. It also obtains the highest average PQ across all five evaluated nuclear categories, with scores of 0.431, 0.582, 0.425, 0.150, and 0.576 for Inflammatory, Neoplastic, Connective, Dead, and Epithelial nuclei, respectively. On the MoNuSeg validation set, Attn-NucleiSeg achieves the highest scores among the compared methods, including 0.680 bPQ, 0.827 precision, 0.882 recall, 0.841 F₁,d, and 0.729 Jaccard. These results indicate that Attn-NucleiSeg can provide consistent nucleus detection and segmentation performance across different histopathology datasets.