Noise2Map: End-to-End Diffusion Model for Semantic Segmentation and Change Detection

Published 30 Apr 2026 in cs.CV | (2604.27889v1)

Abstract: Semantic segmentation and change detection are two fundamental challenges in remote sensing, requiring models to capture either spatial semantics or temporal differences from satellite imagery. Existing deep learning models often struggle with temporal inconsistencies or in capturing fine-grained spatial structures, require extensive pretraining, and offer limited interpretability - especially in real-world remote sensing scenarios. Recent advances in diffusion models show that Gaussian noise can be systematically leveraged to learn expressive data representations through denoising. Motivated by this, we investigate whether the noise process in diffusion models can be effectively utilized for discriminative tasks. We propose Noise2Map, a unified diffusion-based framework that repurposes the denoising process for fast, end-to-end discriminative learning. Unlike prior work that uses diffusion only for generation or feature extraction, Noise2Map directly predicts semantic or change maps using task-specific noise schedules and timestep conditioning, avoiding the costly sampling procedures of traditional diffusion models. The model is pretrained via self-supervised denoising and fine-tuned with supervision, enabling both interpretability and robustness. Our architecture supports both tasks (SS and CD) through a shared backbone and task-specific noise schedulers. Extensive evaluations on the SpaceNet7, WHU, and xView2 buildings damaged by wildfires datasets demonstrate that Noise2Map ranks on average 1st among seven models on semantic segmentation and 1st on change detection by a cross-dataset rank metric (average F1 primary, IoU tie-break). Ablation studies highlight the robustness of our model against different training noise schedulers and timestep control in the diffusion process, as well as the ability of the model to perform multi-task learning.

Abstract PDF Upgrade to Chat

Authors (3)

Summary

The paper introduces a novel discriminative diffusion process that directly predicts segmentation and change detection outputs from noisy satellite imagery.
It leverages a unified denoising UNet architecture with task-specific noise schedulers to achieve superior F1 and IoU performance across multiple benchmarks.
The approach offers significant computational efficiency, being up to 13.49× faster and considerably smaller than traditional diffusion-based models.

End-to-End Diffusion for Semantic Segmentation and Change Detection in Remote Sensing: The Noise2Map Approach

Introduction

Segmentation and change detection (CD) in remote sensing (RS) are fundamental to environmental monitoring, disaster response, and land-use analysis. However, the spatial heterogeneity and temporal variability inherent to satellite imagery present persistent challenges for both semantic segmentation (SS) and CD. Traditional CNN-based and Transformer-based models have addressed these to varying degrees, but often at the cost of computational efficiency, reliance on extensive pretraining datasets, or limited interpretability.

Recent advances in diffusion models, originally designed for generative tasks, demonstrate substantial potential for learning expressive visual representations through progressive denoising. The Noise2Map framework (2604.27889) directly leverages the diffusion process within an end-to-end discriminative paradigm, enabling both SS and CD with a unified backbone architecture. Unlike prior works focused either on generation or feature extraction via diffusion, Noise2Map introduces targeted noise schedules and direct prediction of discriminative outputs, eschewing costly sampling-intensive inference typical of traditional denoising diffusion probabilistic models (DDPMs).

Technical Overview

Diffusion Formulation for Discriminative Tasks

Noise2Map adapts the diffusion process for direct mapping tasks. For semantic segmentation, intermediate noisy representations $x^{(t)}$ are produced through a variance-scheduled forward process, and the model is trained to predict segmentation masks from these representations across all sampled timesteps. This not only acts as a strong regularizer, exposing the network to a richer distribution of corrupted observations, but also enables robustness to varying image conditions.

For change detection, bi-temporal inputs $[x_{t_1}, x_{t_2}]$ are concatenated. A task-specific noise scheduler morphs these pairs smoothly over $T$ timesteps toward their reversed ordering $[x_{t_2}, x_{t_1}]$ , breaking the symmetry inherent in simple difference-based CD methods. This formulation encodes both the magnitude and directionality of change, equipping the denoiser with temporal trajectory information, rather than static magnitude differences.

Model Architecture

The backbone is a denoising attention UNet with five downsampling/upsampling blocks, ResNet modules, and self-attention at the bottleneck. Fixed, non-learnable noise schedulers control the diffusion process; only the backbone parameters are subject to gradient updates. Timesteps are encoded via sine-based embeddings and provided as inputs along with the noisy image representations. The same model supports both SS and CD, with task-specific noise schedulers and the capability to share representation learning in a multi-task setup.

Training Paradigm

Noise2Map employs a two-stage procedure:

Self-supervised pretraining: The backbone is trained using the DDPM objective to reconstruct clean images from noise using 10,000 unlabeled AID satellite images, learning general-purpose RS representations.
Supervised fine-tuning: The pretrained model is fine-tuned on downstream tasks using weighted cross-entropy loss, structured noise schedules, and bi-temporal or single-image input as appropriate.

A key distinction is that noise is only injected during training. At inference, predictions are made using clean input at the final diffusion timestep, providing practical efficiency by avoiding iterative denoising.

Experimental Results

Datasets and Benchmarks

Experiments were conducted on SpaceNet7, WHU Building, and xView2-wildfire datasets, covering various spatial and temporal RS scenarios. Comparison spanned leading CNN, Transformer, state-space, and diffusion-based architectures (UNet, DeepLabV3+, SegFormer, UPerNet, RS3Mamba, DDPM-CD, ChangeFormer, CGNet-CD, etc.).

Quantitative Results

Noise2Map consistently achieves top aggregate rank for both semantic segmentation and change detection when compared via mean F1 and IoU across all datasets. Notable numerical results include:

WHU-SS: F1 = 95.69, IoU = 92.90
xView2-wildfire-SS: F1 = 86.90, IoU = 78.55
SpaceNet7-CD: F1 = 71.43, IoU = 61.91
xView2-wildfire-CD: F1 = 86.91, IoU = 78.59

Performance improvements over classic baselines (e.g., UNet, DeepLabV3+) are substantial, with increases of up to +5.98 F1 / +7.54 IoU in challenging wildfire detection scenarios.

Efficiency is a key differentiator: compared to the diffusion-based generative baseline (DDPM-CD), Noise2Map is 13.49× faster and 3.85× smaller in terms of parameter count, due to its single-step discriminative inference design.

Ablation and Analysis

Ablations confirm the necessity of the discriminative diffusion formulation for optimal results; removing the timestep/noise process leads to a drop of 3–4% F1 in both tasks.
Domain-specific pretraining: RS-specific pretraining (AID, MAJOR-TOM) yields significant performance boosts over generic ImageNet or random initialization, validating the importance of domain alignment.
Scheduler robustness: Across DDIM, DDPM, and PNDM, the model retains high performance; Heun, designed for continuous-time SDEs, is less effective in this discrete-step discriminative setting.
Multi-task learning: Sharing the backbone across SS and CD heads is positively synergistic only with careful loss weighting; improper balancing leads to negative transfer, but with optimized weights, both tasks can improve.
Interpretability: Inspecting predictions across denoising steps reveals progressive refinement of structural details, and quantitative analysis shows F1 increasing as noise is removed—an implicit measure of the model’s predictive trajectory.

Theoretical Implications and Future Directions

Noise2Map establishes that the denoising trajectory of a diffusion process can encode rich discriminative information, with noise acting as a structured supervisory signal even when the final inference is conducted in a single step. This challenges the paradigm that diffusion models are fundamentally bound to generative or sampling-heavy inference and shows that their representational benefits can be realized efficiently within discriminative pipelines.

The break from iterated sampling to direct prediction introduces opportunities for real-time or large-scale RS deployments, where both speed and explainability are crucial. The model also provides a template for leveraging diffusion structures for additional temporal RS tasks, such as multi-frame progression modeling or multi-modal satellite fusion.

Conclusion

Noise2Map (2604.27889) introduces an efficient, interpretable, and robust discriminative diffusion framework for remote sensing segmentation and change detection. Through domain-specific pretraining, end-to-end supervised learning, and task-aligned noise processes, it sets a new benchmark in both tasks according to cross-dataset F1/IoU metrics. The direct exploitation of the diffusion trajectory for discriminative mapping, together with significant computational gains over generative diffusion baselines, positions Noise2Map as a valuable methodological advance for RS visual understanding.

The findings suggest several promising avenues for future work: scaling to larger and more heterogeneous pretraining datasets, extending to additional downstream or temporally-evolving tasks, and integrating heterogeneous RS modalities within the diffusion discriminative learning paradigm.

Markdown Report Issue