Operator Learning for Smoothing and Forecasting

Sign up for free to create and run prompts on this paper using GPT-5.

Overview

This paper is about teaching computers to “fill in the blanks” and “look ahead” in systems that change over time, like the weather. The authors study two tasks:

• Smoothing: using what we can observe to reconstruct what we can’t see over the same period of time.
• Forecasting: using past observations to predict what will happen next.

They build a mathematical foundation that shows when these tasks can be done purely from data (without using the exact physics equations), and they prove that a certain kind of machine learning model—called a neural operator—can learn to do them well.

What questions does the paper ask?

• When we can only observe part of a system, can we reconstruct the hidden parts over the same time window? (Smoothing)
• From past observations, can we predict the future of the part we can observe? (Forecasting)
• Can a data-driven model learn these mappings reliably, in theory, not just in practice?
• What conditions must hold in the real system to make learning possible?

How did they approach the problem?

The authors use a two-step strategy:

1. Show the mapping to be learned actually exists.

• Imagine a machine that takes in a whole curve (a time series) and outputs another curve. In math, that machine is called an operator.
• The paper shows that under an “observability” condition, there really is an operator that:
  • maps the observed part of the system to the hidden part (for smoothing), and
  • maps past observations to future observations (for forecasting).
• Observability, simply put, means: from what you can see, you can figure out what you can’t see. The authors use a classic idea from control theory that provides a practical test (called an observability-rank condition) for whether this is true. If the test passes, nearby states are uniquely determined by the observations.

Analogy: If you watch a car’s speed and how it changes, sometimes you can figure out the engine’s hidden state. That’s observability—different hidden states would lead to different visible behaviors, so you can tell them apart.

2. Prove that neural operators can learn this mapping.

• A neural operator is like a neural network designed to take in a function (a whole time series) and output another function. Transformers (the kind used in LLMs) can be adapted to be neural operators for time series.
• The authors use a “universal approximation theorem” for neural operators. This theorem says: if the true mapping is continuous and you have enough model capacity and data, a neural operator can approximate it as closely as you like.
• Combining step 1 and step 2, they prove the first general results showing that purely data-driven methods can, in principle, learn to do smoothing and forecasting in continuous time.

They also run experiments on well-known test systems (Lorenz ’63, Lorenz ’96, and the Kuramoto–Sivashinsky equation) to show the ideas work in practice.

Key ideas explained simply

• Dynamical system: A set of rules that says how things change over time (like weather patterns evolving day by day).
• Data assimilation: Using observations to estimate the system’s state, including parts you can’t directly observe.
• Smoothing: After collecting data over a time window, reconstruct the hidden parts over that same window.
• Forecasting: Predict what comes next based on what you’ve seen so far.
• Operator: A “function of functions”—it takes a whole time series in and gives a whole time series out.
• Neural operator: A learnable model that maps one time series to another. Think of it as a flexible tool that can learn “how to go from input curves to output curves.”
• Observability: Can you deduce the hidden state from what you can see? If yes, then a unique solution exists nearby.
• Universal approximation: With enough capacity, the model class can approximate any target mapping of the right kind, as closely as you want.

A neat example from the paper: In the Lorenz ’63 system (a classic simplified weather model), if you observe the x-variable over time, you can reconstruct the hidden y and z variables—except for rare cases where x starts exactly at zero. But if you only observe z, you cannot reconstruct x and y uniquely because two different hidden states produce the same z. This shows why observability matters.

What did they find, and why is it important?

Here are the main findings:

• They prove the existence of the “smoothing operator” and the “forecasting operator” under a clear, checkable observability condition. That means the targets you want to learn actually exist and are well-behaved.
• They prove universal approximation results: neural operators (including transformer-based ones) can learn these operators to any desired accuracy on appropriate sets of inputs.
• They demonstrate the approach on benchmark systems (Lorenz ’63, Lorenz ’96, Kuramoto–Sivashinsky), showing that learned models can reconstruct hidden states and make accurate forecasts from observations alone.

Why this matters:

• It provides a rigorous foundation for purely data-driven methods that are becoming popular in areas like weather prediction, where running full physics models can be very expensive.
• It helps explain when and why these learned models will work, guiding practitioners to use them responsibly.
• It connects modern machine learning (transformers/neural operators) with classic control theory (observability), giving a clear set of conditions to check before trusting predictions.

What’s the big picture impact?

This work bridges a gap between practice and theory. Many teams already use data-driven forecasting models, but until now, the math behind when they should work wasn’t fully developed. This paper:

• Shows that if the system is observable (you can infer hidden parts from what you can see), then there is a real, continuous mapping from observations to what you want to know.
• Proves that transformer-like models can learn this mapping in principle.
• Offers a roadmap for building fast, data-driven tools for smoothing and forecasting in complex systems.

In short, the paper gives scientists and engineers stronger reasons to trust and further develop data-driven methods for understanding and predicting dynamical systems—especially when full physics models are unavailable or too slow to run.

Knowledge gaps, limitations, and open questions

Below is a concise, actionable list of what the paper leaves uncertain, missing, or unexplored, to guide future research:

• Noise-free assumption: Extend the theory to observational noise and model error, proving existence/continuity of smoothing and forecasting operators under noisy, discretely sampled measurements and deriving robustness bounds for learned operators.
• Local (not global) guarantees: The observability-rank condition yields only local invertibility; develop methods to characterize, construct, and “stitch” together maximal domains of validity for global continuous smoothing/forecasting maps, and analyze stability near non-observable points (e.g., x=0 in Lorenz ‘63).
• Practical verifiability of observability: Provide data-driven, testable conditions (without access to f,g) to verify observability from trajectories; give constructive procedures to select the minimal derivative order n and linear map L that satisfy the observability-rank condition.
• Dependence on derivatives of p: Replace reliance on high-order time derivatives (unstable under noise/discretization) with derivative-free formulations (e.g., delay embeddings) and rigorously connect to Takens-style embeddings for partial observations.
• From local to the assumed global operators in experiments: Justify existence of the global operators used in Section 5 (defined on all of C([0,T])) from the local inverse-function-theorem results, or explicitly quantify the subset of trajectories on which the operators are well-defined.
• Forecast horizon and chaos: Analyze how approximation and generalization errors scale with forecast horizon τ and system instability (e.g., Lyapunov exponents), including regimes where deterministic point forecasts are ill-posed and distributional forecasts are required.
• Approximation rates (not just existence): Derive quantitative approximation rates for neural operators (e.g., transformer neural operators) in terms of architecture size, regularity k, and input dimension, rather than only universal approximation existence.
• Sample complexity and generalization: Establish learning-theoretic results (e.g., sample complexity, covering numbers, or stability bounds) for operator learning in smoothing/forecasting on compact sets S^I, including dependence on dynamics, τ, and data distribution.
• Discrete-time and irregular sampling: Extend the theory to discrete-time observations and irregular grids, carefully linking continuous-time operator existence to discretization-consistent learning and error bounds.
• Cross-attention operator theory: Provide a formal universal approximation result for the cross-attention neural operator variant used for forecasting on different input/output grids (mentioned for some experiments but not theoretically detailed in the main text).
• Non-observable cases: Develop frameworks for systems that are not observable (e.g., z-only observation in Lorenz ‘63), such as set-valued or probabilistic operator learning that returns uncertainty sets/distributions rather than single-valued outputs.
• General observation functions: Generalize beyond observing the p component (projection) to non-linear and possibly non-invertible observation operators h(p,q), with corresponding observability conditions and operator-learning guarantees.
• Lower regularity and non-smooth dynamics: Relax the C^k assumptions on f,g and trajectories; extend to piecewise-smooth dynamics, systems with shocks, or nonsmooth observation operators, and characterize what continuity/approximation results still hold.
• Stochastic dynamics: Extend to SDEs and randomly forced PDEs, addressing whether pathwise smoothing/forecasting operators exist and can be learned, and how uncertainty propagates.
• Robustness and stability: Quantify Lipschitz moduli or stability margins of the true and learned operators with respect to perturbations in input trajectories, especially near points where the observability-rank degrades.
• Domain identification from data: Give algorithms to estimate the compact domain I (or its image under the flow) on which the operator exists and is continuous, and assess sensitivity to coverage gaps in the training set.
• Physical constraints and invariants: Incorporate and analyze methods to enforce hard physical constraints (e.g., conservation laws) in learned operators, and study their impact on approximation and generalization.
• Hybrid (model-informed) extensions: Investigate hybrid formulations that leverage partial model knowledge with data-driven operators, and analyze error decomposition between model and learned components.
• Scalability and complexity: Characterize computational and memory scaling of transformer neural operators for high-dimensional/long-horizon PDE settings (e.g., Lorenz ‘96, Kuramoto–Sivashinsky), and propose architectures or sparsity priors that remain efficient.
• Out-of-distribution (OOD) robustness: Study performance under domain shift (e.g., parameter changes, different forcings) and develop theories/algorithms for operator adaptation or transfer across related dynamical regimes.
• Uncertainty quantification: Move beyond point estimates to probabilistic operator learning for smoothing/forecasting, providing calibrated predictive distributions and theoretical guarantees (e.g., coverage, proper scoring).
• Online filtering: Extend the operator-learning framework to online (filtering) settings, where the operator must be evaluated sequentially as new data arrive, and derive stability/error accumulation bounds.
• Multi-sensor and partial-channel fusion: Generalize to heterogeneous, intermittent, and partial observations across multiple channels/sensors, with theory guaranteeing identifiability and stable operator learning.
• Implementation–theory gap: Bridge the gap between continuum-space theory and finite, discretized training (choice of grids, quadrature/feature maps), providing consistency results and practical guidance for discretization-invariant training.

Immediate Applications

The following applications can be deployed now by leveraging the paper’s operator-learning framework for smoothing and forecasting, together with existing neural-operator toolchains and domain data.

• Forecast acceleration for weather and oceanography
  • Sectors: energy, agriculture, insurance/finance, public safety
  • Tools/products/workflows:
  • Forecast-as-Operator API that maps recent observations to short-term forecasts (minutes–days) without full data assimilation cycles
  • Integration with existing NWP pipelines as a fast “nowcasting” module
  • Use transformer neural operators (e.g., NVIDIA Modulus/neuraloperator, DeepONet-style libraries) trained on reanalysis/observational archives
  • Assumptions/dependencies:
  • Sufficient coverage of training data near the operational regime; observability of chosen variables; handling of measurement noise via pre-filtering or robust loss; domain shift monitoring
• Data-driven smoother for hidden state reconstruction in industrial IoT and manufacturing
  • Sectors: manufacturing, process engineering, robotics (industrial arms, CNC, AM)
  • Tools/products/workflows:
  • Operator Smoother SDK that maps observed sensor streams to latent quality/process variables over a time window
  • Batch-quality estimation and fault forensics without an explicit physics model
  • Assumptions/dependencies:
  • Partial observability rank condition approximately holds; data sufficiently represent modes of operation; reliable synchronization and time-stamping; tolerable noise or denoising pipeline
• Real-time grid operations: fast load and renewable generation forecasting
  • Sectors: energy (utilities, ISOs, DER aggregators)
  • Tools/products/workflows:
  • Operator-based short-horizon renewable (wind/solar) forecasting from SCADA observations
  • Local forecasters at the edge for feeders/DERs; ensemble of operator forecasters for probabilistic dispatch heuristics
  • Assumptions/dependencies:
  • Stationarity over training/serving windows; retraining cadence for seasonality; data access agreements and latency guarantees
• Financial markets micro-forecasting and state smoothing
  • Sectors: finance (quant trading, risk)
  • Tools/products/workflows:
  • Operator forecaster mapping recent observed order-flow features to near-term mid-price/volatility projections
  • Operator smoother reconstructing latent liquidity or risk-factor proxies from observed streams for backtesting
  • Assumptions/dependencies:
  • Regime shifts and adversarial behavior require frequent revalidation; careful regularization; strict latency constraints
• Robotics and autonomous systems: state smoothing under partial sensing
  • Sectors: robotics, autonomous driving, drones
  • Tools/products/workflows:
  • Learned smoother to recover unobserved states (e.g., velocities, contact states) from onboard sensors for model-predictive control
  • Drop-in alternative to hand-tuned Kalman/extended Kalman smoothers in offline pipelines
  • Assumptions/dependencies:
  • Availability of representative simulated/real trajectories; observability of target states; safety validation before deployment in the loop
• Healthcare time-series: retrospective reconstruction (smoothing) for clinical decision support
  • Sectors: healthcare, digital health, wearables
  • Tools/products/workflows:
  • Operator smoother to reconstruct latent physiological trajectories (e.g., cardiac/respiratory drive) from observed vitals for chart reviews or post-op analysis
  • Wearable analytics that denoise and impute unobserved metrics using prior days’ data
  • Assumptions/dependencies:
  • Strong safeguards for noise, missingness, and shifts; bias audits; clinical validation; HIPAA/GDPR compliance
• Scientific computing emulators and initial-condition estimation
  • Sectors: R&D, academia, aerospace, climate labs
  • Tools/products/workflows:
  • Operator emulator to initialize PDE solvers by mapping observed fields to missing fields (smoothing) and to near-future states (forecasting), reducing spin-up and compute cost
  • Assumptions/dependencies:
  • Training data generated from trusted solvers or experiments; known operating envelope; local validity per observability neighborhood
• Observability diagnostics for sensor placement and experiment design
  • Sectors: aerospace, automotive, process engineering, environmental monitoring
  • Tools/products/workflows:
  • “Observability Checker” utility using symbolic/learned dynamics to approximate Lie-derivative rank tests and suggest sensor configurations that make target states observable
  • Assumptions/dependencies:
  • Access to approximate dynamics or data-driven surrogates; numeric stability of derivative estimation; discrete-time approximations of the continuous-time theory
• Education and workforce upskilling: operator learning for DA/forecasting
  • Sectors: education, professional training
  • Tools/products/workflows:
  • Course modules and labs replicating Lorenz ’63/’96 and Kuramoto–Sivashinsky experiments; assignments combining control observability with neural-operator training
  • Assumptions/dependencies:
  • Open datasets and reproducible code; modest GPU resources; curricular integration with control, ML, and PDE courses
• Enhanced mobile weather apps and consumer energy insights
  • Sectors: consumer software, smart home
  • Tools/products/workflows:
  • Lightweight operator forecasters for hyperlocal short-term precipitation/wind; HVAC optimization via short-horizon load forecasts
  • Assumptions/dependencies:
  • On-device or edge inference feasibility; reliable local sensor/third-party feeds; periodic retraining

Long-Term Applications

These applications require further research on noise, online filtering, uncertainty quantification, scaling to high-dimensional PDEs, and governance for safety-critical deployment.

• End-to-end data-driven numerical weather prediction with uncertainty quantification
  • Sectors: public safety, insurance/finance, agriculture, logistics
  • Tools/products/workflows:
  • Operator-ensemble forecasting replacing or tightly coupling with data assimilation cycles; probabilistic outputs (e.g., CRPS-calibrated)
  • Foundation models for dynamical systems trained on multi-sensor Earth data
  • Assumptions/dependencies:
  • Robust handling of noise and sparse observations; out-of-distribution detection; compute-efficient global training; regulatory acceptance
• Certified operator-based pipelines for safety-critical autonomy
  • Sectors: autonomous driving, aviation, medical devices
  • Tools/products/workflows:
  • Verified operator smoothers/forecasters with formal reachability/safety envelopes; hybrid controllers that fall back to certified filters
  • Assumptions/dependencies:
  • Formal methods for operator models; conservative uncertainty bounds; standards compliance (e.g., ISO 26262, DO-178C)
• Operator-driven digital twins at plant/asset scale
  • Sectors: energy (power plants, wind farms), manufacturing, smart cities
  • Tools/products/workflows:
  • Multi-physics digital twins where operator learners provide fast latent-state smoothing and rollouts within larger simulators for planning and anomaly response
  • Assumptions/dependencies:
  • Persistent data streams and MLOps; drift monitoring; co-simulation with physics engines; lifecycle governance
• Adaptive sensor placement and active learning guided by observability
  • Sectors: environmental monitoring, industrial inspection, precision agriculture
  • Tools/products/workflows:
  • Closed-loop policies that select next measurements to maximize observability and reduce uncertainty in latent states or forecasts
  • Assumptions/dependencies:
  • Efficient online estimators; exploration–exploitation balance; real-time constraints; multi-agent coordination
• Hybrid DA systems: operator smoothers inside ensemble Kalman or variational frameworks
  • Sectors: geosciences, oceanography, aerospace
  • Tools/products/workflows:
  • Use operator learners for model-error correction, state augmentation, or as fast surrogates for observation operators within EnKF/4D-Var
  • Assumptions/dependencies:
  • Stable coupling between learned operators and physics solvers; consistent adjoints; uncertainty propagation
• Operator learning for individualized medicine and closed-loop therapeutics
  • Sectors: healthcare, pharma
  • Tools/products/workflows:
  • Patient-specific state estimators (e.g., glucose–insulin dynamics) and short-horizon therapy controllers trained on longitudinal data; home digital twins
  • Assumptions/dependencies:
  • Strong clinical evidence; robust handling of noise/missingness; ethical oversight; regulatory approval
• Grid-wide probabilistic operations and market design enabled by operator forecasts
  • Sectors: energy markets, policy
  • Tools/products/workflows:
  • Markets that internalize operator-based uncertainty for reserve sizing, locational marginal pricing with forecast risk, and DER coordination
  • Assumptions/dependencies:
  • Transparent model governance; standardized benchmarks; stakeholder buy-in; cybersecurity
• Operator-assisted risk analytics in finance and climate stress testing
  • Sectors: finance, policy/regulation
  • Tools/products/workflows:
  • Scenario generators that map observables to latent risk states and future stress paths; integration with supervisory stress tests
  • Assumptions/dependencies:
  • Model risk management frameworks; explainability; robust backtesting across regimes
• Large-scale spatiotemporal PDE forecasting (e.g., air quality, flooding)
  • Sectors: environmental policy, disaster management
  • Tools/products/workflows:
  • Operator forecasters over high-resolution meshes; coupling with hydrological/dispersion models to deliver rapid, localized early warnings
  • Assumptions/dependencies:
  • High-dimensional scalability with memory-efficient operators; multi-modal data fusion; reliable ground-truth curation
• Standardization and auditing frameworks for operator-learning models
  • Sectors: policy, industry consortia, standards bodies
  • Tools/products/workflows:
  • Benchmarks, documentation standards, and audit trails for operator-based smoothing/forecasting; disclosure of observability assumptions and training domains
  • Assumptions/dependencies:
  • Cross-institutional collaboration; legal clarity on data use; reproducibility infrastructure

Cross-cutting dependencies and caveats

• The paper’s theory assumes noise-free, continuous-time trajectories, local observability, and compactness; engineering deployments must address measurement noise, discretization, and domain shift.
• Universal approximation guarantees hold on compact sets and do not ensure generalization outside the training support; active monitoring, retraining, and uncertainty estimates are essential.
• Observability-rank checks require either known dynamics or good surrogates; in black-box settings, empirical tests (e.g., performance vs. sensor subsets) may be used as proxies.
• Compute and data access are practical constraints; privacy and governance considerations apply in healthcare/finance/public-sector deployments.

• Analog forecasting: A purely data-driven forecasting method that predicts future states by finding and reusing past trajectories similar to the current one. Example: "and go by the name of analog forecasting;"
• Banach space: A complete normed vector space; here, the space of continuous functions with the supremum norm. Example: "the infinite dimensional Banach space of continuous functions mapping the set D to the d-dimensional vector space R^d"
• Bootstrap particle filter: A sequential Monte Carlo filtering method that propagates and resamples weighted particles to approximate posterior distributions. Example: "bootstrap particle filter weights collapse in high dimensions"
• Burn-in time: An initial period during which trajectories are evolved so that their distribution approaches a target (e.g., invariant) distribution before sampling. Example: "over some burn-in time"
• Control theory: The mathematical study of dynamical systems with inputs, focusing on observability, controllability, and feedback design. Example: "a foundational work in control theory"
• Cross-attention: An attention mechanism where queries attend to keys/values from a different sequence, used here to map inputs on one grid to outputs on another. Example: "we use a variant of this architecture which uses cross-attention to approximate"
• Data assimilation (DA): The combination of observational data with dynamical models (or data-driven surrogates) to estimate states and parameters of a system. Example: "The field of data assimilation (DA) is concerned with the use of observational data to perform state estimation in dynamical systems"
• Dynamic mode decomposition: A data-driven method that approximates linear dynamics underlying complex systems, often linked to the Koopman operator framework. Example: "dynamic mode decomposition"
• Ensemble Kalman filter (EnKF): A Monte Carlo variant of the Kalman filter that uses an ensemble of states to approximate means and covariances for high-dimensional systems. Example: "ensemble Kalman filters (EnKF)"
• Evaluation functional: A linear functional that evaluates a function at a specific point in its domain. Example: "the evaluation functional at time $t=0$, namely $$\delta_0 \colon \bigotimes_{j=0}^n C^{k-j}([0,T];R^{d_p}) \to R^{(n+1)d_p}$$, is linear and bounded"
• ExKF (extended Kalman filter): A nonlinear extension of the Kalman filter that linearizes dynamics around the current estimate. Example: "extended (ExKF) and ensemble Kalman filters (EnKF)"
• Filtering: Online, sequential state estimation as observations arrive, typically in real time. Example: "filtering, which concerns the online estimation of an underlying state, sequentially as observations are received"
• Homeomorphism: A continuous bijection with a continuous inverse; here used for time rescaling mappings between function spaces. Example: "rescaling operators are linear homeomorphisms"
• Inverse function theorem: A result guaranteeing local invertibility of a differentiable map when its Jacobian is nonsingular at a point. Example: "the inverse function theorem for Euclidean spaces"
• Kernel analog forecasting: A smoothed, kernel-based version of analog forecasting that provides continuity and theoretical guarantees. Example: "In the last decade kernel analog forecasting has been developed"
• Kuramoto-Sivashinsky equation: A nonlinear PDE exhibiting spatiotemporal chaos, used as a benchmark dynamical system. Example: "Kuramoto-Sivashinsky dynamical systems."
• Lie derivative: The derivative of a function along a vector field, capturing how the function changes following system dynamics. Example: "we also define $\mathcal{L}_w\,f(v)$, the Lie derivative of $f$ along $w$, as"
• Lipschitz boundary: A regularity condition on domain boundaries ensuring well-posedness of PDE/functional-analytic results. Example: "a bounded domain with Lipschitz boundary"
• Lorenz `63: A three-dimensional chaotic ODE model used as a canonical test of nonlinear dynamics and chaos. Example: "Consider the Lorenz `63 dynamical system"
• Lorenz `96: A high-dimensional chaotic ODE model often used to study predictability and DA algorithms. Example: "Lorenz 63, Lorenz96 and Kuramoto-Sivashinsky dynamical systems."
• Neural operator: A neural network architecture that learns mappings between function spaces (operators) rather than between finite-dimensional vectors. Example: "neural operator architectures"
• Non-autonomous equation: A dynamical system whose evolution depends explicitly on time or on an external time-dependent input. Example: "can be integrated as a non-autonomous equation for $q(\cdot)$, driven by the observed $p(\cdot).$"
• Observability-rank condition: A rank condition ensuring local invertibility of the mapping from states to observations (and their derivatives), enabling state recovery. Example: "We introduce an observability condition on the dynamics, as appearing in control theoretic literature"
• Observation function: A function mapping the full system state to the observed components. Example: "the observation function $h(\mathfrak{p},\mathfrak{q}) = \mathfrak{p}$"
• Pushforward measure: The distribution obtained by mapping a random variable through a measurable function (e.g., time evolution). Example: "Measure $\nu$ is computed as the pushforward, over some burn-in time, of the simpler measure $\nu_0.$"
• Self-attention: An attention mechanism where queries, keys, and values come from the same sequence; used here in transformer neural operators. Example: "the transformer neural operator based on self-attention"
• Semigroup of operators: A family of operators parameterized by time that map initial conditions to evolved states and satisfy semigroup properties. Example: "we define by $\Phi:R^+ \timesR^{d_p+d_q}\toR^{d_p+d_q}$ the semigroup of operators associated to the dynamical system"
• Takens' delay embedding theorem: A theorem showing that generic observations of a dynamical system can reconstruct its state via delay coordinates. Example: "Takens' delay embedding theorem"
• Transformer neural operator: A neural operator instantiation using transformer-style attention mechanisms to learn operator mappings. Example: "the transformer neural operator"
• Universal approximation theorem: A result asserting that a class of neural operators can approximate any continuous operator on a compact set to arbitrary accuracy. Example: "universal approximation theorem for purely data-driven algorithms"
• Weight collapse: Degeneracy in particle filters where a few particles dominate the weight distribution, leading to poor approximations. Example: "bootstrap particle filter weights collapse in high dimensions"

Sign up for free to add this paper to one or more collections.