AI Mastery

PREVIOUS PAGE INDEX PAGE NEXT PAGE

Differential Equations in AI: Dynamic Systems & Time-Series Prediction

N.B.- All my books are exclusively available on Amazon. The free notes/materials on globalcodemaster.com do NOT match even 1% with any of my PUBLISHED BOoks. Similar topics ≠ same content. Books have full details, exercises, chapters & structure — website notes do not.No book content is shared here. We fully comply with Amazon policies.

TABLE OF CONTENT

1. Foundations of Differential Equations for AI Practitioners 1.1 Ordinary Differential Equations (ODEs): first-order, higher-order, linear vs nonlinear 1.2 Systems of ODEs and vector fields: existence, uniqueness, and Picard–Lindelöf theorem 1.3 Phase space, equilibrium points, stability, and phase portraits 1.4 Autonomous vs non-autonomous systems 1.5 Partial Differential Equations (PDEs): classification (parabolic, hyperbolic, elliptic) 1.6 From discrete sequences to continuous limits: why differential equations matter in AI 1.7 Key differences between classical numerical solvers and neural approaches

2. Neural Ordinary Differential Equations (Neural ODEs) 2.1 Continuous-depth models: replacing discrete layers with ODE solvers 2.2 The adjoint method: memory-efficient backpropagation through ODE solvers 2.3 ODE solvers in practice: Euler, RK4, Dopri5, adaptive step-size (dopri5, tsit5) 2.4 Neural ODE variants: ODE-RNN, Continuous-time RNN, Augmented Neural ODEs 2.5 Training stability: stiffness, gradient explosion, and regularization tricks 2.6 Empirical advantages: constant memory cost, arbitrary depth, smooth trajectories 2.7 Limitations and modern mitigations (2025–2026): stiffness-aware solvers, symplectic integrators

3. Neural Controlled Differential Equations (Neural CDEs) 3.1 Path-dependent dynamics: from discrete time-series to continuous paths 3.2 Controlled differential equations and rough path theory 3.3 Neural CDE architecture: controlled path + neural vector field 3.4 Log-signatures vs. discrete interpolation: when to use each 3.5 State-of-the-art performance on irregular time-series (MIMIC, PhysioNet, ETTh) 3.6 Extensions: Double Controlled CDEs, Conditional Neural CDEs

4. State-Space Models and Continuous-time Architectures 4.1 Classical linear state-space models (Kalman filter connection) 4.2 Structured State Space sequence models (S4, S5, Mamba family) 4.3 HiPPO framework: high-order polynomial projection operators 4.4 Discretization strategies: bilinear, zero-order hold, exact exponential 4.5 Mamba-2 and structured variants: diagonal + low-rank, state expansion 4.6 Long-range dependency capture without quadratic attention cost 4.7 Hybrid SSM + Transformer architectures (2025–2026 frontier)

5. Physics-Informed Neural Networks (PINNs) and Operator Learning 5.1 Embedding differential equations into the loss function 5.2 Soft vs hard constraints: collocation points, boundary/initial conditions 5.3 Fourier Neural Operator (FNO): global spectral convolution in frequency domain 5.4 DeepONet and variants: operator regression for PDE families 5.5 PI-DeepONet, MIONet, and multi-fidelity operator learning 5.6 Spectral methods in operator learning: Wavelet Neural Operators, Geo-FNO 5.7 Applications: fluid dynamics, climate modeling, molecular dynamics

6. Diffusion Models and Score-Based Generative Modeling 6.1 Forward and reverse SDEs: Ornstein–Uhlenbeck, variance-preserving/variance-exploding 6.2 Score function estimation: denoising score matching objective 6.3 Continuous-time perspective: probability flow ODE vs. reverse-time SDE 6.4 Neural SDE solvers in diffusion: VP-SDE, VE-SDE, sub-VP 6.5 Flow-matching and rectified flow: simulation-free training of ODE-based generative models 6.6 Diffusion bridges and stochastic interpolants 6.7 2025–2026 frontiers: diffusion on manifolds, Lie-group diffusion, diffusion for time-series

7. Time-Series Forecasting with Differential Equations 7.1 Classical ODE-based forecasting vs. deep learning hybrids 7.2 Neural ODEs for multivariate time-series (Time-series ODE-RNN) 7.3 Temporal Point Processes via Neural Hawkes Processes (intensity ODEs) 7.4 Latent ODEs for irregular time-series (Latent-ODE, GRU-ODE-Bayes) 7.5 Continuous-time transformers and ODE-augmented attention 7.6 State-space models for probabilistic forecasting (DeepState, Chronos-SSM hybrids) 7.7 Benchmark performance: ETTh, Electricity, Weather, Traffic, MIMIC-IV

8. Stability, Numerical Challenges, and Theoretical Insights 8.1 Stiffness in neural differential equations: detection and adaptive solvers 8.2 Symplectic and structure-preserving integrators for Hamiltonian systems 8.3 Spectral stability of discretization schemes 8.4 Lipschitz constants, gradient norms, and explosion prevention 8.5 Infinite-width limits: Neural ODEs → kernel regression with path signatures 8.6 Frequency-domain analysis: Fourier perspective on ODE learning dynamics 8.7 Generalization bounds for continuous-depth models

9. Advanced Applications and 2025–2026 Frontiers 9.1 Neural SDEs for uncertainty quantification and stochastic optimal control 9.2 Diffusion-based molecular dynamics and protein structure generation 9.3 Operator learning for climate simulation and weather forecasting 9.4 Continuous normalizing flows (CNFs) and rectifying flows 9.5 Lie-group ODEs and equivariant continuous models 9.6 Hybrid discrete-continuous architectures (Mamba + ODE layers) 9.7 Open challenges: scaling to trillion-parameter continuous models, stiffness at extreme scales, theoretical convergence rates

1. Foundations of Differential Equations for AI Practitioners

This opening chapter provides the essential mathematical groundwork in differential equations that every AI practitioner needs to understand modern continuous-time models (Neural ODEs, Neural CDEs, SSMs/Mamba, diffusion models, PINNs, operator learning, etc.). The goal is to build intuition for dynamics, stability, and continuous limits — without requiring a full mathematics degree — while highlighting exactly why these concepts are crucial in 2025–2026 deep learning.

1.1 Ordinary Differential Equations (ODEs): first-order, higher-order, linear vs nonlinear

Ordinary Differential Equation (ODE) An ODE relates a function y(t) (or vector y(t)) to its derivatives with respect to an independent variable t (usually time):

First-order (scalar): dy/dt = f(t, y) or y' = f(t, y)

First-order system (vector form, most relevant in AI): dy/dt = f(t, y) where y ∈ ℝᵈ, f: ℝ × ℝᵈ → ℝᵈ

Higher-order example (second-order): d²y/dt² + p(t) dy/dt + q(t) y = g(t)

Linear vs Nonlinear

• Linear ODE (very important in classical control, Kalman filters, linear SSMs): y' = A(t) y + b(t) Superposition holds: solutions can be added, scaled.

• Nonlinear ODE (dominant in neural differential equations): y' = f(t, y) where f is nonlinear in y (e.g., neural network with tanh, ReLU, GELU, etc.)

Why this distinction matters in AI

• Linear dynamics → closed-form solutions possible (matrix exponential) → exact discretization in S4/Mamba

• Nonlinear dynamics → universal approximation power → Neural ODEs / diffusion models can model arbitrary continuous transformations

Examples in AI

• First-order: Neural ODE: dh/dt = NeuralNet(h(t), t)

• Second-order: Hamiltonian Neural Networks, position-velocity systems in physics simulation

1.2 Systems of ODEs and vector fields: existence, uniqueness, and Picard–Lindelöf theorem

Most modern continuous models are autonomous first-order vector fields:

dy/dt = f(y) (time-invariant) or dy/dt = f(t, y) (explicit time dependence)

f(y) is called the vector field — it assigns a velocity vector to every point in phase space ℝᵈ.

Existence and uniqueness (core theoretical foundation):

Picard–Lindelöf theorem (local Lipschitz version): If f(t, y) is continuous in t and locally Lipschitz continuous in y (i.e., |f(t,y₁) − f(t,y₂)| ≤ L |y₁ − y₂| for y in compact set), then for any initial condition y(t₀) = y₀ there exists a unique solution on some interval [t₀ − δ, t₀ + δ].

Why this matters in deep learning

• Neural networks with Lipschitz activations (tanh, sigmoid) or gradient clipping → locally Lipschitz → guarantees unique solution trajectory

• ReLU networks are piecewise linear → not globally Lipschitz → multiple solutions possible in theory (rare in practice due to regularization)

• Exploding gradients → violation of local Lipschitz → numerical solver failure

1.3 Phase space, equilibrium points, stability, and phase portraits

Phase space = ℝᵈ where each point is a possible state y(t). A solution y(t) is a trajectory (curve) in phase space.

Equilibrium point (fixed point): y* such that f(y*) = 0 → dy/dt = 0 → system stays at y* forever if started there.

Stability:

• Asymptotically stable: nearby trajectories converge to y* as t → ∞

• Unstable: nearby trajectories move away

• Marginally stable / neutrally stable: stay nearby but do not converge

Linearization around equilibrium (most important analysis tool): Let y(t) = y* + δ(t), then dδ/dt ≈ Df(y*) δ (Jacobian Df at y*)

Eigenvalues of Jacobian Df(y*) determine local stability:

• All Re(λ) < 0 → asymptotically stable

• Any Re(λ) > 0 → unstable

• Re(λ) = 0 with multiplicity → need higher-order terms

Phase portrait = sketch of representative trajectories in phase space → In 2D: easy to draw (sinks, sources, saddles, centers, limit cycles) → In high-D (AI): visualize via PCA projection of trajectories

AI relevance

• Stable fixed points in latent space of continuous VAEs / normalizing flows

• Limit cycles in oscillatory time-series modeling

• Unstable equilibria explain mode collapse in GANs / diffusion reverse processes

1.4 Autonomous vs non-autonomous systems

Autonomous dy/dt = f(y) (right-hand side does not depend explicitly on t)

→ Time-invariant dynamics → phase portrait is fixed → Equilibrium points are constant → Most Neural ODEs, SSMs (S4/Mamba), diffusion SDEs are autonomous

Non-autonomous dy/dt = f(t, y) (explicit time dependence)

→ Time-varying vector field → phase portrait changes with t → Equilibrium points can move (non-constant attractors) → Examples: time-dependent forcing in physics-informed models, controlled systems, seasonal time-series

Hybrid cases in AI

• Neural CDEs: path-dependent (non-autonomous in classical sense, but driven by input path)

• Time-conditioned diffusion: reverse process has explicit t dependence

1.5 Partial Differential Equations (PDEs): classification (parabolic, hyperbolic, elliptic)

PDE = equation involving partial derivatives of multivariable function u(t, x₁, …, x_d)

Classification (second-order linear PDEs):

a u_xx + 2b u_xy + c u_yy + … = 0

Discriminant Δ = b² − a c

TypeDiscriminant ΔExample EquationBehavior / AI RelevanceEllipticΔ < 0Laplace: ∇²u = 0Steady-state, boundary-value problems; PINNs for Poisson eq.ParabolicΔ = 0Heat/diffusion: u_t = ∇²uDiffusion models, smoothing, score-based generative modelingHyperbolicΔ > 0Wave: u_tt = c² ∇²uWave propagation; some physics simulators, acoustic modeling

AI connection

• Diffusion models = reverse parabolic PDE (denoising score matching)

• Fourier Neural Operators / DeepONet learn solution operators for all three classes

• Elliptic PINNs for equilibrium problems (materials, electrostatics)

• Hyperbolic solvers for transport-dominated phenomena (fluids, traffic)

1.6 From discrete sequences to continuous limits: why differential equations matter in AI

Discrete → Continuous motivation:

• RNNs / Transformers = discrete dynamical systems (recurrent updates)

• As number of layers → ∞ or time-step → 0 → discrete dynamics converge to continuous ODE

• Continuous limit → constant memory (no need to store all hidden states)

• Arbitrary “depth” via adaptive solvers

• Natural handling of irregular time-series (no fixed Δt)

Core advantages in modern AI:

• Memory efficiency: adjoint method → O(1) memory for any depth

• Resolution invariance: model trained at one time-grid works at finer/coarser grids

• Theoretical elegance: Neural ODE = residual network at infinitesimal step size

• Physics alignment: direct incorporation of known laws (PINNs, operator learning)

• Expressive power: universal approximation for continuous operators

Historical progression (key papers): 2018: Neural ODE → continuous-depth revolution 2020: Neural CDE → irregular time-series 2020–2022: S4 / Mamba → efficient long-range continuous-time modeling 2023–2026: Flow-matching, rectified flow, diffusion bridges → ODE-centric generative modeling

1.7 Key differences between classical numerical solvers and neural approaches

AspectClassical Numerical SolversNeural Differential EquationsPractical Implication in AIPurposeSolve given ODE accuratelyLearn unknown ODE from dataNeural = model discovery, classical = evaluationParametersFixed (method parameters: step size, order)Neural network weights (millions–billions)End-to-end differentiable learningBackpropagationAdjoint not needed (no training)Adjoint method (continuous backprop)Constant memory regardless of “depth”Time-steppingFixed or adaptive (user controls error)Solver adaptive, but neural field learnedNeural can learn stiff / multi-scale dynamicsRegularizationNumerical stability constraintsWeight decay, Lipschitz penalties, etc.Neural solvers regularized implicitly via lossScalabilityLimited by dimension & stiffnessScales to high-D latent spacesNeural ODEs used in 1000+ dimensional latent spacesInterpretabilityTransparent (known method error bounds)Black-box vector field (but trajectories smooth)Neural trajectories often more interpretable visually

2025–2026 consensus

• Use classical solvers (diffrax, torchdiffeq, scipy.integrate) as the numerical backbone

• Neural approaches excel when the dynamics are unknown and must be learned from data

• Hybrid: learned vector field + structure-preserving classical integrator (e.g., symplectic for Hamiltonian systems)

This foundational chapter equips you with the language and intuition needed to understand why continuous-time models are revolutionizing sequence modeling, generative AI, scientific computing, and time-series analysis in deep learning.

2. Neural Ordinary Differential Equations (Neural ODEs)

Neural Ordinary Differential Equations (Neural ODEs), introduced by Chen et al. in 2018, represent one of the most influential paradigm shifts in deep learning: replacing discrete stacked layers with a continuous-depth model defined by an ordinary differential equation. This chapter covers the core architecture, the revolutionary adjoint method for backpropagation, practical solvers, important variants, training challenges, empirical strengths, and the state-of-the-art mitigations as of 2025–2026.

2.1 Continuous-depth models: replacing discrete layers with ODE solvers

Classical residual network (ResNet) A ResNet block is:

h_{t+1} = h_t + f(h_t, θ_t) ⋅ Δt (with Δt = 1 implicitly)

As the number of layers → ∞ and step size Δt → 0, this Euler discretization converges to the continuous limit:

dh/dt = f(h(t), θ(t), t) h(0) = x

Neural ODE definition The output h(T) is obtained by solving the above initial-value problem from t=0 to some final time T (which can be fixed or learned):

h(T) = h(0) + ∫₀^T f(h(t), θ(t), t) dt

f(·) is a neural network (with parameters θ) that defines the vector field.

Key conceptual shift

• Depth is now continuous (not discrete integer)

• Number of function evaluations controlled by ODE solver (not by manual layer count)

• Model can learn arbitrary continuous transformations rather than discrete steps

Forward pass in practice Use any black-box ODE solver (Euler, RK4, adaptive Dopri5, etc.) to integrate from t=0 to t=T.

2.2 The adjoint method: memory-efficient backpropagation through ODE solvers

The breakthrough that made Neural ODEs scalable: adjoint sensitivity method (continuous backpropagation).

Standard backprop through solver would require storing every intermediate state → O(number of steps) memory → impossible for fine time grids.

Adjoint method: Define the adjoint state a(t) = ∂L/∂h(t) (gradient of loss L w.r.t. hidden state at time t)

The adjoint evolves backward in time according to another ODE:

da/dt = − a(t)^T ⋅ (∂f/∂h)(h(t), θ(t), t)

With terminal condition a(T) = ∂L/∂h(T)

Parameter gradients: dL/dθ = ∫₀^T a(t)^T ⋅ (∂f/∂θ)(h(t), θ(t), t) dt

Memory cost

• Only need to store initial state h(0) and final state h(T)

• During backward pass: re-solve forward ODE to get h(t) on-the-fly while integrating adjoint backward → Total memory = O(1) w.r.t. number of time-steps (constant memory)

2025–2026 status Adjoint method remains the gold standard; implemented in torchdiffeq, diffrax (JAX), torchsde. Variants include checkpointing + adjoint for even lower memory in very deep models.

2.3 ODE solvers in practice: Euler, RK4, Dopri5, adaptive step-size (dopri5, tsit5)

Fixed-step solvers (simple but limited):

• Euler: h_{t+Δt} = h_t + f(h_t) Δt → First-order, cheap, but inaccurate and unstable for stiff problems

• RK4 (Runge–Kutta 4th order): classical 4-stage method → Good balance of accuracy and cost for non-stiff problems

Adaptive-step solvers (dominant in Neural ODEs):

• Dopri5 (Dormand–Prince 5(4)): embedded Runge–Kutta pair (5th order solution + 4th order error estimate) → Adaptive step-size control based on local error tolerance → Most popular default in torchdiffeq and diffrax

• Tsit5 (Tsitouras 5(4)): improved embedded RK pair → Often faster and more stable than Dopri5 on many Neural ODE tasks

• Other strong options (2025–2026): KenCarp4, Rodas5P (stiff-aware), Heun’s method with projection

Choosing a solver in practice

• Start with Dopri5 or Tsit5 (adaptive, robust)

• Use atol=rtol=1e-4 to 1e-7 depending on task precision

• For stiff dynamics: switch to implicit/exponential solvers (LSODA, KenCarp)

• For very long trajectories: Dopri853 (higher order, fewer steps)

2.4 Neural ODE variants: ODE-RNN, Continuous-time RNN, Augmented Neural ODEs

ODE-RNN Combines RNN-style discrete updates with continuous evolution between observations:

Between times t_i and t_{i+1}: h(t) = ODE-RNN(h(t_i), t; θ) At observation: h_{i+1} ← GRU/MLP(h(t_{i+1}^-), x_{i+1})

Continuous-time RNN Fully continuous: input is a continuous path x(t) → dh/dt = f(h(t), x(t))

Augmented Neural ODE (ANODE, 2019) To avoid numerical instability and information loss in long trajectories:

Augment state: [h(t); z(t)] where z(t) is auxiliary variable dh/dt = f(h, z), dz/dt = g(h, z)

→ Increases expressivity and numerical robustness

Later variants (2023–2026)

• SONODE / Heavy-ball Neural ODE: second-order ODEs (inertia/momentum)

• CondNeural ODE: time-dependent conditioning

• Augmented + symplectic hybrids for physics tasks

2.5 Training stability: stiffness, gradient explosion, and regularization tricks

Stiffness Problem: some directions evolve very fast (large eigenvalues), others very slowly → solver takes tiny steps → slow training

Detection:

• Solver statistics: very small step sizes, many rejected steps

• Trajectory inspection: sudden jumps or oscillations

Mitigations (2025–2026 best practices):

• Use stiff-aware solvers (LSODA, Rosenbrock, BDF)

• Spectral normalization / Lipschitz regularization on f(·)

• Augmented state (ANODE-style) → spreads eigenvalues

• Gradient clipping + weight decay

• Curriculum: start with short time horizons, gradually increase

• Symplectic integrators for conservative systems (energy preservation)

Gradient explosion Common in long trajectories → mitigated by adjoint method + clipping + careful initialization (orthogonal/spectral)

2.6 Empirical advantages: constant memory cost, arbitrary depth, smooth trajectories

Constant memory Adjoint method → memory independent of number of solver steps → enables “infinite” depth in practice

Arbitrary depth Time horizon T can be treated as hyperparameter or learned → model automatically finds optimal “depth”

Smooth trajectories Continuous dynamics → hidden states evolve smoothly → better interpolation, extrapolation, uncertainty estimation

Empirical wins (repeatedly confirmed 2018–2026):

• Superior on irregular time-series (PhysioNet, activity recognition)

• Competitive or better on long-sequence modeling when combined with SSMs

• Natural handling of continuous labels / physics constraints

2.7 Limitations and modern mitigations (2025–2026): stiffness-aware solvers, symplectic integrators

Main limitations:

• Stiffness → very slow training on stiff problems (chemical kinetics, high-frequency oscillators)

• Expressive power lower than discrete Transformers on some sequence tasks

• Solver overhead → slower per epoch than fixed-layer models

• Numerical error accumulation in very long trajectories

Modern mitigations (2025–2026 frontier):

• Stiffness-aware solvers: KenCarp4, Rodas5P, ESIRK, DIRK → implicit/exponential Rosenbrock methods

• Symplectic & structure-preserving integrators: for Hamiltonian, reversible, or conservative systems (HNN, SRNN, Symplectic Neural ODEs)

• Hybrid discrete-continuous: Mamba + Neural ODE blocks, Transformer + ODE layers

• Learned solvers: meta-learned adaptive step-size or vector-field preconditioning

• Flow-matching & rectified flow: bypass traditional ODE solvers entirely → direct path straightening

• Parallel-in-time training techniques → reduce sequential solver bottleneck

Neural ODEs remain foundational: they inspired Neural CDEs, diffusion reverse processes, continuous normalizing flows, and large parts of the state-space revolution (S4 → Mamba).

3. Neural Controlled Differential Equations (Neural CDEs)

Neural Controlled Differential Equations (Neural CDEs), introduced by Kidger et al. in 2020, represent a major advancement over Neural ODEs for modeling irregularly sampled, continuous-time time-series data. While Neural ODEs assume a fixed time grid or smooth evolution driven by a time-dependent vector field, Neural CDEs treat the input itself as a continuous path that drives (controls) the hidden state dynamics. This makes them particularly powerful for real-world sequential data with missing values, asynchronous sampling, or continuous observations — common in healthcare, finance, climate, and sensor networks.

3.1 Path-dependent dynamics: from discrete time-series to continuous paths

Classical discrete models (RNNs, Transformers, LSTMs) operate on fixed time-steps:

h_{t+1} = f(h_t, x_{t+1})

→ Require regular sampling or imputation → lose information when data is naturally irregular.

Continuous path perspective Real-world time-series are better viewed as continuous paths X(t): [0,T] → ℝ^{d_in}, where t is continuous time and X(t) is defined even between observations (via interpolation).

The hidden state h(t) evolves continuously, driven by the entire input path X:

dh(t)/dt = f(h(t), dX(t)/dt) or more precisely in differential form:

dh(t) = f(h(t)) dX(t)

→ The evolution depends on the increments dX(t), not just point values x_t.

Key advantage

• Naturally handles irregular sampling, missing data, asynchronous multi-variate series without imputation

• Captures accumulation of information over continuous time intervals

• Generalizes Neural ODEs: when X(t) = t (scalar time), Neural CDE reduces to Neural ODE

Real-world examples

• MIMIC-III/IV ICU data: vital signs recorded at irregular intervals

• PhysioNet Challenge datasets: ECG, EEG with variable sampling rates

• Financial tick data: trades occur at unpredictable times

3.2 Controlled differential equations and rough path theory

Controlled differential equation (Lyons 1998, rough path theory):

dh(t) = f(h(t)) dX(t)

where X(t) is the driving path (input), and f(·) is the vector field (neural network).

Rough path theory provides the mathematical foundation for making sense of this integral when X(t) is very irregular (e.g., Brownian motion, highly oscillatory, or non-differentiable paths).

Key concepts:

• For smooth X(t), ordinary Riemann–Stieltjes integral suffices

• For rougher X(t) (Hölder continuous with exponent <1), need lifted path (iterated integrals / log-signatures) to define the integral unambiguously

• Neural CDE uses log-signature or discrete interpolation to lift discrete observations into a continuous path with sufficient regularity

Why rough paths matter in AI

• Guarantee unique solution even for non-smooth inputs

• Provide theoretical stability and generalization bounds

• Enable principled handling of discrete observations as limits of continuous paths

3.3 Neural CDE architecture: controlled path + neural vector field

Core components:

1. Input path X(t): discrete observations {(t_i, x_i)} → lifted to continuous path via interpolation or log-signature

2. Neural vector field f_θ(h) = NeuralNet(h) ∈ ℝ^{d_hidden × d_path} → Maps current hidden state h(t) to a linear map that acts on dX(t)

3. Controlled differential equation:

dh(t) = f_θ(h(t)) dX(t) h(0) = h_0 (usually MLP(initial observation))

4. Readout (optional): y(t) = g_θ(h(t)) or final h(T)

Forward pass:

• Lift discrete data to continuous path (via cubic spline, linear interpolation, or log-signature)

• Solve the CDE using an ODE solver (same as Neural ODE: Dopri5, Tsit5, etc.)

• Output predictions at desired times

Adjoint method (same as Neural ODE) enables memory-efficient backpropagation.

3.4 Log-signatures vs. discrete interpolation: when to use each

Two main path-lifting strategies:

MethodDescriptionProsConsBest Use CasesDiscrete interpolation (cubic spline, linear, natural cubic)Interpolate between observation points to create smooth path X(t)Simple, fast, preserves local structureCan introduce artificial smoothness; sensitive to noise/outliersRegularly sampled or mildly irregular dataLog-signatureCompute iterated integrals (signature) up to depth k, then reconstruct pathTheoretically sound for rough paths; compact representation; robust to irregularityComputationally heavier (O(n k²)); requires choice of depth kHighly irregular, asynchronous, or high-frequency data

Practical guidelines (2025–2026):

• Start with cubic spline interpolation (fast, good baseline)

• Switch to log-signature (depth 2–4) when data is very irregular or performance plateaus

• Hybrid: use interpolation for short gaps, log-signature for long irregular segments

• Libraries: signatory (Python), iisignature, diffrax (built-in log-ODE solver)

3.5 State-of-the-art performance on irregular time-series (MIMIC, PhysioNet, ETTh)

Neural CDEs have consistently set or approached state-of-the-art results on irregular and continuous-time benchmarks:

Key datasets & results (2020–2026 literature):

• MIMIC-III / MIMIC-IV (ICU vital signs, mortality / length-of-stay prediction): Neural CDE variants (NCDE, Double CDE) outperform GRU-D, ODE-RNN, and Transformers on AUROC and AUPRC

• PhysioNet Challenge 2012/2019 (sepsis, mortality): top entries frequently use Neural CDE or NCDE hybrids

• ETTh1/ETTm1/ETTm2 (electricity transformer temperature, long-horizon forecasting): Neural CDE + attention hybrids competitive with PatchTST, iTransformer, and Mamba-based models

• Activity recognition (HAR, UCI dataset with missing data): Neural CDE robust to dropped samples

• Climate / weather (ERA5 subsets): continuous-time models excel on irregularly sampled reanalysis data

2025–2026 trend: Neural CDE + Mamba-style state expansion + flow-matching training → pushing SOTA on long-horizon irregular forecasting.

3.6 Extensions: Double Controlled CDEs, Conditional Neural CDEs

Double Controlled CDEs (Kidger et al. extensions, 2021–2023):

• Two driving paths: one for input observations, one for auxiliary covariates or time

• dh(t) = f(h(t)) dX(t) + g(h(t)) dZ(t) → Better modeling of exogenous variables (e.g., treatment in medical data)

Conditional Neural CDEs (Kidger & Lyons, 2021+):

• Condition the vector field on global context c (patient ID, static covariates): f(h(t), c)

• Or condition on latent global parameters learned end-to-end

Other notable extensions:

• NCDE + Transformer hybrids → combine global attention with local continuous dynamics

• Variational Neural CDE → add stochasticity for uncertainty quantification

• Controlled diffusion models → extend CDE framework to SDEs for generative modeling of irregular sequences

Neural CDEs remain the gold standard for truly irregular, continuous-time sequential modeling — bridging classical control theory, rough paths, and deep learning in a principled way.

4. State-Space Models and Continuous-time Architectures

State-Space Models (SSMs) have emerged as one of the most powerful alternatives to Transformers for long-sequence modeling, offering linear scaling with sequence length while capturing long-range dependencies effectively. This chapter traces the evolution from classical linear control theory to the modern structured, continuous-time SSMs (S4 → S5 → Mamba family) that dominate efficient sequence modeling in 2025–2026, especially for time-series, audio, genomics, language, and scientific data.

4.1 Classical linear state-space models (Kalman filter connection)

Classical continuous-time linear state-space model:

dx/dt = A x + B u(t) y(t) = C x(t) + D u(t)

• x(t) ∈ ℝ^N : latent (hidden) state

• u(t) ∈ ℝ^M : input/control

• y(t) ∈ ℝ^P : output/observation

• A, B, C, D : system matrices (learnable in neural SSMs)

Discrete-time version (used in digital signal processing, RNNs):

x_{t+1} = A_d x_t + B_d u_t y_t = C_d x_t + D_d u_t

Kalman filter connection The Kalman filter is the optimal estimator for linear Gaussian state-space models:

• Predict step: propagate state mean and covariance

• Update step: incorporate new observation → correct estimate

Relevance to deep learning:

• Early neural SSMs (e.g., Deep State Space Models) were inspired by Kalman filtering for probabilistic forecasting

• Modern SSMs (Mamba, S4) retain the linear state transition structure but replace fixed A,B,C,D with structured, learnable parameterizations

4.2 Structured State Space sequence models (S4, S5, Mamba family)

S4 (Structured State Space Sequence model, Gu et al. 2021–2022) Introduced the key insight: parameterize A as a low-displacement-rank (HiPPO) matrix → enables efficient long-range modeling.

Core S4 recurrence (discretized):

x_{t+1} = A_d x_t + B_d u_t y_t = C_d x_t

But A_d is structured (diagonal + low-rank or companion form) → allows O(N) per step instead of O(N²) matrix multiplication (N = state dimension).

S5 (2022–2023) Improved S4 with better discretization and parallelizable scan → state expansion to 1M+ dimensions possible.

Mamba family (2023–2025):

• Mamba-1 (Gu & Dao 2023): selective SSM — input-dependent B and C matrices → context-aware dynamics

• Mamba-2 (2024): reformulates as structured linear recurrence with diagonal + low-rank structure → 2–8× faster inference/training

• Mamba-2 variants (Jamba, MambaByte, Vision Mamba): byte-level, multimodal, vision backbones

• Mamba-3 / MambaOut (2025–2026 frontier): deeper stacking, hybrid attention-SSM blocks, state expansion to 16M+

Why SSMs win over Transformers on long sequences:

• Linear time & memory complexity O(L N) vs O(L²)

• Constant state size → fixed memory regardless of sequence length

• Strong inductive bias for continuous dynamics

4.3 HiPPO framework: high-order polynomial projection operators

HiPPO (High-order Polynomial Projection Operators, Gu et al. 2020) The key theoretical innovation behind S4/Mamba: design A matrix so that the state x(t) remembers high-order polynomial moments of the input history.

Intuition:

• To capture long dependencies, the hidden state should store coefficients of a high-degree polynomial approximation of the input u(τ) for τ ≤ t

• HiPPO derives the optimal A matrix that minimizes reconstruction error for polynomial inputs

Mathematical core: A is chosen as the companion matrix of a scaled Laguerre or Legendre polynomial basis (orthogonal polynomials on [0,∞) or [-1,1]).

Result:

• State transition matrix A has eigenvalues on the negative real axis → stable

• Memory of past inputs decays polynomially (not exponentially) → theoretically ideal for long-range dependencies

• Enables S4/Mamba to achieve Transformer-level performance at linear cost

2025–2026 extensions:

• Generalized HiPPO bases (Bessel, Jacobi, Gegenbauer)

• Learned HiPPO matrices → adaptive polynomial projection

• Multi-resolution HiPPO → multi-scale state representations

4.4 Discretization strategies: bilinear, zero-order hold, exact exponential

SSMs start in continuous time → must discretize for digital computation.

Common discretization methods:

MethodFormula (for x_{t+1} = A_d x_t + B_d u_t)ProsConsTypical Use CaseZero-order hold (ZOH)A_d = exp(A Δt), B_d = A⁻¹ (exp(A Δt) − I) BExact for constant input over intervalRequires matrix exp (expensive)High-fidelity physics simulationBilinear (Tustin)A_d = (I + A Δt/2) (I − A Δt/2)⁻¹, B_d = ...Preserves stability, simpleApproximate, can distort high frequenciesAudio processing, S4 defaultExact exponentialA_d = exp(A Δt), B_d = ∫₀^{Δt} exp(A s) B dsTheoretically exactComputationally heavy (padé approx or scaling-squaring)Mamba-2, long-step discretizationForward EulerA_d = I + A Δt, B_d = B ΔtExtremely cheapUnstable for stiff systemsQuick prototyping, short horizons

2025–2026 best practice:

• Mamba-2 uses exact exponential with fast diagonal + low-rank structure

• S4 family prefers bilinear for speed and stability

• Hybrid: exact exp for long steps, bilinear for short

4.5 Mamba-2 and structured variants: diagonal + low-rank, state expansion

Mamba-2 (Dao & Gu 2024) reformulates the SSM recurrence as:

y_t = C_t (A_d x_t + B_t u_t)

But with structured A_d (diagonalizable + low-rank correction) → enables fast parallel scan and kernel fusion.

Key innovations:

• Diagonal + low-rank structure → matrix multiplication becomes O(N) per token

• State expansion → effective state dimension up to 16M+ without quadratic cost

• Selective mechanism → input-dependent B_t and C_t (like attention’s query-key)

• Hardware-aware kernel → FlashAttention-style fusion → 2–8× faster than Mamba-1

Variants (2025–2026):

• Jamba / Jamba-1.5 → hybrid Mamba + Transformer blocks

• Vision Mamba (Vim, VMamba) → 2D selective scan

• MambaByte → byte-level tokenization + SSM

• MambaOut → pure SSM without attention fallback

4.6 Long-range dependency capture without quadratic attention cost

How SSMs achieve long-range modeling:

• Hidden state x_t compresses entire history into fixed-size vector (N dimensions)

• Linear recurrence allows exact parallel computation via associative scan

• HiPPO matrix ensures polynomial memory → theoretically captures dependencies up to length ~N²

• Selective mechanism (Mamba) adds context-sensitivity → rivals attention on many tasks

Empirical scaling (2025–2026 benchmarks):

• DNA / genomics (million-length sequences): Mamba outperforms Transformer

• Audio / speech (LibriSpeech, long-form ASR): linear-time advantage clear

• Long-horizon time-series (Weather, Traffic, ETTh): Mamba-2 + hybrids lead leaderboards

• Language modeling: Mamba-2 1.4B rivals Llama-3 8B on many tasks at 5–10× inference speed

No quadratic bottleneck → enables context windows of 1M+ tokens on consumer GPUs

4.7 Hybrid SSM + Transformer architectures (2025–2026 frontier)

Hybrid designs combine SSM linear scaling with Transformer’s global attention:

• Block alternation: SSM block → Attention block → repeat

• Jamba / Jamba-1.5 (2024–2025): Mamba layers dominate, attention only every k layers

• MambaFormer / Zamba hybrids → selective scan + sliding-window attention

• Vision hybrids (VMamba + Swin): local window attention + global SSM scan

• MoE + SSM → mixture-of-experts routing over Mamba experts

2025–2026 frontier trends:

• Depth-wise SSM → deeper stacks with residual connections

• Multi-scale SSM → hierarchical state representations

• SSM + Flow-matching → continuous-time generative modeling

• End-to-end learned discretization → meta-learn Δt and discretization scheme

• Hardware co-design → Triton/Pallas kernels for fused SSM + attention

SSMs (especially Mamba family) are now considered a legitimate third paradigm alongside Transformers and CNNs — offering the best speed–accuracy trade-off for long-context and continuous-time tasks.

5. Physics-Informed Neural Networks (PINNs) and Operator Learning

This chapter explores one of the most impactful intersections between differential equations and deep learning: using neural networks to solve, approximate, and learn solutions to physical systems governed by PDEs/ODEs. Physics-Informed Neural Networks (PINNs) and operator learning frameworks (FNO, DeepONet family) have become cornerstone methods in scientific machine learning (SciML), enabling data-driven discovery, surrogate modeling, and simulation acceleration in fields where traditional numerical solvers are too slow or require excessive computational resources.

5.1 Embedding differential equations into the loss function

Core idea of PINNs (Raissi, Perdikaris, Karniadakis 2019):

Instead of fitting data alone, train a neural network u_θ(x,t) to minimize a composite loss that includes:

1. PDE residual loss (collocation points inside domain Ω): L_PDE = (1/N_f) Σ || ℱ[u_θ](x_f, t_f) ||² where ℱ is the differential operator (e.g., ∂u/∂t − ν ∂²u/∂x² = 0 for Burgers’ equation)

2. Boundary/initial condition loss (points on boundary ∂Ω and t=0): L_BC/IC = (1/N_b) Σ || u_θ(x_b, t_b) − g(x_b, t_b) ||²

Total loss: L = λ_PDE L_PDE + λ_BC L_BC + λ_data L_data (if any labeled data)

Advantages:

• No need for labeled solution pairs — only boundary/initial conditions and PDE form

• Mesh-free: collocation points can be randomly sampled (Latin hypercube, Sobol sequences)

• Naturally incorporates physical laws → better extrapolation, fewer data points needed

Challenges:

• Balancing multiple loss terms (adaptive weighting, NTK-based balancing)

• Hard to enforce exact boundary conditions → soft constraints dominate

5.2 Soft vs hard constraints: collocation points, boundary/initial conditions

Soft constraints (standard PINN):

• Boundary/initial conditions added as loss terms → network approximates them approximately

• Pros: simple, differentiable, works with automatic differentiation

• Cons: can violate BC/IC significantly → poor accuracy near boundaries

• Mitigation: higher weight λ_BC/IC, causal training (start from t=0), gradient-enhanced losses

Hard constraints (strong enforcement):

• Parameterize u_θ(x,t) = u_BC(x,t) + (1 − x/L_x)(1 − t/T) v_θ(x,t) → v_θ is free neural net, multiplicative factor enforces BC/IC exactly

• Pros: exact satisfaction of BC/IC → better accuracy and convergence

• Cons: more complex architecture, harder to generalize to complex geometries

• Modern variants: use Fourier features, distance functions, or signed distance functions (SDF) to enforce boundaries

Collocation point sampling:

• Uniform random, Latin hypercube, Sobol sequences, residual-based adaptive sampling (RAR)

• RAR-G: residual-adaptive refinement with gradient-based importance → focuses on high-error regions

2025–2026 best practice:

• Hybrid: hard BC/IC for simple domains, soft + adaptive sampling for complex geometries

• Use NTK-PINN or gradient-balanced weighting to stabilize multi-objective optimization

5.3 Fourier Neural Operator (FNO): global spectral convolution in frequency domain

Fourier Neural Operator (Li et al. 2020) learns mappings between infinite-dimensional function spaces (PDE solution operators).

Core architecture:

• Lift input function a(x) → higher channel dimension v_0(x)

• Apply 4D FFT → frequency domain

• Pointwise linear transform in frequency space (global spectral convolution): v_{l+1}(k) = R_l(k) ⋅ v_l(k) (R_l is learned tensor, k = frequency)

• Truncate high frequencies (low-pass filter) → inverse FFT

• Local mixing (MLP on spatial grid) → stack layers

• Project final feature to output u(x)

Why spectral convolution wins:

• Global receptive field with O(N log N) cost (FFT) vs O(N²) for attention

• Resolution-invariant: trained on coarse grid, tested on finer grid

• Captures low-frequency dominant physics (Navier–Stokes, Darcy flow)

Variants:

• AFNO (Adaptive FNO): adaptive frequency truncation

• Geo-FNO: unstructured meshes via graph Fourier transform

• U-FNO / U-Net + FNO hybrids: combine local and global mixing

5.4 DeepONet and variants: operator regression for PDE families

Deep Operator Network (DeepONet, Lu et al. 2019) learns the solution operator G: a(·) → u(·) for families of PDEs.

Architecture:

• Branch net: encodes input function a(x) at fixed sensor points → b(a)

• Trunk net: encodes location (x,t) → t(x,t)

• Output: u(x,t) ≈ ⟨b(a), t(x,t)⟩ (inner product)

Advantages:

• Separable architecture → efficient for many queries

• Learns parametric PDE families (different initial conditions, coefficients, geometries)

Variants:

• PI-DeepONet (Physics-Informed): adds PDE residual loss → data-free training

• Fourier DeepONet: branch net uses Fourier features

• Multi-fidelity DeepONet: combines low- and high-fidelity simulations

5.5 PI-DeepONet, MIONet, and multi-fidelity operator learning

PI-DeepONet (Karniadakis group extensions):

• Combines DeepONet with PINN-style residual loss

• Learns operator without any labeled solution pairs — only PDE, BC/IC

MIONet (Multi-Input Operator Network):

• Multiple branch nets for different input functions (e.g., initial condition + boundary condition + forcing term)

• Generalizes to complex multi-physics PDEs

Multi-fidelity operator learning:

• Train on cheap low-fidelity simulations + few high-fidelity points

• Transfer learning or hierarchical DeepONet → large cost savings in CFD, climate modeling

2025–2026 frontier:

• MIONet + graph-based trunk nets → unstructured geometries

• Multi-fidelity + uncertainty quantification (Bayesian DeepONet)

5.6 Spectral methods in operator learning: Wavelet Neural Operators, Geo-FNO

Wavelet Neural Operator (Tripura & Chakraborty 2022–2023):

• Replace Fourier basis with wavelet basis → better localization for sharp fronts / discontinuities

• Multi-resolution wavelet transform → captures both global and local features

Geo-FNO (Li et al. extensions):

• Fourier transform on unstructured meshes via graph Fourier or manifold harmonics

• Handles complex geometries (airfoils, blood vessels, climate grids)

Other spectral operators:

• Spectral Neural Operator (learnable eigenfunctions)

• ChebNet / GraphONet hybrids: spectral graph convolutions + operator learning

Why spectral methods dominate operator learning:

• Diagonalization of translation-invariant or stationary operators

• Low-frequency bias aligns with physics (smooth solutions)

• Resolution invariance → train on coarse, infer on fine

5.7 Applications: fluid dynamics, climate modeling, molecular dynamics

Fluid dynamics (Navier–Stokes, Darcy flow):

• FNO / DeepONet predict velocity/pressure fields from initial conditions or boundary forcing

• 2–3 orders of magnitude speedup over traditional CFD solvers

• PINNs + FNO hybrids for turbulence modeling

Climate modeling:

• WeatherBench / ERA5 benchmarks: FNO / Geo-FNO forecast temperature, wind, precipitation

• Operator learning captures global atmospheric dynamics at fraction of GCM cost

• Multi-fidelity training: coarse-resolution + sparse high-res observations

Molecular dynamics:

• Neural ODE + PINNs for Langevin/Newtonian dynamics

• Diffusion models + score-based SDEs for sampling molecular conformations

• Operator learning for force fields → MD simulation acceleration

2025–2026 impact:

• PINNs + operator learning now routine in SciML toolkits (DeepXDE, NeuralPDE.jl, Modulus Sym)

• Hybrid solvers: PINN for coarse solution → classical solver refinement

• Real-world deployment: climate agencies, pharmaceutical companies, aerospace

This chapter demonstrates how embedding physics into neural architectures unlocks unprecedented simulation speed and data efficiency — transforming scientific computing from compute-bound to data-informed.

6. Diffusion Models and Score-Based Generative Modeling

Diffusion models and score-based generative modeling have become the dominant paradigm for high-quality image, video, audio, molecular, and time-series generation by 2025–2026. This chapter explains the continuous-time stochastic differential equation (SDE) formulation that unifies denoising diffusion probabilistic models (DDPM), score-based generative models (SGM), and modern ODE-based variants (flow-matching, rectified flow). It covers the forward/reverse processes, score estimation, the crucial ODE interpretation, practical neural SDE solvers, simulation-free training methods, and the cutting-edge frontiers.

6.1 Forward and reverse SDEs: Ornstein–Uhlenbeck, variance-preserving/variance-exploding

Forward diffusion process (gradual noising):

Most models define a continuous-time forward SDE that slowly adds noise to data x₀ ~ p_data:

dx = f(x,t) dt + g(t) dW

where W is a Wiener process (Brownian motion), f is the drift, g is the diffusion coefficient.

Common choices (Song et al. 2020–2021):

1. Variance Preserving (VP) SDE (most popular in DDPM-style models):

   • f(x,t) = −(1/2) β(t) x

   • g(t) = √β(t) → Ornstein–Uhlenbeck-like process → Variance of x_t is preserved ≈ 1 for all t ∈ [0,T]

2. Variance Exploding (VE) SDE:

   • f(x,t) = 0

   • g(t) = √(d[σ²(t)]/dt) → Pure Brownian motion with time-dependent variance → σ²(t) grows → variance explodes

3. sub-VP (intermediate): combines advantages of VP and VE for better sampling stability

Reverse-time SDE (generative direction):

The reverse process (denoising) is another SDE running backward from pure noise x_T ~ 𝒩(0,I) to data x_0:

dx = [f(x,t) − g(t)² ∇_x log p_t(x)] dt + g(t) dW̄

The crucial term is the score function s_θ(x,t) ≈ ∇_x log p_t(x) — the gradient of the log-probability density at time t.

6.2 Score function estimation: denoising score matching objective

Score-based generative modeling (Song & Ermon 2019–2021):

Train a time-dependent score network s_θ(x,t) to match the true score ∇_x log p_t(x) via denoising score matching:

L(θ) = E_{t,x_0,x_t} [ ‖ s_θ(x_t,t) − ∇{x_t} log p{0t}(x_t | x_0) ‖² ]

For Gaussian forward process, the conditional is tractable:

∇{x_t} log p{0t}(x_t | x_0) = − (x_t − √(1−σ_t²) x_0) / σ_t²

→ Denoising objective becomes:

L_simple(θ) = E_{t,x_0,ε} [ ‖ s_θ(√(1−σ_t²) x_0 + σ_t ε, t) + ε / σ_t ‖² ]

This is exactly equivalent to the simplified DDPM loss (Ho et al. 2020) — score models and DDPMs are mathematically the same under Gaussian assumptions.

Practical note:

• Score network usually shares weights with a denoising U-Net

• Time t is embedded via sinusoidal or learned embeddings

6.3 Continuous-time perspective: probability flow ODE vs. reverse-time SDE

Key insight (Song et al. 2021): the reverse diffusion process can be represented as a deterministic ODE (probability flow ODE):

dx = [f(x,t) − (1/2) g(t)² s_θ(x,t)] dt

→ No stochasticity in sampling → faster, more stable generation

Probability flow ODE vs reverse SDE:

AspectReverse SDEProbability Flow ODEWhen to useSampling pathStochastic (adds noise)DeterministicODE for faster/more reproducible samplingEquivalent marginalsYesYesBoth reach the same p_0(x)Training objectiveDenoising score matchingSameIdentical trainingGeneration speedSlower (many small noisy steps)Faster (larger deterministic steps)ODE variants (DPM-Solver, flow-matching)StabilityCan divergeMore stablePreferred for high-resolution / long trajectories

DPM-Solver family (2022–2025): high-order solvers tailored for probability flow ODE → 10–50× fewer steps than DDPM sampling.

6.4 Neural SDE solvers in diffusion: VP-SDE, VE-SDE, sub-VP

Neural SDE solvers integrate the reverse SDE/ODE with learned score:

• VP-SDE (variance preserving): most common in Stable Diffusion, Imagen, etc.

• VE-SDE (variance exploding): used in NCSN++ → better for very high-dimensional data

• sub-VP (Song et al.): combines VP drift with VE diffusion → improved sample quality and stability

Solvers used in practice (2025–2026):

• Euler–Maruyama (simple, noisy)

• Heun’s method (2nd-order predictor-corrector)

• DPM-Solver++ / UniPC / DEIS: high-order multi-step solvers for ODE path

• Ancestral sampling with restart (restart after k steps) → higher diversity

Fast samplers:

• DDIM (deterministic inversion of DDPM)

• PLMS / PNDM (pseudo-numerical methods)

• Consistency models (distillation to 1–4 steps)

6.5 Flow-matching and rectified flow: simulation-free training of ODE-based generative models

Flow-matching (Lipman et al. 2023) and rectified flow (Liu et al. 2022–2023) eliminate simulation during training:

Flow-matching objective: Train velocity field v_θ(x,t) so that dx/dt = v_θ(x,t) transports noise → data along straight-line paths (conditional flow matching) or optimal transport paths.

Loss: L = E [ ‖ v_θ(x(t),t) − u(t) ‖² ] where u(t) is target velocity of chosen path

Rectified flow:

• Straighten trajectories iteratively → learn straight-line ODE from noise to data

• No stochasticity at all → purely ODE-based generation

• Distillation to 1-step generation possible

Advantages over diffusion:

• Simulation-free training → no need to sample intermediate noisy states

• Straight paths → fewer steps, better mode coverage

• Competitive or superior FID/IS on ImageNet, zero-shot video generation

2025–2026 status:

• Flow-matching + rectified flow dominate new diffusion-style models (Stable Diffusion 3, Flux, SD3-Turbo)

• Latent rectified flow for efficient high-resolution generation

6.6 Diffusion bridges and stochastic interpolants

Diffusion bridges: Connect two arbitrary distributions p_0 and p_1 via a bridge process (Schrödinger bridge, generative bridge matching):

Train to sample paths from p_0 to p_1 (or vice versa) while matching marginals at t=0 and t=1.

Stochastic interpolants (Albergo & Vanden-Eijnden 2023): Generalize bridges to any interpolating path between noise and data → unify diffusion, flow-matching, OT-Flow.

Applications:

• Image-to-image translation without paired data

• Generative modeling of time-series transitions

• Molecular conformation sampling (diffusion bridges for protein backbones)

6.7 2025–2026 frontiers: diffusion on manifolds, Lie-group diffusion, diffusion for time-series

Diffusion on manifolds:

• Riemannian score-based models (Huang et al., 2022–2025): geodesic distances, Laplace–Beltrami operator

• Subspace diffusion, toroidal diffusion (periodic data)

• Lie-group diffusion: diffusion on SO(3), SE(3), SPD manifolds (protein structures, robotics poses)

Time-series diffusion:

• CSDI / TimeGrad / DiffTime: conditional score models for forecasting

• Score-based continuous-time models + Neural CDEs → irregular time-series generation

• DiffWave / AudioLDM 2: high-fidelity audio via latent diffusion + continuous-time priors

Other frontiers:

• Consistency trajectory models (distillation to few-step ODE)

• Rectified flow + flow-matching hybrids (Flux.1, SD3 family)

• Manifold-corrected diffusion (Lie-group equivariance)

• Diffusion for scientific data (climate fields, molecular dynamics trajectories)

Diffusion/score-based models remain the gold standard for high-fidelity generation, with continuous-time formulations (flow-matching, rectified flow, bridges) increasingly replacing traditional DDPM sampling.7. Time-Series Forecasting with Differential Equations

Time-series forecasting is one of the most practical and high-impact applications of differential equations in AI. By modeling temporal evolution as continuous dynamics, ODE-based and continuous-time models offer natural advantages for irregular sampling, long-horizon prediction, probabilistic forecasting, and multi-variate dependencies. This chapter compares classical approaches with modern deep learning hybrids, covers key architectures (Neural ODEs, Latent ODEs, Neural Hawkes Processes, continuous-time Transformers, SSM hybrids), and reviews benchmark performance on standard datasets as of 2025–2026.

7.1 Classical ODE-based forecasting vs. deep learning hybrids

Classical ODE-based forecasting Traditional methods fit parametric ODEs to data:

• Linear ODEs (ARIMA, exponential smoothing with trend/seasonality)

• Nonlinear ODEs (Lotka–Volterra for predator–prey, SIR/SEIR for epidemiology)

• Parameter estimation via least squares, maximum likelihood, or Kalman filtering

Limitations:

• Fixed functional form → poor generalization to complex real-world dynamics

• Struggle with high-dimensional, irregular, or multi-modal data

• Require manual feature engineering (lags, seasonality)

Deep learning hybrids (2018–2026 revolution):

• Replace fixed f(·) with neural network → universal approximation power

• Learn dynamics end-to-end from raw time-series

• Handle irregularity, missing values, exogenous variables, and probabilistic outputs

• Continuous-time formulation → resolution-invariant, memory-efficient for long horizons

Key advantages of deep hybrids:

• No need to specify equation form → data-driven discovery

• Capture nonlinear, multi-scale, and stochastic effects

• Integrate physics (PINNs-style residuals) or domain knowledge

• Probabilistic forecasting via latent variables or score matching

Trade-offs:

• Classical: interpretable, fast inference, low data requirement

• Deep hybrids: higher accuracy on complex data, but black-box, computationally intensive training

7.2 Neural ODEs for multivariate time-series (Time-series ODE-RNN)

Neural ODE-RNN (Chen et al. 2018 + extensions): Hybrid model that combines discrete updates (at observation times) with continuous evolution between observations.

Architecture:

• At observation time t_i: h(t_i^+) = GRU/MLP( h(t_i^-), x_i ) (incorporate new measurement x_i)

• Between t_i and t_{i+1}: dh/dt = f_θ(h(t), t) → solve ODE from t_i to t_{i+1}

• Prediction at future time: integrate ODE forward from last hidden state

Advantages for multivariate time-series:

• Handles missing/irregular data naturally (no imputation needed)

• Continuous memory → better long-term dependency capture than discrete RNNs

• Constant memory via adjoint method

Empirical usage:

• Strong on PhysioNet, MIMIC-III/IV (vital signs forecasting)

• Competitive on ETTh/ETTm (electricity transformer temperature) when combined with attention

Modern refinements (2025–2026):

• ODE-RNN + selective SSM scan (Mamba-style) → faster inference

• Augmented ODE-RNN (extra state variables) → improved numerical stability

7.3 Temporal Point Processes via Neural Hawkes Processes (intensity ODEs)

Temporal Point Processes (TPP) model event times (e.g., earthquakes, trades, hospital admissions) as point processes.

Hawkes Process (self-exciting point process): Intensity λ(t) = μ + ∑_{t_i < t} α exp(−β (t − t_i))

Neural Hawkes Process (Xiao et al. 2019 + extensions):

• Replace parametric intensity with neural network: λ_θ(t) = f_θ(h(t))

• Hidden state h(t) evolves via ODE between events: dh/dt = g_θ(h(t)) (decay or excitation dynamics)

• At event time t_i: jump h(t_i^+) = h(t_i^-) + jump_θ(h(t_i^-))

Modern variants:

• Neural Temporal Point Process with ODE (Neural TPP): full continuous-time intensity ODE

• Continuous-time Hawkes with Neural CDE → path-dependent excitation

• Log-normal intensity or Gated recurrent TPP → better for long-memory events

Applications:

• MIMIC-IV event prediction (admissions, procedures)

• Financial tick data (order book dynamics)

• Epidemic modeling (case arrival times)

7.4 Latent ODEs for irregular time-series (Latent-ODE, GRU-ODE-Bayes)

Latent ODE (Chen et al. 2018 extension):

• Introduce latent stochastic process z(t) evolving via ODE

• Observations x_i = decoder(z(t_i)) + noise

• Encoder maps irregular observations → initial latent z(0) (amortized inference)

• Forward prediction: integrate latent ODE from z(0)

GRU-ODE-Bayes (De Brouwer et al. 2019):

• GRU-style discrete update at observations + continuous ODE evolution between

• Bayesian treatment: uncertainty via dropout or variational inference

• Probabilistic forecasting with uncertainty bands

Advantages for irregular data:

• Naturally handles asynchronous multi-variate series

• Latent state captures hidden confounders

• Uncertainty quantification critical for healthcare/finance

2025–2026 extensions:

• Latent Neural CDE → path-driven latent dynamics

• Latent SSM hybrids (Mamba + latent ODE) → scalable uncertainty

7.5 Continuous-time transformers and ODE-augmented attention

Continuous-time Transformers (2022–2025):

• Replace discrete positional encoding with continuous-time embeddings (Fourier, RBF)

• Attention computed at arbitrary times via ODE integration or interpolation

• ODE-augmented attention: hidden states evolve continuously between tokens

Key designs:

• COT (Continuous-Time Transformer): attention kernel integrated via ODE

• TimeSformer + ODE hybrids → video understanding with continuous temporal mixing

• Neural CDE + Transformer (Kidger et al. extensions): CDE backbone + global attention head

Benefits:

• Resolution-invariant sequence modeling

• Better handling of long, irregular horizons

• Combines global context (attention) with local continuous dynamics (ODE/CDE)

7.6 State-space models for probabilistic forecasting (DeepState, Chronos-SSM hybrids)

DeepState (2018–2020):

• Classical SSM with RNN-learned emission/transition → probabilistic output via Gaussian likelihood

• Extended to GLU-based variants for scalability

Chronos-SSM hybrids (2024–2026):

• Chronos (Amazon, 2024): tokenizes time-series → Transformer backbone

• Chronos-SSM: replace Transformer with Mamba/SSM backbone → linear scaling

• Probabilistic output: Gaussian mixture or score-based heads

Advantages:

• SSMs excel at long-horizon probabilistic forecasting (ETTh, Weather, Traffic)

• Linear complexity → handles 100k+ length series

• Uncertainty quantification via latent state sampling or score heads

2025–2026 leaders:

• Mamba-2 + probabilistic head → SOTA on many long-horizon benchmarks

• SSM + flow-matching → continuous-time probabilistic paths

7.7 Benchmark performance: ETTh, Electricity, Weather, Traffic, MIMIC-IV

Standard benchmarks & 2025–2026 leaderboard trends:

• ETTh1/ETTh2 / ETTm1/ETTm2 (Electricity Transformer Temperature, multivariate, 96–720 step horizons): Mamba-2 hybrids, Chronos-SSM, Neural CDE + attention → MSE/MAE top ranks PatchTST / iTransformer still competitive but SSMs win on longer horizons

• Electricity / Traffic (long-horizon, high-dimensional): Mamba family + DeepState-style probabilistic heads → best CRPS (continuous ranked probability score) Neural CDEs strong when irregularity present

• Weather (ERA5-derived, global fields): FNO + PINN hybrids lead for spatial-temporal forecasting Continuous-time SSMs (Mamba-2) close gap on univariate series

• MIMIC-IV (ICU time-series, mortality/length-of-stay): Neural CDE + Latent ODE → top AUROC/AUPRC GRU-ODE-Bayes and Neural Hawkes strong for event prediction Continuous-time Transformers competitive when events are dense

Overall trend:

• Irregular → Neural CDE / Latent ODE dominant

• Regular long-horizon → Mamba-2 / SSM hybrids lead

• Probabilistic → SSM + score/flow heads or latent variables win

• Hybrids (SSM + Transformer + ODE) → best overall Pareto front

Continuous-time and ODE-based models have become essential for state-of-the-art time-series forecasting, especially when irregularity, long horizons, or probabilistic outputs are required.8. Stability, Numerical Challenges, and Theoretical Insights

This chapter addresses the practical and theoretical difficulties that arise when training and deploying neural differential equations (Neural ODEs, Neural CDEs, SSMs, diffusion models, etc.). While continuous-time models offer elegant mathematical structure and strong inductive biases, they introduce unique numerical and stability challenges that discrete-layer architectures largely avoid. Understanding these issues — stiffness, gradient explosion, spectral properties of discretizations, Lipschitz control, infinite-width limits, frequency-domain biases, and generalization theory — is essential for reliable, high-performance continuous models in 2025–2026 practice.

8.1 Stiffness in neural differential equations: detection and adaptive solvers

Stiffness occurs when a system has widely separated time-scales (some components evolve very rapidly, others very slowly). In neural differential equations, stiffness is extremely common because neural vector fields can have eigenvalues spanning many orders of magnitude.

Detection signs (during training or inference):

• Solver statistics: extremely small step-sizes (e.g., < 10⁻⁶), large number of rejected steps, excessive function evaluations

• Trajectory inspection: sudden jumps, oscillations, or numerical blow-up

• Gradient norms: exploding/vanishing gradients during adjoint backward pass

• Loss spikes or NaNs after a few epochs

• Profiling tools: diffrax/tsit5/dopri5 return step-size history and error estimates

Common causes in neural models:

• ReLU-like activations → piecewise linear vector fields → discontinuous derivatives

• Deep residual blocks → large Lipschitz constants in some directions

• High-dimensional latent spaces → heterogeneous eigenvalue spectrum

• Long time-horizons → accumulated numerical error

Adaptive solvers (standard mitigation):

• Dopri5 / Tsit5 (explicit RK): good for non-stiff, but fail quickly on stiff problems

• KenCarp4 / Rodas5P / ESIRK (implicit/exponential Rosenbrock): designed for stiff ODEs → take much larger steps

• LSODA (automatic switching between non-stiff Adams and stiff BDF) → robust default in many SciML libraries

• Proj-integrators (projection onto manifold) → enforce stability constraints

2025–2026 best practices:

• Start with Tsit5 or Dopri5 → monitor step-size stats

• Switch to KenCarp4 or Rodas5P if step-size drops below 10⁻⁵–10⁻⁶

• Use stiffness-aware preconditioning (learned diagonal scaling of vector field)

• Curriculum learning: start with short horizons → gradually increase

8.2 Symplectic and structure-preserving integrators for Hamiltonian systems

Many physical systems are Hamiltonian (energy-conserving): robotics, molecular dynamics, celestial mechanics.

Standard integrators (Euler, RK4) do not preserve energy → artificial dissipation or explosion over long trajectories.

Symplectic integrators preserve the symplectic structure (phase-space volume, energy bounds):

• Leapfrog / Verlet (second-order, simplest symplectic)

• Yoshida / Forest-Ruth (fourth-order symplectic)

• Implicit midpoint (symplectic Runge–Kutta)

• Symplectic Euler (cheap, first-order)

Neural structure-preserving variants:

• Hamiltonian Neural Networks (HNN, Greydanus et al. 2019): learn energy function H_θ(q,p) → vector field = ∇H_θ via symplectic form

• Symplectic Neural ODEs (2021–2025): use symplectic integrators in forward/backward pass

• H-Symplectic ODE (2023+): enforce symplectic structure via Lie-group constraints or canonical coordinates

Benefits:

• Long-term stability without artificial damping

• Energy conservation → physically plausible trajectories

• Better extrapolation beyond training time horizon

2025–2026 usage:

• Molecular conformation sampling, robotics control, climate subgrid modeling

• Combined with flow-matching → energy-conserving generative flows

8.3 Spectral stability of discretization schemes

Discretization turns continuous ODE dy/dt = f(y) into discrete map y_{n+1} = Φ(y_n).

Spectral stability:

• Eigenvalues of Jacobian of Φ must lie inside unit disk for stability

• For explicit methods (Euler, RK4): stability region limited → small step-size needed for stiff problems

• Implicit methods (BDF, trapezoidal): unbounded stability region → stable for large steps on stiff problems

Key results for neural ODEs:

• Forward Euler: stability |1 + λ Δt| ≤ 1 → Δt ≤ 2/|λ_max|

• RK4: larger stability region, but still explicit → fails on very stiff systems

• Implicit midpoint / BDF: A-stable → unconditionally stable for linear negative eigenvalues

• Exponential integrators (exp(A Δt)): exact for linear systems → preserve spectrum perfectly

Practical implication:

• Use bilinear or exact exponential discretization in SSMs/Mamba → excellent spectral stability

• Avoid forward Euler for production models with stiff dynamics

8.4 Lipschitz constants, gradient norms, and explosion prevention

Lipschitz continuity of vector field f(h,t) guarantees existence/uniqueness (Picard theorem) and controls gradient growth.

Gradient explosion in adjoint method:

• Backward pass solves da/dt = −aᵀ ∂f/∂h → if ‖∂f/∂h‖ large → a explodes backward in time

• Forward pass explosion if ‖f‖ large over long horizons

Prevention techniques (2025–2026 standard toolkit):

• Spectral normalization / weight normalization on f_θ layers → bound ‖∂f/∂h‖ ≤ 1

• Lipschitz regularization: add penalty λ ‖∂f/∂h‖₂ during training

• Gradient clipping (global norm or per-layer)

• Augmented state (ANODE): extra dimensions spread eigenvalue spectrum

• Learned time-reparameterization → slow down fast directions

• Curriculum on time horizon T → start small, increase gradually

8.5 Infinite-width limits: Neural ODEs → kernel regression with path signatures

Infinite-width Neural ODE → continuous-depth residual network at initialization.

Theoretical result (2020–2024): In infinite width, Neural ODE training dynamics become kernel regression with the Neural Controlled Differential Equation kernel (or path signature kernel).

Path signatures (Lyons 1998): Iterated integrals along path X(t):

S(X)_{(i₁…iₖ)} = ∫ … ∫ dX^{i₁} … dX^{iₖ}

→ Complete feature set for continuous paths → universal approximation for path functionals

Infinite-width Neural CDE → kernel regression on path signatures → principled generalization bounds

Implications:

• Explains why Neural CDEs generalize well on irregular data

• Path signatures provide theoretical inductive bias for continuous-time models

• Connects to rough path theory → stability and robustness guarantees

8.6 Frequency-domain analysis: Fourier perspective on ODE learning dynamics

Fourier view of learning dynamics (Rahaman et al. 2019 extensions to continuous case):

• Neural ODEs exhibit strong spectral bias → learn low-frequency components first

• In frequency domain: vector field f(h,t) acts as convolution-like operator

• Eigenfunctions of linear ODEs are exponentials → low-frequency modes decay slowly

2025–2026 insights:

• Continuous-time spectral bias stronger than discrete → prefers smooth trajectories

• Stiffness linked to high-frequency modes → fast-decaying eigenvalues

• Frequency-aware regularization (high-frequency penalization) → accelerates convergence

• Fourier Neural Operator connection: global spectral mixing helps overcome bias

Visualization:

• Fourier coefficients of learned trajectories decay rapidly for high frequencies

• Early training captures smooth trends → late training fits wiggles

8.7 Generalization bounds for continuous-depth models

Theoretical generalization results (2021–2026):

• Neural ODEs → generalization bounds scale with Lipschitz constant of f_θ and time horizon T → Rademacher complexity O(√(Lip(f) T / n)) where n = samples

• Neural CDEs → bounds involve signature norm and path variation → robust to irregularity

• Mamba / S4 family → NTK-style analysis shows polynomial memory → low generalization error for long dependencies

• Flow-matching / rectified flow → straight-line paths → tighter bounds than curved diffusion paths

Practical implications:

• Regularize Lip(f) and T → stronger generalization

• Use path signatures or log-signatures as features → provable universality

• Infinite-depth limit → kernel regression → connects to classical statistical learning theory

This chapter highlights why continuous models, despite their elegance, demand careful numerical and theoretical treatment — but when handled properly, deliver superior performance on long-range, irregular, and physical time-series tasks.

9. Advanced Applications and 2025–2026 Frontiers

This final chapter surveys the most exciting real-world applications and emerging research directions in differential-equation-based AI as of early 2026. These frontiers leverage continuous-time dynamics to solve previously intractable problems in uncertainty-aware modeling, molecular science, climate prediction, generative modeling, geometric deep learning, and scalable architectures — while highlighting the key open challenges that remain at trillion-parameter scales.

9.1 Neural SDEs for uncertainty quantification and stochastic optimal control

Neural Stochastic Differential Equations (SDEs) extend Neural ODEs by adding a diffusion term:

dX_t = f_θ(X_t, t) dt + g_θ(X_t, t) dW_t

Uncertainty quantification:

• Forward pass samples multiple trajectories → Monte-Carlo ensemble for predictive distributions

• Backward pass uses adjoint SDE (stochastic adjoint method) → memory-efficient gradients

• Applications: Bayesian filtering, risk-sensitive forecasting, safe reinforcement learning

Stochastic optimal control:

• Neural SDEs model stochastic dynamics in control problems

• Policy gradient or actor-critic methods optimize control u(t)

• Hamilton–Jacobi–Bellman equation approximated via neural PDE solvers

• Key 2025–2026 works: Neural SDE control with score-based priors, diffusion-guided MPC

Practical status:

• torchsde / diffrax.sde libraries dominant

• Strong results on stochastic robotics, financial option pricing, epidemic control under uncertainty

9.2 Diffusion-based molecular dynamics and protein structure generation

Diffusion models on molecular manifolds:

• Forward SDE adds noise to atomic coordinates / torsion angles / graph embeddings

• Reverse SDE denoises toward valid 3D structures

• Equivariant score networks (E(3)-invariant or SE(3)-equivariant) ensure rotational/translational invariance

Key advances 2024–2026:

• DiffDock / RFdiffusion / Chroma → state-of-the-art protein–ligand docking and de novo design

• FrameFlow / EquiFold → flow-matching on rigid-body frames (SE(3)) → faster, more stable sampling

• Torsional diffusion → diffusion in dihedral angle space → avoids coordinate singularities

• Coarse-grained + all-atom diffusion pipelines → multi-resolution generation

Applications:

• Antibody design, enzyme engineering, small-molecule drug discovery

• AlphaFold3-style multimodal diffusion (protein + ligand + nucleic acid)

2025–2026 trend:

• Diffusion bridges for conformational transitions

• Active learning loops: generate → simulate → refine score model

9.3 Operator learning for climate simulation and weather forecasting

Operator learning approximates infinite-dimensional maps (e.g., weather at t → weather at t+Δt).

FNO / Geo-FNO dominance:

• Global spectral mixing → captures teleconnections (e.g., ENSO effects)

• Resolution-invariant → train on 0.25° grid, infer at 0.1°

• Multi-step rollout stable over weeks (FourCastNet, GraphCast hybrids)

2025–2026 breakthroughs:

• FourCastNet v2 / FengWu / GraphCast 2 → FNO + graph + physics hybrids → beat ECMWF IFS on many variables

• ClimaX / Prithvi WxC → foundation models pretrained on ERA5 + CMIP6 → zero-shot forecasting

• Operator learning + data assimilation → hybrid NWP + ML → improved initialization

• Multi-fidelity + uncertainty → low-res ensembles + high-res correction via DeepONet

Impact:

• 10⁴–10⁶× speedup over traditional GCMs

• Probabilistic ensemble forecasting at operational cost

9.4 Continuous normalizing flows (CNFs) and rectifying flows

Continuous Normalizing Flows (CNFs):

• Density evolution via ODE: d log p / dt = − ∇ · f_θ(x,t)

• Change-of-variables formula → exact likelihood

• Training: maximize log-likelihood via Hutchinson trace estimator

Rectified flows (Liu et al. 2022–2025):

• Straighten curved flow trajectories iteratively → learn linear ODE paths from noise to data

• Simulation-free training (conditional flow matching)

• Distillation to 1–4 step generation → extremely fast inference

2025–2026 status:

• Flow Matching + Rectified Flow → backbone of Flux.1, SD3, AuraFlow

• Latent rectified flow → efficient high-resolution image/video generation

• CNFs + diffusion hybrids → best log-likelihood on density estimation benchmarks

9.5 Lie-group ODEs and equivariant continuous models

Lie-group ODEs evolve states on manifolds with symmetry:

dX/dt = f_θ(X) X (left-invariant vector field on Lie group G)

Key groups in AI:

• SO(3)/SE(3): 3D rotations/translations → equivariant protein / robotics models

• SPD(n): symmetric positive-definite matrices → covariance estimation, diffusion on SPD

• Unitary group U(n): complex-valued networks, quantum simulation

Equivariant continuous models:

• LieConv / LieResNet → group-equivariant convolutions via exponential maps

• Equivariant Neural ODEs → preserve group action under continuous dynamics

• SE(3)-Diffusion → diffusion on rigid-body configurations

• Gauge-equivariant flows → for lattice gauge theories in physics

2025–2026 applications:

• Equivariant diffusion for molecular conformers

• Lie-group state-space models for robotics control

• Continuous equivariant transformers

9.6 Hybrid discrete-continuous architectures (Mamba + ODE layers)

Hybrid designs combine discrete efficiency with continuous inductive bias:

• Mamba + Neural ODE blocks: discrete Mamba scan + continuous ODE refinement

• Transformer + ODE-augmented attention: attention at discrete tokens + ODE evolution between

• Jamba / Zamba hybrids: Mamba layers for local mixing + sparse attention for global context

• Continuous-discrete flow-matching: discrete tokens → continuous paths → discrete readout

2025–2026 frontier:

• Depth-wise continuous stacking (Mamba-2 + ODE residuals)

• Multi-scale hybrids (short-range discrete + long-range continuous)

• Learned discretization + hybrid solvers → end-to-end differentiable

9.7 Open challenges: scaling to trillion-parameter continuous models, stiffness at extreme scales, theoretical convergence rates

Scaling to trillion parameters:

• Memory bottleneck: adjoint method still requires storing solver states (checkpointing helps but limited)

• Communication overhead in distributed training of continuous models

• Solver synchronization across shards → novel parallel-in-time methods needed

Stiffness at extreme scales:

• Trillion-param vector fields → eigenvalue spectrum spans >20 orders → ultra-stiff

• Classical implicit solvers too expensive → need learned, hardware-aware stiff solvers

• Preconditioning via spectral normalization at scale → open research

Theoretical convergence rates:

• Finite-width Neural ODEs/CDEs → NTK-like analysis incomplete for long horizons

• Generalization bounds weak for continuous-depth models → need tighter path-dependent Rademacher complexity

• Stochastic convergence of flow-matching / rectified flow → early theoretical results promising but incomplete

• Operator learning convergence → data requirements for PDE families still exponential in some regimes

2025–2026 open directions:

• Hardware-native continuous models (Triton/Pallas kernels for ODE solvers)

• Adaptive structure (learn when to switch discrete ↔ continuous)

• Theoretical unification of SSMs, Neural ODEs, diffusion, and flow-matching

• Extreme-scale benchmarks (trillion-param Neural ODE on climate / genomics)

Continuous-time and differential-equation-based AI has matured into a major pillar of frontier modeling — rivaling Transformers in efficiency and surpassing them in physical realism and long-range reasoning.

PREVIOUS PAGE INDEX PAGE NEXT PAGE

Join AI Learning

Get free AI tutorials and PDFs