On the Regularity of Generalized Conjugate Functions

Regularity Properties of Generalized Conjugate Functions Induced by Abstract Couplings

Introduction

The study of generalized conjugate functions via non-Euclidean or non-standard coupling functions has critical implications for nonsmooth and nonconvex optimization, variational analysis, and optimal transport. This paper systematically investigates how regularity properties—existence, single-valuedness, continuity, Lipschitz continuity, and higher-order differentiability—of the generalized proximal mappings and their associated envelopes (i.e., generalized conjugates) can be established under minimal, verifiable assumptions on the coupling, extending classical results on Moreau envelopes and subdifferentials to highly generic settings.

Crucially, the framework utilizes a fully abstract coupling $\Phi(x, y)$, thus encompassing as special cases the Euclidean (quadratic) coupling, left and right Bregman divergences, Legendre kernels, anisotropic couplings, and $\varphi$-divergences. The paper's contributions unify regularization results that previously required ad hoc analysis for each coupling and provides new regularity criteria for couplings not previously analyzed.

Theoretical Framework

$\Phi$-Convexity and Generalized Conjugates

The paper adopts $\Phi$-convexity as a unifying concept, where a function $f$ is $\Phi$-convex if it can be expressed as the pointwise supremum of affine shifts of $\Phi(\cdot, y)$ over a suitable parameter set. The associated $\Phi$-conjugate $f^\Phi$ generalizes the Fenchel conjugate:

$f^\Phi(y) = \sup_{x \in X} \Phi(x, y) - f(x).$

The $\varphi$0-subdifferential, extending classical subgradients, is central to connecting the minimizers of $\varphi$1 to generalized subgradients.

Regularity properties of these conjugates and associated generalized proximal mappings are fundamentally tied to twist conditions and local invertibility properties of the partial gradient $\varphi$2. The paper examines a spectrum of couplings satisfying one-to-one (twist) or local strong twist properties, characterizing when such invertibility leads to the existence and continuity of $\varphi$3-subdifferentials.

Generalized Proximal Mappings and Regularity

A generalized proximal mapping is the inverse of the $\varphi$4-subdifferential:

$\varphi$5

which reduces to the classical proximal mapping for Euclidean couplings and to Bregman or anisotropic proximity operators in more general settings.

Verifiable conditions for existence, single-valuedness, and local Lipschitz continuity of this mapping are provided. These rely on generalized prox-regularity, a property guaranteeing the subgradient inequality's local stability under the $\varphi$6-coupling, and a graphical localization (Attouch-Wets) of subdifferentials.

The analysis draws on a nonsmooth implicit function theorem for generalized equations. This technical tool leverages second-order variational analysis (including strict proto-differentiability and graphical regularity), extending classical invertibility conditions to nonsmooth and set-valued mappings.

Main Results

Existence and Continuity of $\varphi$7-Subgradients

Utilizing Ekeland’s variational principle, the paper proves that for $\varphi$8 a local strong twist and $\varphi$9 finite, lower semicontinuous, and $\Phi$0-convex, the $\Phi$1-subdifferential is nonempty and compact. Continuity of the $\Phi$2-subdifferential (in the singleton case) is characterized in terms of continuity of the gradient and the coupling's twist map.

Differentiability and Twice Differentiability

A central result establishes that under a generalized prox-regularity condition, which extends the Euclidean concept to general couplings, the mapping $\Phi$3 is locally single-valued and continuous, and $\Phi$4 is continuously differentiable with gradient:

$\Phi$5

The strict twice differentiability of $\Phi$6 is obtained under strong smoothness assumptions: $\Phi$7 must be strictly twice epi-differentiable (a property standard in modern variational analysis), and the coupling must be $\Phi$8 with appropriate invertibility of mixed Hessians. The Hessian formula generalizes all existing known special-case results:

$\Phi$9

Lipschitz Continuity

If $\Phi$0 is prox-regular and the coupling's quadratic term’s largest eigenvalue is sufficiently small, the generalized proximal mapping is Lipschitz continuous, leading directly to Lipschitz differentiability of the envelope.

Notably, for Bregman, anisotropic, and $\Phi$1-divergence couplings, this paper provides the first rigorous higher-order regularity results for the associated envelope functions.

Consequences for Optimization and Analysis

These results provide new theoretical foundations for:

• Design and Analysis of Algorithms: The continuous differentiability and strict smoothness of generalized envelopes allow the extension of fast algorithms (such as quasi-Newton and line search methods) from the convex Euclidean Moreau setting to wide classes of nonconvex, nonsmooth problems using rich couplings.
• Subgradient and Proximal Algorithms: The identification of explicit conditions under which envelopes serve as smooth merit functions directly enables the extension of global convergence guarantees for subgradient-type methods in nonconvex settings.
• Optimal Transport: In the context of $\Phi$2-concavity, these regularity results translate to explicit smoothness conditions for transport maps emerging from generalized duality, extending classical results (Villani, Gangbo & McCann) by providing explicit higher-order differentiability guarantees under minimal assumptions.

Theoretical and Practical Implications

The explicit, verifiable, and minimal assumptions required for the main regularity theorems have immediate consequences in augmenting the scope of regularization techniques in nonsmooth, nonconvex analysis, machine learning, and variational problems. In practical terms, the identification of "twist" and prox-regularity properties for new couplings allows designers of optimization algorithms greater freedom in tailoring regularizers, divergence metrics, and envelope constructions beyond classical quadratic norms. This is of particular importance in applications such as robust learning, optimal transport, and high-dimensional statistical modeling, where classical smoothness properties often fail.

Theoretically, the results unify a diversity of regularization regimes under the umbrella of $\Phi$3-convexity and provide new tools for higher-order variational analysis—including Hessian formulae that are compatible with automatic differentiation and implicit function theory, facilitating analysis and computation in large scale and nonconvex regimes.

Future Directions

Several fruitful directions emerge from this work. The development of generalized forward-backward merit functions and line search schemes for composite minimization under abstract coupling is suggested, as is the investigation of global convergence rates for stochastic and deterministic subgradient-type methods in the $\Phi$4-convex setting, informed by the envelope's smoothness. Given the unification with optimal transport, further exploration of generalized smoothness notions in duality, geometric analysis, and deep learning architectures parameterized via abstract couplings is warranted.

Conclusion

This work consolidates the regularity theory of generalized conjugate functions and their proximal maps, delivering practical and theoretically rigorous smoothness criteria for a wide range of couplings pertinent to optimization, variational analysis, and optimal transport. The results unify disparate strands of envelope function theory and open new avenues for algorithm design and analysis in nonconvex, nonsmooth environments.