Quasi-Constant Modulus Design for Nonlinearity-Tolerant Geometric Shaped Four Dimensional Modulation Format

Quasi-Constant Modulus Design for Nonlinearity-Tolerant 4D Modulation

Overview

The paper "Quasi-Constant Modulus Design for Nonlinearity-Tolerant Geometric Shaped Four Dimensional Modulation Format" (2604.18435) introduces a novel approach to the problem of nonlinearity mitigation in optical fiber communication. By leveraging the quasi-constant modulus (QCM) property in the design of four-dimensional (4D) geometric-shaped quadrature amplitude modulation (QCM-QAM) constellations, it attains increased tolerance to fiber nonlinearities at high spectral efficiencies (SEs). The work provides both theoretical analysis and empirical validation of QCM-QAM formats, demonstrating robust performance in wavelength-division multiplexing (WDM) systems across both standard single-mode fiber (SSMF) and non-zero dispersion-shifted fiber (NZDSF).

Motivation and Theoretical Foundations

The increasing demand for optical transmission capacity necessitates modulation formats with higher SEs. Conventional PM-M-QAM and constellation shaping (probabilistic or geometric) have reached maturity, but the nonlinearity of the fiber channel—manifesting as self-phase modulation (SPM) and cross-phase modulation (XPM)—limits further scaling. The constant-modulus constraint, previously shown to mitigate nonlinear interference (NLI) due to suppression of signal energy fluctuations, becomes impractically restrictive as constellation cardinality and SE rise.

This paper relaxes the constraint to a quasi-constant modulus, where 4D symbol energy is approximately constant. Theoretical analysis, based on simplified nonlinear interference models, shows that the QCM property retains most of the mitigation benefits of the ideal constant-modulus, notably suppressing variance in the power difference term ($P_{ry}(t) - P_{avg}$), which is a driver of nonlinear phase noise in SPM and XPM.

QCM-QAM Construction and Principles

QCM-QAM constellations are constructed by partitioning the 2D QAM constellation for each polarization into internal and external points (energy-based division), and pairing these such that internal and external points never co-occur within a 4D symbol. This reduces the total variance in symbol energy across the 4D space, closely approximating a constant modulus. This cross-polarization constraint actuates the QCM property, at the cost of reduced cardinality compared to unconstrained PM-M-QAM.

For example, PM-32QAM would nominally yield $32 \times 32 = 1024$ symbols, but QCM-QAM constrains the selection to $32 \times 16 = 512$ symbols, reducing the SE from 10 to 9 bits/4D-symbol. At higher M, similar reductions are seen, facilitating 2048-ary and 8192-ary QCM-QAM formats with 11 and 13 bits/4D-symbol, respectively.

A Gray-like labeling strategy is utilized given the difficulty of strict Gray code mappings in high-SE 4D constellations. Additionally, the QCM-QAM approach induces inter-polarization dependencies and limits achievable information rates compared to conventional schemes.

Nonlinearity Tolerance Analysis

The suppression of $P_{ry}(t) - P_{avg}$ power fluctuations, quantitatively evaluated via power spectral density (PSD) measurements, provides direct evidence of phase noise mitigation. Compared to set-partitioning QAM (SP-QAM), QCM-QAM achieves PSD reductions of 3.62 dB, 2.83 dB, and 2.56 dB at SEs of 9, 11, and 13 bits/4D-sym, respectively. This impact scales with SE, but remains significant even as constellation size grows.

Simulation of unrepeatered WDM transmission systems, typical for data center interconnect applications, incorporates both SSMF and NZDSF scenarios. Transmission metrics include generalized mutual information (GMI) and effective SNR ($SNR_{eff}$), calculated post-digital signal processing (DSP) and FEC.

Numerical Results and Comparative Analysis

QCM-QAM outperforms SP-QAM in nonlinear regimes, with strong numerical results:

• GMI gains in SSMF of 0.22, 0.09, and 0.21 bits/4D-sym, and in NZDSF of 0.24, 0.10, and 0.22 bits/4D-sym for 512-, 2048-, and 8192-ary formats.
• Effective SNR gains in SSMF: 0.59 dB, 0.58 dB, 0.54 dB. In NZDSF: 0.61 dB, 0.60 dB, 0.55 dB.
• Transmission reach extensions: In SSMF: 1.6%, 0.9%, 1.7%. In NZDSF: 1.7%, 1.5%, 1.8%.

These improvements are most pronounced at higher launch powers, where nonlinear effects scale nonlinearly (approximately cubic with power). Minor GMI degradation is observed in low-power, linear regimes due to the labeling constraint, but this is offset by superior performance in nonlinear scenarios.

Optimal launch power for QCM-QAM is consistently higher than for SP-QAM, validating its resilience to fiber nonlinearities. The effective SNR and GMI curves exhibit clear shifts, substantiating the theoretical predictions.

Practical and Theoretical Implications

The QCM-QAM family provides an efficient compromise between the strict constant-modulus requirement and SE scalability. Its robust nonlinear mitigation is germane for systems operating in high-power, high-SE regimes (e.g., data center interconnects). The practical implications include longer transmission reach, better spectral utilization, and enhanced tolerance to Kerr nonlinearity—facilitating future network upgrades and higher bandwidth services.

Theoretically, the approach demonstrates that cross-polarization symbol mapping constraints offer substantial NLI suppression even as constellation sizes grow, suggesting further potential in multi-dimensional modulation format optimization.

The modest reduction in linear regime performance is an acceptable tradeoff, with opportunities for further optimization via advanced labeling (e.g., binary switching optimization) and joint optimization of symbol coordinates and bit mappings.

Future Prospects in Optical Transmission

The paper suggests further developments, including the use of adaptive optimization algorithms such as binary switching optimization to enhance bit-to-symbol mappings under the QCM constraint. Integrating QCM property into broader constellation optimization frameworks could produce constellations that jointly maximize GMI and mitigate NLI, particularly in DP-4D and higher-dimensional settings.

This work positions QCM-QAM as a foundation for future high-SE modulation formats in optical fiber communications, balancing information rate, reach, and nonlinear tolerance. The paradigm may extend to emerging scenarios with diverse fiber types, higher channel counts, and advanced DSP/FEC architectures.

Conclusion

By introducing the quasi-constant modulus design for 4D geometric-shaped QAM, the paper delivers a compelling solution to the challenge of fiber nonlinearity in high-SE optical transmission. QCM-QAM achieves superior nonlinear tolerance and transmission reach relative to conventional set-partitioning QAM, confirmed by both theoretical models and empirical simulation results across SSMF and NZDSF. The underlying principles are scalable, and further refinements in constellation labeling and optimization algorithms are likely to widen its applicability. This work advances the state of the art in modulation format design for nonlinearity-tolerant optical communication systems.