Share Email Print

Proceedings Paper

Overfitting of artificial-neural-network-based nonlinear equalizer for multilevel signals in optical communication systems
Author(s): Kai Ikuta; Yuta Otsuka; Moriya Nakamura
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

We investigated the problem of overfitting of artificial neural networks (ANNs) which are used for digital nonlinear equalizers in optical communication systems. ANN-based digital signal processing (DSP) can be used in the time domain or the frequency domain to compensate for the optical nonlinearity. We have reported the advantages of ANN over Volterra series transfer function (VSTF) in terms of the computational complexity. However, when pseudo-random binary sequence (PRBS) data is used to evaluate the ANN-based nonlinear equalizers, the ANNs can potentially learn the repeated PRBS patterns, resulting in overestimation of the equalization performance. In this paper, we clarify that the overfitting of the ANNs hardly occurs or requires much more neuron units than used for common nonlinear equalization, if we employ multi-level modulation signals including simple 4-ary pulse-amplitude modulation (PAM4). In our study, we compared binary signal and 4-level signal generated by PRBS data. White Gaussian noise (WGN) was added to the signals. The ANN-based nonlinear equalizers were trained using a least mean squares (LMS) algorithm, to attempt to “compensate” for the noise. The signal quality was evaluated using the error vector magnitude (EVM). When the overfitting occurs, the EVM is improved by the nonlinear equalization. The results revealed that, in the case of 4-level signal, the influence of the overfitting was suppressed in comparison with the case of the binary signal.

Paper Details

Date Published: 24 February 2020
PDF: 5 pages
Proc. SPIE 11299, AI and Optical Data Sciences, 1129916 (24 February 2020); doi: 10.1117/12.2545803
Show Author Affiliations
Kai Ikuta, Meiji Univ. (Japan)
Yuta Otsuka, Meiji Univ. (Japan)
Moriya Nakamura, Meiji Univ. (Japan)

Published in SPIE Proceedings Vol. 11299:
AI and Optical Data Sciences
Bahram Jalali; Ken-ichi Kitayama, Editor(s)

© SPIE. Terms of Use
Back to Top
Sign in to read the full article
Create a free SPIE account to get access to
premium articles and original research
Forgot your username?