Abstract: In order to solve the problem that the gradient descent algorithm has a slow convergence speed, uses all the sample points each time to update the gradient, and has a large amount of calculation and high complexity, a novel dispersion compensated FIR (fine length impulse response) filter for coherent optical receiver is proposed. Through building coherent optical communication system baseband multi-rate transmission model in Matlab, using the Mini-Batch Gradient Descent method with 1% of the total samples to update the weights of filter tap h(n) every time calculating the mean square error and updating the filter gradient, the computational complexity can be reduced to 1% of the original. When the learning efficiency is 0.05, the convergence value reaches 0.005 5, which has a smaller mean square error convergence value. When the mean square error converges, the weight of the filter tap is obtained. The filter weights obtained by this method has a better filtering effect and lower bit error in the whole frequency band. After using baseband pulse shaping to limit the effective bandwidth of the baseband signal, the number of filter taps is 197, which is 37.5% less than the number of taps in the full frequency band. Therefore, an optimal filter with fewer taps and better filtering effects can be designed in a narrow frequency band.

Key words: coherent optical communication system model, mean square error, FIR filter, tap weights, mini-batch gradient descent, optimal filter