Optimal linear receiving filter for digital transmission over nonlinear channels

The optimum linear receiving filter for digital transmission over a nonlinear channel is specified. Under the assumption that the channel has a finite memory and that the noise is additive Gaussian, an optimum receiver is sought in the form of a linear receiving filter followed by a symbol-rate sampler and a memoryless decision device. The receiving filter is chosen so as to minimize the mean-square error between the input to the decision device and the transmitted symbol. It is shown that the structure of the optimum receiving filter corresponds to a bank of matched filters, each followed by a linear transversal filter. The number of matched filters is equal to the number of linearly independent waveforms that can be observed at the channel output in a symbol period. This result is a generalization of a fact which is well known for linearly modulated signals transmitted over linear channels.