Abstract: In scenarios where the underwater acoustic channel structure is unknown, a novel iterative reception scheme, referred to as pattern-coupled expectation-maximization vector approximate message passing in the frequency domain, is proposed to eliminate inter-symbol interference in received signals. The pattern-coupled prior assumption facilitates the automatic identification of single-cluster or multi-cluster underwater acoustic channel tap coefficients. The expectation-maximization algorithm updates prior information that controls the sparse structure of the channel and the noise variance. Linear estimation of the received signal is performed based on the estimates of the channel and noise, while nonlinear estimation is conducted using prior information from the decoder. External information exchange is facilitated between the two estimation modes to exploit the constrained gains of the symbols to be estimated. The simulation and sea trial results confirm the advantages of the proposed scheme. In a non-sparse and time-varying deep-sea vertical communication experiment, the proposed method, combined with Doppler estimation and compensation, enabled high-order modulation underwater communication at a transmission distance of 3832 m with a rate of 27.32 kbit/s. In a cluster-sparse and low signal-to-noise ratio shallow-water seabed cross-medium communication experiment, the proposed method automatically identified various channel cluster-sparse structures, enabling coherent underwater communication at a transmission distance of 4505 m with a rate of 3.84 kbit/s.