Abstract: A solution based on the orthogonal subspace to the problem of source depth discrimination in a downward refracting shallow water waveguide is presented for the case of a limited vertical line array. The trapped subspace is preprocessed by convex optimization to remove the interference terms and improve the estimation accuracy of the subspace energy. The Schmidt orthogonalization is applied to construct orthogonal subspaces of the trapped and free components, preserving the complete mode space to increase the mode information involved in the source depth discrimination. The performance is predicted with simulations, allowing one to compare the proposed and reference methods. The effects of signal-to-noise ratio and the vertical aperture on the performance are numerically investigated, and the results show that the proposed method is more robust than the matched subspace discriminator and the regular least squares-based mode filter, which are used as reference methods, for a limited-vertical aperture case. Finally, the approach is validated on experimental data collected with a vertical line array deployed in a shallow sea experiment. The results show that with signal-to-noise ratio ranging from 7 dB to 16 dB, the proposed method can still discriminate the surface and submerged sources at an array aperture of 0.16 times the water depth with higher correct discrimination rates than the reference methods.