Urban Land Surface Models (ULSMs) simulate energy and water exchanges between the urban surface and atmosphere. When part of numerical weather prediction, ULSMs provide a lower boundary for the atmosphere and improve the applicability of model results in the urban environment compared with non-urban land surface models. However, earlier systematic ULSM comparison projects assessed the energy balance but ignored the water balance which is coupled to the energy balance. Here, we analyze the water balance representation in 19 ULSMs participating in the Urban-PLUMBER project using results for 20 sites spread across a range of climates and urban form characteristics. As observations for most water fluxes are unavailable, we examine the water balance closure, flux timing, and magnitude with a score derived from seven indicators. We find that the water budget is only closed in 57% of the model-site combinations assuming closure when annual total incoming fluxes (precipitation and irrigation) fluxes are within 3% of the outgoing (all other) fluxes. Results show the timing is better captured than magnitude. No ULSM has passed all good water balance indicators for any site. Our results indicate models could be improved by explicitly verifying water balance closure and revising runoff parameterizations. By expanding ULSM evaluation to the water balance and related to latent heat flux performance, we demonstrate the benefits of evaluating processes with direct feedback mechanisms to the processes of interest.