Millimeter-wave (mmWave) beamspace massive multiple-input multiple-output (mMIMO) system with lens antenna array can minimize transceiver hardware complexity without compromising performance. However, the number of supported portable user terminals (PUTs) cannot exceed the number of radio frequency (RF) blocks accessible at the same time, frequency, and coding resources. As a result, we propose the integration of rate-splitting multiple access (RSMA) into a beamspace mMIMO system to support a larger number of PUTs than the number of available RF blocks while minimizing intra-beam interferences. Moreover, orthogonal random precoding (ORP) is utilized in the downlink of the beamspace mMIMO system to mitigate the inter-beam interferences and extend the cell coverage area. Then, we develop an optimization problem to optimize the system’s overall throughput while keeping the minimum needed throughput and power budget in consideration. The nonconvex optimization issue is then turned into a convex optimization problem using the successive convex approximation approach. Following that, we offer an alternating method to solve the approximate optimization issue and select an optimal solution. Furthermore, the suggested method’s effectiveness is evaluated in terms of total throughput, energy efficiency, and cell coverage area. Finally, numerical results confirm the superior performance of the proposed method over benchmark techniques in terms of sum throughput, energy efficiency, and cell coverage area.