The calculations of atmospheric radiative transfer are among the most time-consuming components of the numerical weather prediction (NWP) models. Therefore, using deep learning to achieve fast radiative transfer has become a popular research direction. We propose a physics-incorporated framework for the radiative transfer model training, in which the thermal relationship between fluxes and heating rates is encoded as a layer of the network so that the energy conservation can be satisfied. Based on this framework, we compared various types of neural networks and found that the model structures with global receptive fields are more suitable for the radiative transfer problem, among which the Bi-LSTM model has the best performance.