A method is proposed to separately model the mechanical dot gain and the optical dot gain. First, an iterative algorithm is proposed to estimate the spatio-spectral transmittance of ink layer from the spatio-spectral reflectance of color halftone print measured with the reflection optical microscope attached with the liquid crystal tunable filter (LCTF). The spatio-spectral transmittance of ink layer is not affected by the optical dot gain and is only affected by the mechanical dot gain. Next, a model is proposed to estimate the effective dot coverage using the estimated spatio-spectral transmittance of ink layer. It corresponds to the analysis of mechanical dot gain. Next, a model is proposed to estimate Yule-Nielsen's n parameter using the effective dot coverage. It corresponds to the analysis of optical dot gain. Finally, the prediction accuracy of the proposed model is evaluated by the delta E94 between the measured and predicted spectral reflectance of offset printing images with cyan and magenta inks. The prediction accuracy of the proposed model was significant since the average delta E94 and the maximum delta E94 of all samples between the measured spectral reflectance and the predicted spectral reflectance were 0.62 and 1.37, respectively.