Bad point cloud accuracy when on roof ridges

I have noticed when processing a set of photos at a lower than normal flight height with 90% forward and 90% side overlap at ground level, that the initial point cloud points created at the top of roof ridge and gutters are quite often wrong by a constant. I believe the issue is mainly due to less overlap being achieved in the areas when the houses are present due to them being several meters off the ground. The problem I have is that the GCP always show good residuals after the calibration step even though the ridge heights will be off by a constant (such as 0.1m). Reoptimizing the calibration step with additional matches seems to greatly help reduce the error constant in the ridge height and gutter heights. Besides flying higher and increasing the GSD and thus the Z error to ensure plenty of overlap, would there be another method to ensure that I generate an accurate point cloud at these heights? Would adding in a manual tie point help? should I calibrate with more points instead having the ‘key points’ set on automatic?