It just doesn't look self-consistent to me. That second screenshot says RA error is 0.39" but looking at the graph the RA is consistently much higher than that and obviously averaging higher than 1.0", it appears the average error is closer to 1.5", definitely not 0.39". It doesn't look like the RA line is even ever as low as 0.39" much less averaging that. The DEC error looks like it matches the graph but RA doesn't match at all.
You seem to have misunderstood the graph. The graph looks at the movement of the star in the RA axis for a certain period of time. Once the right end of the graph has been reached, you have an average RMS error of that time period.
Now the important thing to note here, is that the 0 indicator (initial star center coordinate) is actually irrelevant for the RA performance.
So, whether the average error is 0.39" at center position or 2 pixels away from center position is irrelevant. The stars will be round. What changed is the entire field of view.
Anyway, I've had that issue before and usually it came down to software issues. One version of PHD2 worked nicely for me, the next one was utter chaos.
As you mentioned, gear mesh could be another issue. Lastly, I'd check balance. Try perfect balance, try East-heavy balance. Make sure you're not getting flexure. Re-tighten every screw on your setup. It's the small things...