one thing I also noticed with my CGEM (and AP600!) is that they are very sensitive to balance. You must be slightly east-heavy, otherwise the guiding errors double. But you already knew that.
In many areas sub 0.5" RMS is not even possible due to the atmosphere, so I guess I got lucky with my 0.3" (which I don't get all the time, BTW). One of the other posters here (andysea) told me he gets about 0.5" with both his Mach1 and Takahashi NJP, hence my estimate of 1" being reasonable for a CGEM - this is just my estimate, not something set in stone. Ed did tell me that the 1.7" figure I was getting with my CGEM was already decent.
One more thing... the sharp jaggies on the guiding graph for the CGEM are due to the motor gearbox, i.e. nothing you can do about it.
Roland has stated that AP uses custom gearboxes (either made-to-order or in-house fabricated - I don't know..) to get around this issue. The spur gears have very fine teeth, which makes them very noisy.
That said, my AP600 is using Vexta spur gearhead steppers (I retrofitted a GoTo to it). These motors are $220 each retail (I paid much less off ebay) and are the exact same motors used on the Takahashi NJP Temma. They are far smoother (less jaggies) than the CGEM Igarashi motors (as should be expected from a motor that costs $220) but they still have more jaggies than the Mach1 motors. Maybe there's something to be said for AP's approach..
Bottom line.. if you're already getting 1" RMS - it's not going to get much better. You are limited by the gearbox now, not the worm and ring gear.
One avenue that I have not explored.. if the final transfer gears between the motor gearbox and the worm were replaced with a belt drive, it might insulate the worm from some of the sharper jaggies.
I know a good number of EQ6 / Atlas users have done this (particularly in the UK, where they even have a kit for it), and the iEQ45 has a belt drive out of the box.
Note - there is one guy here on the CN forum who reported 0.5" RMS guiding with an Atlas (!!!) which, if true, is phenomenal, that being Mach1 territory and all. But.. it also is believable, because the Atlas has a very simple reduction gearbox compared to the CGEM (it has 2 gears and an idler). Less gears in the gearbox equals less jaggies on the guiding graph.