I think the problem is in the interpolation somewhere.
I bought some more IEEE articles but they are beyond my understanding. That's what comes of not being practicing EE for 15 years.
I thought of a way to "sanity check" the interpolation. By wiring comparators to the A and B analog inputs, I can now get A and B digital signals. With quadrature decoding this is a ~ 65" accuracy every time one of these inputs change.
So whenever there is a rising or falling edge on the A and B, I know that the interpolated angle must be 0, 90, 180, or 270 degrees (for sure! to within the limit of the encoder) and I can use that information to check the interpolation algorithm.
Later this weekend.. I have to finish my SAO exam..