I've tried masking my 18" Obsession to 4" off-axis a few times, but my Stellarvue 4" Apo does a better job. Might be the mirror was not cooled down all the way, or collimation very slightly off, but I think Apo's just rule at small apertures.
When I first got my 18" I made a 6.5" off-axis aperture mask to look at Jupiter. It looked really nice, then I used it at full aperture and it looked amazing.
In a reflector there are reflective losses and scatter. And while at first glance, it seems like the OA Newtonian would have all the virtues of an F/11 or slower scope, the mirror is an off-axis section of a very fast mirror and so the off-axis aberrations are those of a much faster scope. Not quite as fast as the original mirror but close. It is not true coma.
The real issue I see is that the big mirror, if of decent quality, without the mask has far better planetary contrast and resolution than it has with the mask. The diffraction effects of the secondary are tiny in comparison to the diffraction effects of the aperture mask(and the refractor's aperture.) And too, most large aperture scopes have COs that are quite small and have essentially no effect on contrast, they are essentially unobstructed. My 25 inch F/5 has a 14% CO, that is less than 2% of the area.
I have a nice TeleVue 4 inch refractor. I also have a 13.1 inch F/5.5 Starsplitter with a Royce mirror. It has a 20% CO so I could mask it down to 4 inches.
But the question is why would I want to do this? The 13.1 inch F/5.5 is a far better planetary scope than the refractor, better fine scale contrast, resolution, brightness.. Masked or unmasked, the set up effort is the same, the thermal issues are similar though some tube currents may be avoided by proper orientation of the mask.
As a planetary scope, the virtues of an apo are quick setup, thermal stability, high transmission, lack of scatter. They do have higher fine scale contrast when compared to a scope of equal aperture but when compared to a scope that is 2x to 3x the aperture, particularly a scope with a small CO, the the APO is at a significant disadvantage in terms of contrast transfer and resolution..
I think if one does not have an APO for comparison purposes and one wants to see what all the hoppla is about it makes sense. Or if the seeing is substandard so that the smaller scope will perform better. In my experience comparing 4 and 5 inch APO/EDs with large Dobs, if the seeing limits the large scope, it will limit the small scope, neither will provide decent planetary performance.