Hope you find the issue amongst all the suggestions.
If you want to eliminate your polar scope and polar alignment to help narrow it down to one of the other possible issues, then once you've done your PA next time ... just "drift" with a star and see if it wanders over ten minutes or so. Or you can google and read up the method I use called DARV. I'm not saying to use it to actually polar align in your case. Just use it after doing your normal Polaris polar align to check that you really are spot on with your alignment. The image below is one of two done (one pointing low in the East and one pointing at 0 degree DEC up at the Meridian). Only takes 80 seconds to check each adjustment - AZ and ALT.
To do this on a star adventurer set 80 second exposure. For the first 10 seconds have the tracking rate button on normal tracking rate to "burn" a bright starting-point "dot". Then turn tracking off (one step turn clockwise of the tracking knob) and let the stars drift for 30 seconds. Then turn the tracking rate to 2x rate for 40 seconds. This will catch up and "drive-past" the stars a bit to enable you to judge better if you really are spot-on with your PA.
Photo below shows the end result when polar aligned (just remember you have to do it twice ... once for ALT and once for AZ). The image shows the star trails directly overlying each other almost perfectly intersecting the initial bright-starting-point "dot" ... so I am polar aligned in ALT in this case. If I wasn't then the two lines would form a "V" with the line no longer passing through the initial bright-starting-point "dot".
If you find you are indeed perfectly polar aligned from using your polar scope then you know it's something else going on.