First off, there is no such thing as a perfect polar alignment. By that, I mean that there is no possible alignment of the RA axis that results in zero drift in both axes at all points in the sky. Any polar alignment is a compromise. A "traditional" drift alignment, for example, will counter declination drift across a large area of sky, but does not eliminate RA drift (this is because older mounts were driven in RA, but not necessarily dec, so RA drift could be corrected by altering the tracking speed).
That said, your question is interesting, and it should be possible to solve it within your given assumption of a "perfect" polar alignment (even though such a thing doesn't actually exist). I don't know the math off the top of my head, but if you use just about any planetarium software, it's easy to switch between JNow and J2000 coordinates to see the effect of 24 years of precession. It's surprising how much movement happens, so our hypothetical polar alignment would definitely need to be updated on a time scale probably in single digit years.
As for continental drift, I would offer this as a reality check: The Earth rotates at just over 1,000mph at the equator. Also, it moves in its orbit at more than 65,000mph. The entire solar system is moving through space at nearly 450,000mph. So I doubt that the effect of continental drift could be measured by any means.
Now continental drift would potentially change the longitude and latitude of the site. But it would take centuries to result in the equivalent change you'd get by setting up on the opposite side of a field. Again, any effect from continental drift would be unmeasurable.
Using Polaris as an example, the parallax motion is around 10 milli-arc-seconds. So that is the effect of moving the observer about 185 million miles (twice the Earth-Sun distance). Again, not something you could measure with amateur gear (at least not easily).
So the question is interesting academically, but has no practical application to amateur astronomers.