Bluntly put, they are biased.
Tests of this sort would be unpublishable in any reputable scientific or
I have followed this thread with much interest, and I think there is some confusion about the kind of “report” that is being provided here. The above comment illustrates this. The reality is that reports such as these ARE frequently published.
If I may provide an analogy to my own professional work: In medicine we distinguish between formal scientific studies and so-called “case reports”. The former are attempts to establish a firm - and generalizable – conclusion, and are characterized by highly formalized methodology and analysis (i.e. design, blinding, controls, statistics etc). They are therefore subject to similarly formal – and often harsh - criticism.
Case reports, however, do not attempt to draw any general conclusion. They are instead isolated, honest, empirical observations of a particular clinical outcome. Data points, if you will. So, for example, if in switching from drug X to Drug Y, a particular outcome is noted by a physician - that might be published as a case report. The physician making the report, however, makes no claim that in future situations Drug X and Y will behave in similar ways. Consequently, the kinds of formal criticism that would be appropriate for a controlled study would be entirely misguided and irrelevant if leveled against a case report.
The utility of such case reporting is only realized over time. So, if dozens or hundreds, of Drug X & Y reports consistently report similar outcomes, then a later investigator might review the literature, collectively examine these case reports, and now see if some general conclusion can be reached about drug X and Y. Perhaps she/he might choose to do a formal study.
The kind of observing report we have here is much like a case report. Consequently, the most important issue is the sincerity/honesty of those reporting – which is clearly not in question. And so, if the project continues and we accumulate more and more mirror X vs mirror Y repots, the collective of these may suggest a general conclusion, even if no one of them could in isolation. And isn’t this really the way much (but not all) of our “held” amateur astronomy knowledge has come to be - as in “spiral structure typically requires dark skies and large aperture” - by way of a preponderance of isolated, informal observations.
Just my 2 cents