Return to the Cloudy Nights Telescope Reviews home page


Observing >> Double Star Observing

Pages: 1
WRAK
scholastic sledgehammer


Reged: 02/18/12

Field Test for WRAK's RoT new
      #6294462 - 01/05/14 05:41 PM Attachment (30 downloads)

After eliminating all observations used earlier for the statistical analysis for my current RoT algorithm to avoid any bias I applied the model without the final TML-check (as this was developped later on) to this reduced data set of still ~385 limit observations to make a realistic field test.
I am rather surprised that the result is quite very good compared with the other field tests for other RoTs so far as the correlation coefficient is still above 0.9 and the standard deviation for the difference between used and proposed aperture is still below 20mm. I would call this a positive field test result.

I sorted the RoT results by the difference between used and proposed aperture and found the following relations:
- Most large negative errors (used aperture far smaller than RoT pA) are the result of overperforming observations with small separation in combination with large delta_m (and interestingly fixed aperture) - this is a king of "error" you would not want to eliminate. The single digit precision for the separations and some errors in the advertised magnitudes may also play a role here
- Most large positive errors (used aperture far larger than RoT pA) occur with larger CO values and with very faint secondaries - this is the field where I knew already before this test that I have room for improvement (using the TML check module might have helped already a bit).

This result motivates me to continue my RoT efforts.
Wilfried
PS: If anybody has access to another data set suitable for a RoT field test I would be interested to repeat this test with data of other origin


Post Extras: Print Post   Remind Me!   Notify Moderator  
Asbytec
Guy in a furry hat
*****

Reged: 08/08/07

Loc: La Union, PI
Re: Field Test for WRAK's RoT new [Re: WRAK]
      #6295073 - 01/05/14 11:26 PM

Nice, Wilfried. Look forward to your results.

Post Extras: Print Post   Remind Me!   Notify Moderator  
drollere
Pooh-Bah
*****

Reged: 02/02/10

Loc: sebastopol, california
Re: Field Test for WRAK's RoT new [Re: Asbytec]
      #6298231 - 01/07/14 02:26 PM

again: the correlation coefficient is a misleading statistic in this situation.

a correlation is calculated on data that have been normalized into z scores -- the averages of the two variables are identical, and their standard deviations are identical. consequently, both the difference in the means (systematic over or under prediction) and the difference in the standard deviations are removed.

take the apertures, add one meter to each value. no one would consider a "prediction" that is off by 1 meter to be useful. now correlate the two. it's 1.0.

i've looked over your data and the results are indeed promising, although there are several odd things going on and some suggestions might be helpful.

the average prediction shows no bias (average signed error is zero), but the average prediction is off by about 20% of aperture, or roughly 8.5" to 12" for a 10" aperture. this is the most direct measure of the prediction accuracy.

if you correlate each of your individual variables with the predicted aperture, you'll find most of the prediction is coming from aperture, NEML, CO and m1'', or m1 coded as 0/1. Dmod (separation plus CO) is the strongest correlate after aperture, but it may just be removing a curvilinear relation on CO or outlier values on separation.

to underscore that point, if you code the over and under predicted values in a treanor plot, there is no clear relationship between the prediction error and the star (stimulus) parameters (rez ratio and mag. diff.), except that every system with a resolution ratio less than 1.0 is overpredicted. i suspect that is because with the variable aperture technique you are measuring the aperture necessary to split a star, while the contributed observations from norme, fred and others show the aperture that is *sufficient* to split a star, and that difference in stimulus presentation will insert significant difficulties into the analysis. for example, a plot of the prediction errors on the original aperture shows a very large spread of values on the 140mm aperture, +60mm to -80mm, or roughly 50% of aperture. this is probably a better picture of the true spread of the model in a wide sample of observers and instruments.

there is another problem. the fact that your variable aperture method underpredicted your visual acuity in comparison to the taylor mask may be related to the fact that your variable aperture method overpredicts the aperture necessary to split close double stars. (i note that the error in the prediction of aperture, 20%, is the same as the difference in the prediction of your acuity.) that would be especially true if you and fred are both using an unmasked aperture at 140mm, and the variance there shows what happens when you don't use the method.

there is a common distinction made in perceptual research between a functional analysis and curve fitting. a functional analysis attempts to concisely reproduce in mathematics the same dynamics that operate in the causes. curve fitting uses as many variables as necessary to reach a certain level of fit. i can't pretend to understand your RoT algorithm, because it includes ratios taken on exponents that are themselves ratios or variables that have been recoded and truncated. but it does seem that you have embarked on curve fitting without keeping in view a theory or an explanation of what is going on in the observer and the instrument.

aside from clarity, the worse problem is that curve fitting typically collapses when applied to a new sample, in this case other observers using other instruments. specifically, most of the prediction seems based on your observations, which eliminates the individual differences that are the single greatest source of error in perceptual data of any kind.

i don't intend to carp, since i know this project is important to you and you have spent enormous time and thought on it. but i'd suggest this is a good point to consider what you have achieved in terms of what you intend for the product. if this is only to be something that predicts what you observe by yourself with a variable aperture diaphragm, nothing more is necessary.

if you intend to produce an algorithm that others can use, then i'd suggest a few points to consider.

the difference in error between the reports at aperture 140 and at smaller apertures (the variable aperture method?) should be explored. is your method biasing your results by misrepresenting the resolution required to split a star?

second, whatever method you use, i think you really stack the deck against yourself by relying on your own observations and using a method not used by other observers. you have to get as much error out of the data as possible, and this means some different eyes and all eyes see things exactly the same way.

third, even if you rely on your own observations and variable aperture method, it's critical to use a wider range of instruments. variable aperture cannot change objective magnification or native focal ratio; an /10 140mm refractor stopped down to 100mm is not the same as an /10 100mm refractor.

finally, your research has taught you a lot about what matters more or less in visual resolution. with that knowledge in hand, i strongly urge you to try to simplify your formula. it is so complicated that i doubt even you can explain what it does (certainly i can't), and it is too complicated to perform well with new observers, new instruments, and new telescopes.

good luck.


Post Extras: Print Post   Remind Me!   Notify Moderator  
WRAK
scholastic sledgehammer


Reged: 02/18/12

Re: Field Test for WRAK's RoT new [Re: drollere]
      #6299582 - 01/08/14 09:14 AM Attachment (7 downloads)

Bruce, thanks for taking the time to have a look at the details - will certainly consider your suggestions for my next step. And yes, the model is rather complex and based on 3 aspects:
- Basic optical theory how things should work with concentration on the factors that seem most important as in total there are simply too many factors to consider
- Parameter values based on nonlinear statistical analysis
- Adaptions in the model when comparing results with data set and expected behavior showed oddities.

As first step I added a column to the field test spreadsheet with error in % of prediction - got here a standard deviation of 24.4%, not this good. Had then a look at the observations with the largest error in % and found two observations with CO 0.4 and 0.5 I should have eliminated as this is not the range the model should work and another odd observation with an 178" separation and this is obviously no longer a topic of double star resolution but of telescope magnitude limit. So I have to look also on my data set for odd entries.
Wilfried


Post Extras: Print Post   Remind Me!   Notify Moderator  
WRAK
scholastic sledgehammer


Reged: 02/18/12

Re: Field Test for WRAK's RoT new [Re: drollere]
      #6300076 - 01/08/14 01:34 PM

This are so many arguments that I have to go step for step:

Quote:

again: the correlation coefficient is a misleading statistic in this situation.

a correlation is calculated on data that have been normalized into z scores -- the averages of the two variables are identical, and their standard deviations are identical. consequently, both the difference in the means (systematic over or under prediction) and the difference in the standard deviations are removed.

take the apertures, add one meter to each value. no one would consider a "prediction" that is off by 1 meter to be useful. now correlate the two. it's 1.0...




I consider the correlation coefficient between predicted and used aperture as highly relevant as this two data columns do not have any numerical relationship - so the meaning of your example is somewhat unclear to me

Quote:

if you correlate each of your individual variables with the predicted aperture, you'll find most of the prediction is coming from aperture, NEML, CO and m1'', or m1 coded as 0/1. Dmod (separation plus CO) is the strongest correlate after aperture, but it may just be removing a curvilinear relation on CO or outlier values on separation.




This seems to be a misunderstanding - aperture is not an explaining part of my RoT model but the target. Main component for correlation with the predicted aperture is Dmod = proposed aperture according to Dawes 116/sep modified by the factor CO (reduced size of Airy disk) as this is the base for equal binaries, all other factors like delta_m etc. do not have any relevant single correlation but work only in context with the other parameters.

To be continued.
Wilfried


Post Extras: Print Post   Remind Me!   Notify Moderator  
WRAK
scholastic sledgehammer


Reged: 02/18/12

Re: Field Test for WRAK's RoT new [Re: drollere]
      #6301650 - 01/09/14 09:49 AM

Quote:

... if you code the over and under predicted values in a treanor plot, there is no clear relationship between the prediction error and the star (stimulus) parameters (rez ratio and mag. diff.), except that every system with a resolution ratio less than 1.0 is overpredicted. i suspect that is because with the variable aperture technique you are measuring the aperture necessary to split a star, while the contributed observations from norme, fred and others show the aperture that is *sufficient* to split a star, and that difference in stimulus presentation will insert significant difficulties into the analysis. for example, a plot of the prediction errors on the original aperture shows a very large spread of values on the 140mm aperture, +60mm to -80mm, or roughly 50% of aperture. this is probably a better picture of the true spread of the model in a wide sample of observers and instruments. ...




This is an excellent point - there exists clearly a systematic overestimating of the difficulty of resolution of rather close doubles with large delta_m (some reports with delta_m of ~3 are near the Dawes criterion) and underestimating of the difficulty of the resolution of wide pairs with very faint secondaries. And curiously this obvious bias is mostly expressed with the 140mm aperture - one reason might be that this aperture is with about 25% of the observations rather prominent in my data set and the other might be that this is a fixed aperture and observations with fixed apertures have a tendency for a wide spread due to both over- and underperforming.

While we have to live with the fact that resolution of double stars is to some degree a random process due to the many factors involved and thus a RoT will always have to cover some range around an average value we should certainly try to keep the range as small as possible.
So maybe Sparrow might be in average a better base value for a RoT than Dawes and regarding wide faint doubles I obviously have to do some more homework especially in the transition area from resolving a double to resolve a faint single star.
Wilfried


Post Extras: Print Post   Remind Me!   Notify Moderator  
fred1871
professor emeritus


Reged: 03/22/09

Loc: Australia
Re: Field Test for WRAK's RoT new [Re: WRAK]
      #6302953 - 01/09/14 08:33 PM

Quote:

Quote:

... if you code the over and under predicted values in a treanor plot, there is no clear relationship between the prediction error and the star (stimulus) parameters (rez ratio and mag. diff.), except that every system with a resolution ratio less than 1.0 is overpredicted. i suspect that is because with the variable aperture technique you are measuring the aperture necessary to split a star, while the contributed observations from norme, fred and others show the aperture that is *sufficient* to split a star, and that difference in stimulus presentation will insert significant difficulties into the analysis. for example, a plot of the prediction errors on the original aperture shows a very large spread of values on the 140mm aperture, +60mm to -80mm, or roughly 50% of aperture. this is probably a better picture of the true spread of the model in a wide sample of observers and instruments. ...




This is an excellent point - there exists clearly a systematic overestimating of the difficulty of resolution of rather close doubles with large delta_m (some reports with delta_m of ~3 are near the Dawes criterion) and underestimating of the difficulty of the resolution of wide pairs with very faint secondaries. And curiously this obvious bias is mostly expressed with the 140mm aperture - one reason might be that this aperture is with about 25% of the observations rather prominent in my data set and the other might be that this is a fixed aperture and observations with fixed apertures have a tendency for a wide spread due to both over- and underperforming.

While we have to live with the fact that resolution of double stars is to some degree a random process due to the many factors involved and thus a RoT will always have to cover some range around an average value we should certainly try to keep the range as small as possible.
So maybe Sparrow might be in average a better base value for a RoT than Dawes and regarding wide faint doubles I obviously have to do some more homework especially in the transition area from resolving a double to resolve a faint single star.
Wilfried




A few thoughts here. First, delta-m pairs ~3, "near the Dawes criterion", are not too surprising if what is meant is secondary stars falling into the Rayleigh gap - the first dark interspace between disc and 1st bright ring. This seems to me a very different case from "wide pairs with very faint secondaries".

Can the same RoT apply to both? In part, it depends on defining "wide", as well as "faint". If Sirius is an example, or Rigel, I think we have a different issue from the unequal and very close pairs. If we're considering wider pairs than Sirius and Rigel, with dimmer companions, I suspect the issue is one of perception of faint stars as faint stars, subject to glare/flare issues from a primary star. How much scattered light is in a field will depend on the optics, if we assume no moonlight and no significantly high light pollution.

As I've remarked before, even a change of eyepiece can have an effect here. To take an obvious (non-double) example, Saturn's moons - with my 140mm refractor, I can see various of these more easily at 160x with a 5mm UO HD Orthoscopic eyepiece than with a 5mm Nagler T6. The latter produces more scattered light. The same effect appears with these eyepieces in observing Sirius, or Rigel. In the case of Saturn's moons, it can make the difference between visibility and invisibility when a moon is dim.

For doubles, dim companions at large delta-m might follow the expected RoT limits if we are dealing with delta-m of say 4 or 5, at separations moderately close, and the secondary star being not too close to the "separate star limit" of the telescope for that observer. Here I think Peterson gives a hint - as stars get to a certain level of dimness, relative to his telescope/eye/magnification limit, the separation needs to be greater. I would expect a similar effect when using a more helpful magnification than Peterson's; his was too low to use the full capacity of his telescope, or of his eye.

This latter study, moderately separated pairs of larger delta-m and the secondaries fairly dim, would need an observing list tuned to the apertures to be used. Obviously the use of diaphragms would make choices easier; pairs too easy for 140mm could be quite suitable for 100mm or 75mm.

When enough data of this kind accumulates, an Rot derived from it could be compared with an RoT derived from somewhat brighter pairs where delta-m is no more than about 4, and the secondary star no fainter than around 10-10.5.

Another list of pairs, close, examples equal and unequal, but of lesser magnitude - Lewis's faint pairs - could be done. Two strands here, as per Lewis - near equal, and significantly unequal. We might find a continuum of some pattern here; intuitively that suggests itself. The study would show whether that holds. Counter-intuitive results in science are not rare.

Stars for this might begin with primary magnitudes in the 8-11 range; for the secondaries, when unequal, delta-m of 2-4. Plainly, 11+15 is not practical; but 8+12 or 9+12 is, or 9+12, or 10+13, and so forth, given middle-size scopes (say 20-30cm). So the lists need to be tuned to practicality (minimum secondary perhaps around mag 13). In the case of equal pairs I'd suggest a limit around mag 11; just below the Couteau limit, plus the sheer visual acuity problem of close pairs even at mag 11. Pairs that dim are not easy, if close, with 235mm; even with a C14 that I used years ago they were tricky, being more seeing-dependent and still not bright, which made small separations tougher, even with 35cm aperture.


Post Extras: Print Post   Remind Me!   Notify Moderator  
WRAK
scholastic sledgehammer


Reged: 02/18/12

Re: Field Test for WRAK's RoT new [Re: fred1871]
      #6303477 - 01/10/14 04:01 AM

Quote:

... Can the same RoT apply to both? ...
... even a change of eyepiece can have an effect here. ...
... observing list tuned to the apertures to be used...




Fred, you point out here several important aspects I think:
- One RoT for all? This is the reason why my model got quite complex over time. Different aspects have to be covered with different modules and switches are used to enable/disable these modules
- Different eyepieces: One of the many influencing factors making it impractical to cover all. So we have to accept the fact that there will always be a range instead of a single data point
- Observing lists: Should be designed to cover as many different parameter values as possible. Statistical analysis can only be as good as the data set allows. My data set used for my current algorithm had only a few wide faint pairs included - so it can not be expected to work well in this range. Same with apertures larger than 150mm
- The idea to make observation lists specifically designed for specific apertures is of interest for two purposes: Testing an existing RoT and getting data points for assumed so far not this well covered areas.
Wilfried


Post Extras: Print Post   Remind Me!   Notify Moderator  
WRAK
scholastic sledgehammer


Reged: 02/18/12

Re: Field Test for WRAK's RoT [Re: drollere]
      #6303957 - 01/10/14 10:58 AM Attachment (6 downloads)

Quote:

... there is another problem. the fact that your variable aperture method underpredicted your visual acuity in comparison to the taylor mask may be related to the fact that your variable aperture method overpredicts the aperture necessary to split close double stars. (i note that the error in the prediction of aperture, 20%, is the same as the difference in the prediction of your acuity.) that would be especially true if you and fred are both using an unmasked aperture at 140mm, and the variance there shows what happens when you don't use the method.

there is a common distinction made in perceptual research between a functional analysis and curve fitting. a functional analysis attempts to concisely reproduce in mathematics the same dynamics that operate in the causes. curve fitting uses as many variables as necessary to reach a certain level of fit. i can't pretend to understand your RoT algorithm, because it includes ratios taken on exponents that are themselves ratios or variables that have been recoded and truncated. but it does seem that you have embarked on curve fitting without keeping in view a theory or an explanation of what is going on in the observer and the instrument.

aside from clarity, the worse problem is that curve fitting typically collapses when applied to a new sample, in this case other observers using other instruments. specifically, most of the prediction seems based on your observations, which eliminates the individual differences that are the single greatest source of error in perceptual data of any kind. ...




The assumption that I did not try to be over eager when reducing the aperture to get to the limit aperture for resolving a given double for whatever reasons seems correct. After eliminating all fixed aperture observations from the data set the picture changes as there is no longer a bias on overperforming with close doubles but what remains are the problems with unerestimating the difficulty of resolving wide faint doubles - so the relation between used apertures and proposed apertures shows some spread (see attached graph).
This does not get much better when repeating the nonlinear regression analysis with the given model structure - there is some better adapting to the data and the spread gets smaller but is still significant.
This may mean that some of the observations are odd for whatever reason (bad seeing, wrong advertised data, typos ...) - to some degree this might be true but I think it more realistic that the structure of the model is no longer up to explain the observations in full extend.

And yes, you are right that I did some curve fitting but only in addition to basic functional considerations. And I think this is necessary to cover all parameters not explicitly part of the model and I think also that with a data set large enough there should also be no real problem with new data. Already the current algorithm shows despite several shortcomings graceful behavior in this regard as this field test resulted in a far better result than expected by me - good reason to proceed further.
Wilfried


Post Extras: Print Post   Remind Me!   Notify Moderator  
azure1961p
Postmaster
*****

Reged: 01/17/09

Loc: USA
Re: Field Test for WRAK's RoT new [Re: WRAK]
      #6305267 - 01/11/14 12:02 AM

Nice to see you making your way along here Wil. Nice points Fred.

Pete


Post Extras: Print Post   Remind Me!   Notify Moderator  
WRAK
scholastic sledgehammer


Reged: 02/18/12

Re: Field Test for WRAK's RoT new [Re: drollere]
      #6305625 - 01/11/14 08:15 AM Attachment (4 downloads)

Quote:

... if you intend to produce an algorithm that others can use, then i'd suggest a few points to consider.

the difference in error between the reports at aperture 140 and at smaller apertures (the variable aperture method?) should be explored. is your method biasing your results by misrepresenting the resolution required to split a star?

second, whatever method you use, i think you really stack the deck against yourself by relying on your own observations and using a method not used by other observers. you have to get as much error out of the data as possible, and this means some different eyes and all eyes see things exactly the same way.

third, even if you rely on your own observations and variable aperture method, it's critical to use a wider range of instruments. variable aperture cannot change objective magnification or native focal ratio; an /10 140mm refractor stopped down to 100mm is not the same as an /10 100mm refractor.

finally, your research has taught you a lot about what matters more or less in visual resolution. with that knowledge in hand, i strongly urge you to try to simplify your formula. it is so complicated that i doubt even you can explain what it does (certainly i can't), and it is too complicated to perform well with new observers, new instruments, and new telescopes.

good luck.




Bruce, finally the response to your suggestions:
- First "difference in error between fixed and variable aperture": Using the term "error" instead of "standard deviation" is my fault so we should better stick to "standard deviation". I separated my data set into variable and fixed aperture and the standard separation is greater for the fixed aperture - and this is to expect as the spread in the observations has to be larger with fixed apertures. Don't see any problem here for my approach with variable aperture avoiding both over and under performing
- Second "relying on my own observations and using a method not used by other observers": I try to get as many usable reports from other observers - but the questions usually remains what can be considered as limit observation. I am very cautious regarding the quality of my own observations but in doubt I tend to rely on my own experience. The technique of variable aperture and CO is a very powerful tool for getting limit observations fast and reliable so it would be counter productive to not use it any more
- Third "use of a wider range of instruments": Yes, this is what I intend to do but also with the possibility of using a variable aperture. An 8" DK with CO of 0.25 is my next step - then I can cover the range of 150 to 200mm aperture myself. I don't think that I will go much further as this would require an equipment difficult to handle as for example a large Newton. SCT's are for me of no good use as the CO is already on the upper limit for useful limit observations. Regarding "an /10 140mm refractor stopped down to 100mm is not the same as an /10 100mm refractor" - this may be true but it is then an f/14 100mm refractor so I don't see the problem. I have several smaller refractors available but prefer the 140mm with variable aperture and variable CO as perfect instrument up to 140mm (only for 60mm I prefer my C9.25 stopped down off-axis to 60mm with stunning images). To some degree I think that larger f/ratio scopes are better suited for splitting doubles but don't see this as major factor
- Fourth "simplify your formula": I might try but currently I am considering to make my formula even more complex with additional modules for CO and near TML faint companions. Currently my formula works best compared with all other RoTs known to me regardless what data set is considered - even when I separate my data set into variable and fixed aperture subsets with the latter predominantly from other observers and not me. I even eliminated all doubles producing errors for Napier-Munn to give this algorithm a fair chance. Results in terms of correlation coefficient:
Data set with variable aperture (=WRAK only): WRAK 0.80, Bruce 0.28, Napier-Munn 0.62, Lord 0.45 and just for comparison Dawes 0.45
Data set with fixed apertures (=predominantly not WRAK, therefore less faint pairs): WRAK 0.79, Bruce 0.55, Napier-Munn 0.63, Lord 0.72 and Dawes 0.59.

I think I will continue on my path and thank you for provoking me to go into some details.
Wilfried


Post Extras: Print Post   Remind Me!   Notify Moderator  
Pages: 1


Extra information
1 registered and 1 anonymous users are browsing this forum.

Moderator:  PJ Anway, Rich (RLTYS), rflinn68 

Print Thread

Forum Permissions
      You cannot start new topics
      You cannot reply to topics
      HTML is disabled
      UBBCode is enabled


Thread views: 621

Jump to

CN Forums Home




Cloudy Nights Sponsor: Astronomics