Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Cannot trust PI's LinearFit, help!

  • Please log in to reply
32 replies to this topic

#1 futuneral

futuneral

    Apollo

  • *****
  • topic starter
  • Posts: 1,262
  • Joined: 27 Dec 2014
  • Loc: Phoenix, AZ

Posted 06 July 2020 - 11:46 PM

I'm having a hard time with the LinearFit process. I think I understand the math theory behind it, thanks to Jon Rista's explanation on his website. However, I'm sure I do not understand what it is actually doing to my images in practice. Sometimes it works fine, but some times it just destroys the image, and because of this unpredictability I cannot really trust it and try to avoid using it.

 

Here is an example. These are integrations of 3 panels of the mosaic, all in Ha. Top left is used as a reference frame. The top row shows the originals (STF'ed), the bottom row is after applying LinearFit. What's happening here? 

 

linearfit.jpg



#2 imtl

imtl

    Viking 1

  • *****
  • Posts: 675
  • Joined: 07 Jun 2016

Posted 06 July 2020 - 11:55 PM

Don't use linear fit with mosaics. Use dnalinearfit script.


  • james7ca likes this

#3 futuneral

futuneral

    Apollo

  • *****
  • topic starter
  • Posts: 1,262
  • Joined: 27 Dec 2014
  • Loc: Phoenix, AZ

Posted 07 July 2020 - 12:02 AM

Don't use linear fit with mosaics. Use dnalinearfit script.

Interesting, thanks, I'll check it out.

 

Inline with the original post however - why? What's going wrong? What does dnalinearfit do differently?

 

Also, this happens not only with mosaics but in normal processing too. Sometimes it just goes haywire.



#4 imtl

imtl

    Viking 1

  • *****
  • Posts: 675
  • Joined: 07 Jun 2016

Posted 07 July 2020 - 12:08 AM

It depends on what reference you choose. Look at your histograms for the frames first before you decide on the reference frame. After you did the linear fit to your p2_clone and p6_clone. Did you apply the autostretch again? Otherwise it will just stay with the original autostretch which will cause the image to look weird like yours does.

 

What you should do is apply the same STF from the reference to the other two frames.

 

In any case, for dnalinearfit you can read Dave's blog post. Its an excellent tool.

http://trappedphotons.com/blog/?p=994


Edited by imtl, 07 July 2020 - 12:11 AM.


#5 futuneral

futuneral

    Apollo

  • *****
  • topic starter
  • Posts: 1,262
  • Joined: 27 Dec 2014
  • Loc: Phoenix, AZ

Posted 07 July 2020 - 12:08 AM

Reading up on dnaLinearFit, just one note: seems the main goal of that algo was to prevent it from raising the dark levels. I understand that would be an issue with mosaics after you align panels to the star field.

 

But in my case I'm LinearFit-ting them before aligning, so that shouldn't be the issue - there is no black level to preserve.



#6 futuneral

futuneral

    Apollo

  • *****
  • topic starter
  • Posts: 1,262
  • Joined: 27 Dec 2014
  • Loc: Phoenix, AZ

Posted 07 July 2020 - 12:11 AM

It depends on what reference you choose. Look at your histograms for the frames first before you decide on the reference frame.

 

In any case, for dnalinearfit you can read Dave's blog post. Its an excellent tool.

http://trappedphotons.com/blog/?p=994

 

Right, but I did P1 to P6 and P6 to P1 and they produce the same result (something like a negative image). What is it about the reference frame that decides whether the operation is going to be successful or not? What if I don't have any frames that match those requirements?

 

p.s. definitely checking out dnalinearfit (and if that works, it'll kind of confirm my doubts :) )



#7 imtl

imtl

    Viking 1

  • *****
  • Posts: 675
  • Joined: 07 Jun 2016

Posted 07 July 2020 - 12:22 AM

How did you choose your reference frame? How does it histogram looks compared to the other frames? You can go with the brightest one to start with. Meaning the frame that has a histogram as far to the right from all. What do you get then?

 

And asking again, after you did the linear fit. Did you apply STF to the frames again?

 

Look at the frames histograms before you do linear fit and after. What do you see?


Edited by imtl, 07 July 2020 - 12:24 AM.


#8 futuneral

futuneral

    Apollo

  • *****
  • topic starter
  • Posts: 1,262
  • Joined: 27 Dec 2014
  • Loc: Phoenix, AZ

Posted 07 July 2020 - 12:52 AM

How did you choose your reference frame? How does it histogram looks compared to the other frames? You can go with the brightest one to start with. Meaning the frame that has a histogram as far to the right from all. What do you get then?

 

And asking again, after you did the linear fit. Did you apply STF to the frames again?

 

Look at the frames histograms before you do linear fit and after. What do you see?

I mean.. it's a six panel. But we could assume it's just two - say P1 and P6 from my screenshot. I tried both ways and neither works.

 

Yes, I did apply the STF again. The histogram looks like a single spike. I used HT to expand that spike to the full range, and compressed within that spike is a low contrast negative of the original...

 

My own theory is that if the images are "too similar" (and I don't know quantitatively what that means), the deltas are really small, maybe at the limit of the float or double (or whatever is used), so the resulting linear function is super close to horizontal, but can be easily thrown off significantly by some noise pixel or something and become totally wrong.



#9 imtl

imtl

    Viking 1

  • *****
  • Posts: 675
  • Joined: 07 Jun 2016

Posted 07 July 2020 - 12:54 AM

Are the frames calibrated? Can you upload 2 of them, calibrated but not processed,  to take a look?


Edited by imtl, 07 July 2020 - 12:55 AM.


#10 futuneral

futuneral

    Apollo

  • *****
  • topic starter
  • Posts: 1,262
  • Joined: 27 Dec 2014
  • Loc: Phoenix, AZ

Posted 07 July 2020 - 01:03 AM

Yes, calibrated and integrated.

 

Oof, not sure about uploading, it's a 400Mb zip with both of them, you sure you want them? ;) Might take a while



#11 imtl

imtl

    Viking 1

  • *****
  • Posts: 675
  • Joined: 07 Jun 2016

Posted 07 July 2020 - 01:11 AM

Just send a drop box link. 400Mb? What do you use a 100Mp camera?! Send a link and I'll take a look



#12 futuneral

futuneral

    Apollo

  • *****
  • topic starter
  • Posts: 1,262
  • Joined: 27 Dec 2014
  • Loc: Phoenix, AZ

Posted 07 July 2020 - 01:25 AM

Haha, it's a drizzle integration and I want to keep it at that scale until final cropping.

 

Here is the file: https://drive.google...iew?usp=sharing

 

Also, dnaLinearFit seem to work well, haven't compiled the mosaic yet, but individual frames with the same STF applied look great. Which contributes to my suspicion that there is a bug in the LinearFit process.



#13 imtl

imtl

    Viking 1

  • *****
  • Posts: 675
  • Joined: 07 Jun 2016

Posted 07 July 2020 - 01:42 AM

Okay so I looked at the data. Linear fit is having a hard time with it. dnalinearfit works fine. I am not sure yet to why its happening but looking at the histogram after doing LT I can see why the image looks like that. A lot of the data is getting clipped. And that is happening because I think your SNR is quite low. It might be because you "over drizzled". If I may suggest something to you. Try and integrate without drizzling. This is a wide field shot and drizzling too much won't win you that much. But you are hurting in SNR.

What is your image scale?



#14 futuneral

futuneral

    Apollo

  • *****
  • topic starter
  • Posts: 1,262
  • Joined: 27 Dec 2014
  • Loc: Phoenix, AZ

Posted 07 July 2020 - 02:19 AM

Hmm, I would think SNR is not supposed to affect LinearFit...

 

But I agree, drizzling doesn't bring me much in terms of detail. I noticed however, that drizzling x2 and then downscaling 50% produces less noisy image. Actually, my drizzle integration is less noisy than the regular one (according to Noise Evaluation script). But generally drizzle->reduce noise->downscale was more manageable to me in terms of controlling noise.

 

We're maybe deviating from the original subject, but can you explain why drizzled image would have a lower SNR? 



#15 futuneral

futuneral

    Apollo

  • *****
  • topic starter
  • Posts: 1,262
  • Joined: 27 Dec 2014
  • Loc: Phoenix, AZ

Posted 07 July 2020 - 02:23 AM

Oh, forgot to answer - I'm at about 1.4"/px currently. I am suspecting this is actually better than most of the seeing I'm getting recently, so yeah, drizzling that may not bring any benefits. But on the off-chance it does - I'll take it.



#16 imtl

imtl

    Viking 1

  • *****
  • Posts: 675
  • Joined: 07 Jun 2016

Posted 07 July 2020 - 02:41 AM

If you have a low SNR then it affects all processing. Again, I'm not claiming anything. I'm just thinking out loud what could cause linearfit to clip your data like that. I tried to play with the rejection scales of it and it helps a bit. dnalinearfit is more robust and gave good fit easily.

 

If you have an image scale of 1.4"/px then that is pretty good already. Drizzle really helps when you are undersampled. When you are properly sampled to your seeing then drizzle does not change much. When you are oversampled it will hurt you.

 

Think about it in the simplest way (in reality of course its not), you take the light from a single pixel and spread it over 4. Yes the algorithm is sophisticated and does magic. No argue about that. But you cannot bypass basic concepts. If you are undersampled then there is "hidden" resolution to be recovered. If you are oversampled then you're just lowering your SNR really without getting more details. 

 

That is how I see things. In a simple way. Its more complicated than that. I'm sure others will have opinions about this as well.

 

By the way, you do not need to drizzle x2 and then downsample x2 to get the effect you're getting with noise reduction. You could just drizzle x1 (yes I know that I wrote x1) and you are going to get the same effect. Its an interpolation algorithm and will do a noise rejection techniques when manipulating the data.

 

 

The best noise reduction technique is... take more data. 


Edited by imtl, 07 July 2020 - 02:52 AM.

  • futuneral likes this

#17 futuneral

futuneral

    Apollo

  • *****
  • topic starter
  • Posts: 1,262
  • Joined: 27 Dec 2014
  • Loc: Phoenix, AZ

Posted 07 July 2020 - 02:53 AM

 

The best noise reduction technique is... take more data. 

I know, I know... But I'm already past 40 hours on this image and I want to get it done before I die from old age haha.

 

I'm planning to get more data after this moon goes away...


  • imtl likes this

#18 imtl

imtl

    Viking 1

  • *****
  • Posts: 675
  • Joined: 07 Jun 2016

Posted 07 July 2020 - 03:08 AM

40 hours/6 panels. In Ha. That is less than 7 hours per panel. You can do better :p :p :p

 

I doing a 9 panel now. Its a pain in the .... And I'm doing LRGB. Its a processing mountain to climb.

 

Good luck!


  • futuneral likes this

#19 futuneral

futuneral

    Apollo

  • *****
  • topic starter
  • Posts: 1,262
  • Joined: 27 Dec 2014
  • Loc: Phoenix, AZ

Posted 07 July 2020 - 11:03 AM

40 hours/6 panels. In Ha. That is less than 7 hours per panel. You can do better tongue2.giftongue2.giftongue2.gif

 

 

I wish. 40 hours is all three filters.So 2 hours per sub - not too deep.



#20 Peregrinatum

Peregrinatum

    Apollo

  • *****
  • Posts: 1,266
  • Joined: 27 Dec 2018
  • Loc: South Central Valley, Ca

Posted 07 July 2020 - 02:04 PM

after I apply LF the image usually  looks terrible, then after I deactivate the STF and then reapply it using the STFProcess and 'hazard' icon it looks fine.



#21 Dean J.

Dean J.

    Messenger

  • -----
  • Posts: 483
  • Joined: 13 Aug 2011
  • Loc: Vista, CA

Posted 07 July 2020 - 02:43 PM

after I apply LF the image usually  looks terrible, then after I deactivate the STF and then reapply it using the STFProcess and 'hazard' icon it looks fine.

Yes, the "old" STF solution after a linear fit will not look like it did before.  You need to refresh the calculation and have it apply the new solution.



#22 spokeshave

spokeshave

    Vanguard

  • *****
  • Posts: 2,302
  • Joined: 08 Apr 2015

Posted 07 July 2020 - 02:57 PM

I have experienced the same thing and it isn't just a matter of resetting the STF. LF completely ruins the image. I remember figuring out why it happens, but I don't remember what I figured out. Unfortunately, I can't find it in my notes either. I seem to recall that it had something to do with black regions around the edges from registering. Have all stacking and registering artifacts been cropped from the images?

 

Tim


  • futuneral likes this

#23 imtl

imtl

    Viking 1

  • *****
  • Posts: 675
  • Joined: 07 Jun 2016

Posted 07 July 2020 - 03:36 PM

I forgot to ask the OP about that actually. If the panels were not cropped then linearfit will throw a fit.



#24 yzhzhang

yzhzhang

    Viking 1

  • *****
  • Posts: 814
  • Joined: 13 Dec 2012
  • Loc: San Diego, CA

Posted 07 July 2020 - 03:45 PM

I just ran into the same issue today. Linearfitting a slightly "dimmer" panel to match a slightly "brigher" one, and LinearFit essentially reject all pixels, leaving an image with constant value across all pixels, making the histogram after LinearFit a single vertical line and the mean of "brigher" frame. Any idea?



#25 futuneral

futuneral

    Apollo

  • *****
  • topic starter
  • Posts: 1,262
  • Joined: 27 Dec 2014
  • Loc: Phoenix, AZ

Posted 07 July 2020 - 03:55 PM

I just ran into the same issue today. Linearfitting a slightly "dimmer" panel to match a slightly "brigher" one, and LinearFit essentially reject all pixels, leaving an image with constant value across all pixels, making the histogram after LinearFit a single vertical line and the mean of "brigher" frame. Any idea?

Exactly this. In my experience if the frames differ only slightly in brightness, this is what happens.

 

 

I forgot to ask the OP about that actually. If the panels were not cropped then linearfit will throw a fit.

They are not cropped, that's true. I do not think I have any black edges though. Gotta try this. But still, the biggest elephant in the room - dnaLinearFit works wonderfully, so it should be possible with whatever data I have.

 

Also, "will throw a fit" - I see what you did there ;)


  • imtl likes this


CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics