Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Turn OFF Local Normalization in PixInsight WeightedBatchPreprocessing

  • Please log in to reply
55 replies to this topic

#1 Michael Covington

Michael Covington

    Author

  • *****
  • topic starter
  • Posts: 9,301
  • Joined: 13 May 2014
  • Loc: Athens, Georgia, USA

Posted 07 June 2022 - 06:18 PM

[NOTE ADDED: This problem was NOT what is seemed to be, and to avoid giving PixInsight a bad name, the moderator has allowed me to add this comment.
The problem does go away if you turn Local Normalization off, but it also goes away if you set Local Normalization to build the reference frame by
integrating the best frames, which is supposed to be its default.  Under UNCOMMON circumstances, a different default from an earlier version can be left
behind, causing the problem reported here.  Read on...]

 

In at least the last 2 versions of PixInsight's WeightedBatchPreprocessing script [EDIT: The latest versions, including today's], the Local Normalization option has defaulted to ON.  I found it was causing a problem with my pictures, and in fact that PixInsight's lead developer (Juan Conejero) recommends NOT using Local Normalization unless there is a specific problem for it to solve. [NOTE ADDED: That is not his current recommendation.  Local Normalization has been improved and normally works well.] Accordingly, I am getting much better results with it OFF, and it sounds like most people will.

Here is Juan Conejero's sage advice:  https://pixinsight.c...115/#post-86421 [NOTE ADDED: Not his current recommendation!]

Here is a thread I started on PixInsight Forums to discuss this, and to request that the default be changed to OFF:
https://pixinsight.c...ocessing.18652/

 

The problem I experienced is that low-level noise and color non-uniformity in the sky background gets turned into dramatic colored patches if Local Normalization is on.  Example below.

This was taken with a Nikon D5500 and C8 EdgeHD with f/7 reducer.  For a long time I thought I was battling the Nikon lossy compression discussed here:
https://www.cloudyni...ncentric-rings/
My sky background level and flats are both well below 2100 ADU, the level at which lossy compression supposedly starts on this camera (they are in fact both a bit below 1000 ADU).  Nonetheless, it's possible this is the cause of the slight discoloration shown in the upper image.  What I know is that Local Normalization makes it much worse.

Sage insights would be welcomed!

Moderator's Note: I have edited this post at the request of Michael to reflect the findings discovered throughout this thread. The edits, all written by Michael, are added inline above. I have BOLDED the "NOTE ADDED" text indicating where these edits were inserted. The edited text is surrounded by [].

Attached Thumbnails

  • 20220606-LocalNormalizationEffect.jpg

Edited by jonnybravo0311, 09 June 2022 - 09:47 PM.
OP request to correct some statements

  • dswtan, lambermo, Jim Waters and 6 others like this

#2 Jim Waters

Jim Waters

    Cosmos

  • *****
  • Posts: 8,525
  • Joined: 21 Oct 2007
  • Loc: Phoenix, AZ USA

Posted 07 June 2022 - 06:36 PM

This could be why everything went to h*** when I upgraded to the current WBPP script.  Thanks.  I will reprocess my images.


  • dswtan, elmiko and Michael Covington like this

#3 Michael Covington

Michael Covington

    Author

  • *****
  • topic starter
  • Posts: 9,301
  • Joined: 13 May 2014
  • Loc: Athens, Georgia, USA

Posted 07 June 2022 - 06:50 PM

Probably the one a few weeks ago, not the one that came out today (which is the same as far as this is concerned).  I had a similar experience -- started seeing more background discoloration than before and not knowing why.


  • arrowspace90 likes this

#4 tkottary

tkottary

    Viking 1

  • *****
  • Posts: 818
  • Joined: 06 Dec 2015
  • Loc: SunnyVale ,CA

Posted 07 June 2022 - 07:52 PM

Not sure what's going with your data set , but this is an old advice. LN was re-written in the last release and is considerably much robust than when this advice was given. How does your LN config look like inside WBPP?

 

In at least the last 2 versions of PixInsight's WeightedBatchPreprocessing script, the Local Normalization option has defaulted to ON.  I found it was causing a problem with my pictures, and in fact that PixInsight's lead developer (Juan Conejero) recommends NOT using Local Normalization unless there is a specific problem for it to solve.  Accordingly, I am getting much better results with it OFF, and it sounds like most people will.

Here is Juan Conejero's sage advice:  https://pixinsight.c...115/#post-86421
 


Edited by tkottary, 07 June 2022 - 07:52 PM.


#5 Michael Covington

Michael Covington

    Author

  • *****
  • topic starter
  • Posts: 9,301
  • Joined: 13 May 2014
  • Loc: Athens, Georgia, USA

Posted 07 June 2022 - 08:08 PM

Not sure what's going with your data set , but this is an old advice. LN was re-written in the last release and is considerably much robust than when this advice was given. How does your LN config look like inside WBPP?

Whatever the default is.  I didn't set the configuration.



#6 Midnight Dan

Midnight Dan

    James Webb Space Telescope

  • *****
  • Posts: 15,823
  • Joined: 23 Jan 2008
  • Loc: Hilton, NY, Yellow Zone (Bortle 4.5)

Posted 07 June 2022 - 08:51 PM

Thanks for this!  I've been battling this issue and thought I was going nuts.  :-)

 

-Dan


  • arrowspace90 and Juno18 like this

#7 fmeschia

fmeschia

    Vanguard

  • *****
  • Posts: 2,027
  • Joined: 20 May 2016
  • Loc: Mountain View, CA

Posted 07 June 2022 - 08:55 PM

Whatever the default is.  I didn't set the configuration.

Can you try to apply the same STF to both pictures in your example? I wonder whether the colored patches just appear more evident due to the overall better equalization of the field, which leads auto-STF to a more aggressive stretch.


Edited by fmeschia, 07 June 2022 - 08:58 PM.


#8 Michael Covington

Michael Covington

    Author

  • *****
  • topic starter
  • Posts: 9,301
  • Joined: 13 May 2014
  • Loc: Athens, Georgia, USA

Posted 07 June 2022 - 09:04 PM

Can you try to apply the same STF to both pictures in your example? I wonder whether the colored patches just appear more evident due to the overall better equalization of the field, which leads auto-STF to a more aggressive stretch.

I thought of that.  To get them to look alike, I did use different STFs.  However, I could not bring out the dramatic pattern at the bottom no matter what I did to the top image.  At worst it still had a considerably weaker and simpler discoloration.  Local Normalization definitely contributes to the problem, as has also been my experience with other images.

I would still like to troubleshoot the original source of the discoloration, of course.


Edited by Michael Covington, 07 June 2022 - 09:35 PM.


#9 SeymoreStars

SeymoreStars

    Skylab

  • *****
  • Posts: 4,218
  • Joined: 08 May 2014
  • Loc: Pennsyltucky

Posted 07 June 2022 - 09:07 PM

If you updated to PI 1.8.9-1 a new version of PI's WBPP is available with many changes.

 

 

Attached Thumbnails

  • Capture.JPG


#10 SeymoreStars

SeymoreStars

    Skylab

  • *****
  • Posts: 4,218
  • Joined: 08 May 2014
  • Loc: Pennsyltucky

Posted 07 June 2022 - 09:08 PM

Get PI 1.8.9-1 here - https://pixinsight.c...65/#post-112629



#11 SeymoreStars

SeymoreStars

    Skylab

  • *****
  • Posts: 4,218
  • Joined: 08 May 2014
  • Loc: Pennsyltucky

Posted 07 June 2022 - 09:14 PM

The WBPP shows all steps and results, very handy.

Attached Thumbnails

  • Capture.JPG


#12 Michael Covington

Michael Covington

    Author

  • *****
  • topic starter
  • Posts: 9,301
  • Joined: 13 May 2014
  • Loc: Athens, Georgia, USA

Posted 07 June 2022 - 09:27 PM

I am using the new version (with PixInsight 1.8.9-1).  When I said "at least the last 2 versions" I meant today's version and the one just before it.


Edited by Michael Covington, 07 June 2022 - 09:36 PM.


#13 Jared

Jared

    Cosmos

  • *****
  • Moderators
  • Posts: 7,970
  • Joined: 11 Oct 2005
  • Loc: Piedmont, California, U.S.

Posted 07 June 2022 - 11:40 PM

Juan's statement was accurate in 2020 when local normalization had some serious down-sides. I doubt very much he would give the same advice today, which is why the default is to use local normalization coupled with some flavor of SNR-based weighting. If everything works properly (and I haven't had any issues with my data), that should yield the best possible SNR with the simples possible gradients for DBE to address.

 

Off-hand, I don't know what went wrong with your data and the local normalization process. I wish I could offer advice. But I don't agree that the default for local normalization should be "off". At least within PI, I would argue it is now a best practice to incorporate local normalization so that one can use an SNR based weighting to get the best possible SNR. Couple that with good error rejection in integration and you should get the sharpest, cleanest results you can for your data. If it isn't working for you, I get it, but giving general advice to turn it off isn't a good idea. It solves a very real problem--of how to simplify gradients and at the same time evaluate and weight images effectively for integration. Based on my testing (running images both with and without, trying various combinations of image weighting factors such as PSF SNR, eccentricity, FWHM, etc.) it is extremely effective and yields the best overall SNR in the integrated image. If it isn't working for you, turn it off till you solve the problem or turn it off indefinitely as you see fit, but I don't think that is a good general practice for everyone.

 

One thing that jumped out in your original statement, by the way... You said that your flats had an average ADU well below 1,000. I don't know anything about the lossy compression issue you mentioned with your Nikon, but generally speaking having flats at 1,000 ADU would be way, way too low. Not saying that's what's causing your issues with blotchy, uneven backgrounds, but are you sure ADU's that low in your flats are a good idea with your camera? Most people would recommend somewhere around 30,000. With most cameras there is a fairly wide range of acceptable, but 1,000 seems to really be pushing it. I have had cameras where the offset alone was that high. Or am I not understanding something specific to the Nikon?


  • Midnight Dan and tkottary like this

#14 Jim Waters

Jim Waters

    Cosmos

  • *****
  • Posts: 8,525
  • Joined: 21 Oct 2007
  • Loc: Phoenix, AZ USA

Posted 08 June 2022 - 12:11 AM

I have what looks like identical problems as Michael.  Unselecting Local Norm somewhat corrected the issues but I need to do further processing. For me this started with PI 1.8.9-1 and the two recent WBPP releases. 

 

I would like to understand what's going on.

 

EDIT - I see this on an ASI2600MC Pro CMOS.


Edited by Jim Waters, 08 June 2022 - 12:14 AM.

  • Midnight Dan likes this

#15 sharkmelley

sharkmelley

    Fly Me to the Moon

  • *****
  • Posts: 7,364
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 08 June 2022 - 12:12 AM

Or am I not understanding something specific to the Nikon?

Yes, there is a nefarious data compression issue specific to Nikon.



#16 Jared

Jared

    Cosmos

  • *****
  • Moderators
  • Posts: 7,970
  • Joined: 11 Oct 2005
  • Loc: Piedmont, California, U.S.

Posted 08 June 2022 - 01:36 AM

Yes, there is a nefarious data compression issue specific to Nikon.

Well, that would certainly be annoying.



#17 17.5Dob

17.5Dob

    Voyager 1

  • *****
  • Posts: 10,266
  • Joined: 21 Mar 2013
  • Loc: Colorado,USA

Posted 08 June 2022 - 01:37 AM

Interesting !!! My last two images, processed last week, doing everything exactly as I always have, were a "What The ...." mess. I'll have to go back and see what's different.
  • Midnight Dan likes this

#18 Jared

Jared

    Cosmos

  • *****
  • Moderators
  • Posts: 7,970
  • Joined: 11 Oct 2005
  • Loc: Piedmont, California, U.S.

Posted 08 June 2022 - 01:42 AM

I have what looks like identical problems as Michael.  Unselecting Local Norm somewhat corrected the issues but I need to do further processing. For me this started with PI 1.8.9-1 and the two recent WBPP releases. 

 

I would like to understand what's going on.

 

EDIT - I see this on an ASI2600MC Pro CMOS.

I know there was a new star detection algorithm incorporated into 1.8.9-1 and that would affect local normalization. I'm still on 1.8.9 so can't speak to the newer release. You didn't see this on 1.8.9? It just appeared on 1.8.9-1?



#19 arbit

arbit

    Apollo

  • -----
  • Posts: 1,365
  • Joined: 19 Feb 2012

Posted 08 June 2022 - 02:40 AM

I've had this when using registered files output from the "old" local normalization algos before 1.8.9 with the new algo. I think it can also happen if the smoothing is not quite right, but that's done automatically in WBPP.

 

I've switched to NSG for any LN now. IMO, it has far better fine control in terms of visual feedback for reference frames (though todays WBPP 2.4.5 closes the gap somewhat), gradient smoothing for best fit, ROI for PSF evaluation, and sub rejection, including a visual guide.



#20 Juno18

Juno18

    Viking 1

  • *****
  • Posts: 816
  • Joined: 07 Dec 2018
  • Loc: Long Beach, Mississippi, USA

Posted 08 June 2022 - 06:22 AM

Dang!

 

Thank you Michael!

 

Had the strange color gradient on the last image. I even rolled back PI to a previous version (from a disk image) trying to figure out this issue. When I reinstalled 1.8.9, the problem came back.

 

I too was using WBPP with LN checked and using defaults. I tried deselecting "Allow clustered sources" (my image was M92) without any success.

 

On an image previous to the last one (M92), I went ahead and processed the master produced by WBPP with the messed up color patches using an extremely aggressive DBE technique and it cleaned it up enough to process. 

 

W and H.PNG

 

Just downloaded the newest WBPP script yesterday (2.4.5). Hopefully, I can use LN without this issue.

 

Thanks a ton for this thread!


Edited by Juno18, 08 June 2022 - 06:27 AM.


#21 Midnight Dan

Midnight Dan

    James Webb Space Telescope

  • *****
  • Posts: 15,823
  • Joined: 23 Jan 2008
  • Loc: Hilton, NY, Yellow Zone (Bortle 4.5)

Posted 08 June 2022 - 06:53 AM

One thing that jumped out in your original statement, by the way... You said that your flats had an average ADU well below 1,000. I don't know anything about the lossy compression issue you mentioned with your Nikon, but generally speaking having flats at 1,000 ADU would be way, way too low. 

Just want to mention that my images were looking exactly like Micheal's and my flats were at 25K.  I'm using an ASI071MC Pro camera

 

-Dan



#22 arbit

arbit

    Apollo

  • -----
  • Posts: 1,365
  • Joined: 19 Feb 2012

Posted 08 June 2022 - 07:04 AM

I've had this when using registered files output from the "old" local normalization algos before 1.8.9 with the new algo. I think it can also happen if the smoothing is not quite right, but that's done automatically in WBPP.

 

I've switched to NSG for any LN now. IMO, it has far better fine control in terms of visual feedback for reference frames (though todays WBPP 2.4.5 closes the gap somewhat), gradient smoothing for best fit, ROI for PSF evaluation, and sub rejection, including a visual guide.

Samples. Test stack of 10 subs each with NSG. Left is at low gradient smoothness (-4), and right with higher (-1).

 

Dont know how visible it is compressed, but the left is more patchy. The impact is much more visible in the full stack.

 

The smoothness graphs are also attached.

Attached Thumbnails

  • NSG_Neg4_Neg1_CN.png
  • NSG_Neg4_Neg1_Smoothness.png


#23 Scott Badger

Scott Badger

    Messenger

  • -----
  • Posts: 427
  • Joined: 13 Apr 2021

Posted 08 June 2022 - 07:33 AM

This issue came up in a Pixinsight forum thread and Juan Conejero suggested using a best-of integration as the reference and also upping the scale to 768, or 1024. What made the most difference though, was drizzling. FWIW.

 

Cheers,
Scott



#24 Michael Covington

Michael Covington

    Author

  • *****
  • topic starter
  • Posts: 9,301
  • Joined: 13 May 2014
  • Loc: Athens, Georgia, USA

Posted 08 June 2022 - 08:33 AM

Update:  Juan Conejero confirms what a couple of you said, that Local Normalization has been greatly improved since he advised people in 2020 not to use it.

However, it appears that its default parameters are incorrect, so I stand by my opinion that it should be off by default, leaving it to the user to turn it on and make appropriate settings (which I haven't delved into yet; I know nothing).

 

In this thread I only wanted to make one point, which is that turning Local Normalization on by default was not the right move; it surprised people by making their images come out worse.

 

My low ADUs are to avoid a Nikon data compression issue.  The ADU scale on this camera goes to 16383, not 64k, so 1000 is not super-low.

 

Admittedly those discolorations are appearing in *extremely* stretched, faint images.  Their amplitude may be less than 1 ADU (possible because stacking is done with floating-point numbers). 

I have not yet had time to read everything on this thread closely -- just wanted to give a quick update -- will be back when time permits.


  • Juno18 likes this

#25 bulrichl

bulrichl

    Viking 1

  • *****
  • Posts: 506
  • Joined: 27 May 2018
  • Loc: La Palma (Canary Islands)

Posted 08 June 2022 - 09:17 AM

Update:  Juan Conejero confirms what a couple of you said, that Local Normalization has been greatly improved since he advised people in 2020 not to use it.

However, it appears that its default parameters are incorrect, ...

Did you reset WBPP choosing the option 'Reset all parameters to factory-default values?' when you tried LocalNormalization?

 

If not, you didn't set the default parameters, instead, the settings of your last WBPP run were used.

 

The default parameters are:

Generate images:              disabled
Reference frame generation:   Integration of best frames
Evaluation critera:           PSF Signal Weight
Normalization scale:          512
Scale evaluation method:      PSF flux evaluation
PSF type:                     Auto
Growth factor:                1.00
Maximum stars:                24576
Minimum detection SNR:        40
Allow clustered sources:      enabled
Low clipping level            4.50e-05
High clipping level:          0.85

You can compare whether these values were used.

 

Bernd

 


  • elmiko likes this


CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics