55 replies to this topic

[Michael Covington]

[NOTE ADDED: This problem was NOT what is seemed to be, and to avoid giving PixInsight a bad name, the moderator has allowed me to add this comment.
The problem does go away if you turn Local Normalization off, but it also goes away if you set Local Normalization to build the reference frame by
integrating the best frames, which is supposed to be its default.  Under UNCOMMON circumstances, a different default from an earlier version can be left
behind, causing the problem reported here.  Read on...]

 

In at least the last 2 versions of PixInsight's WeightedBatchPreprocessing script [EDIT: The latest versions, including today's], the Local Normalization option has defaulted to ON.  I found it was causing a problem with my pictures, and in fact that PixInsight's lead developer (Juan Conejero) recommends NOT using Local Normalization unless there is a specific problem for it to solve. [NOTE ADDED: That is not his current recommendation.  Local Normalization has been improved and normally works well.] Accordingly, I am getting much better results with it OFF, and it sounds like most people will.

Here is Juan Conejero's sage advice:  https://pixinsight.c...115/#post-86421 [NOTE ADDED: Not his current recommendation!]

Here is a thread I started on PixInsight Forums to discuss this, and to request that the default be changed to OFF:
https://pixinsight.c...ocessing.18652/

 

The problem I experienced is that low-level noise and color non-uniformity in the sky background gets turned into dramatic colored patches if Local Normalization is on.  Example below.

This was taken with a Nikon D5500 and C8 EdgeHD with f/7 reducer.  For a long time I thought I was battling the Nikon lossy compression discussed here:
https://www.cloudyni...ncentric-rings/
My sky background level and flats are both well below 2100 ADU, the level at which lossy compression supposedly starts on this camera (they are in fact both a bit below 1000 ADU).  Nonetheless, it's possible this is the cause of the slight discoloration shown in the upper image.  What I know is that Local Normalization makes it much worse.

Sage insights would be welcomed!

Moderator's Note: I have edited this post at the request of Michael to reflect the findings discovered throughout this thread. The edits, all written by Michael, are added inline above. I have BOLDED the "NOTE ADDED" text indicating where these edits were inserted. The edited text is surrounded by [].

Attached Thumbnails

• 

Edited by jonnybravo0311, 09 June 2022 - 09:47 PM.
OP request to correct some statements

[Jim Waters]

This could be why everything went to h*** when I upgraded to the current WBPP script.  Thanks.  I will reprocess my images.

[Michael Covington]

Probably the one a few weeks ago, not the one that came out today (which is the same as far as this is concerned).  I had a similar experience -- started seeing more background discoloration than before and not knowing why.

[tkottary]

Not sure what's going with your data set , but this is an old advice. LN was re-written in the last release and is considerably much robust than when this advice was given. How does your LN config look like inside WBPP?

 

In at least the last 2 versions of PixInsight's WeightedBatchPreprocessing script, the Local Normalization option has defaulted to ON.  I found it was causing a problem with my pictures, and in fact that PixInsight's lead developer (Juan Conejero) recommends NOT using Local Normalization unless there is a specific problem for it to solve.  Accordingly, I am getting much better results with it OFF, and it sounds like most people will.

Here is Juan Conejero's sage advice:  https://pixinsight.c...115/#post-86421
 

Edited by tkottary, 07 June 2022 - 07:52 PM.

[Michael Covington]

Not sure what's going with your data set , but this is an old advice. LN was re-written in the last release and is considerably much robust than when this advice was given. How does your LN config look like inside WBPP?

Whatever the default is.  I didn't set the configuration.

[Midnight Dan]

Thanks for this!  I've been battling this issue and thought I was going nuts.  :-)

 

-Dan

[fmeschia]

Whatever the default is.  I didn't set the configuration.

Can you try to apply the same STF to both pictures in your example? I wonder whether the colored patches just appear more evident due to the overall better equalization of the field, which leads auto-STF to a more aggressive stretch.

Edited by fmeschia, 07 June 2022 - 08:58 PM.

[Michael Covington]

Can you try to apply the same STF to both pictures in your example? I wonder whether the colored patches just appear more evident due to the overall better equalization of the field, which leads auto-STF to a more aggressive stretch.

I thought of that.  To get them to look alike, I did use different STFs.  However, I could not bring out the dramatic pattern at the bottom no matter what I did to the top image.  At worst it still had a considerably weaker and simpler discoloration.  Local Normalization definitely contributes to the problem, as has also been my experience with other images.

I would still like to troubleshoot the original source of the discoloration, of course.

Edited by Michael Covington, 07 June 2022 - 09:35 PM.

[SeymoreStars]

If you updated to PI 1.8.9-1 a new version of PI's WBPP is available with many changes.

 

 

Attached Thumbnails

• 

[SeymoreStars]

Get PI 1.8.9-1 here - https://pixinsight.c...65/#post-112629

[SeymoreStars]

The WBPP shows all steps and results, very handy.

Attached Thumbnails

• 

[Michael Covington]

I am using the new version (with PixInsight 1.8.9-1).  When I said "at least the last 2 versions" I meant today's version and the one just before it.

Edited by Michael Covington, 07 June 2022 - 09:36 PM.

[Jared]

Juan's statement was accurate in 2020 when local normalization had some serious down-sides. I doubt very much he would give the same advice today, which is why the default is to use local normalization coupled with some flavor of SNR-based weighting. If everything works properly (and I haven't had any issues with my data), that should yield the best possible SNR with the simples possible gradients for DBE to address.

 

Off-hand, I don't know what went wrong with your data and the local normalization process. I wish I could offer advice. But I don't agree that the default for local normalization should be "off". At least within PI, I would argue it is now a best practice to incorporate local normalization so that one can use an SNR based weighting to get the best possible SNR. Couple that with good error rejection in integration and you should get the sharpest, cleanest results you can for your data. If it isn't working for you, I get it, but giving general advice to turn it off isn't a good idea. It solves a very real problem--of how to simplify gradients and at the same time evaluate and weight images effectively for integration. Based on my testing (running images both with and without, trying various combinations of image weighting factors such as PSF SNR, eccentricity, FWHM, etc.) it is extremely effective and yields the best overall SNR in the integrated image. If it isn't working for you, turn it off till you solve the problem or turn it off indefinitely as you see fit, but I don't think that is a good general practice for everyone.

 

One thing that jumped out in your original statement, by the way... You said that your flats had an average ADU well below 1,000. I don't know anything about the lossy compression issue you mentioned with your Nikon, but generally speaking having flats at 1,000 ADU would be way, way too low. Not saying that's what's causing your issues with blotchy, uneven backgrounds, but are you sure ADU's that low in your flats are a good idea with your camera? Most people would recommend somewhere around 30,000. With most cameras there is a fairly wide range of acceptable, but 1,000 seems to really be pushing it. I have had cameras where the offset alone was that high. Or am I not understanding something specific to the Nikon?

[Jim Waters]

I have what looks like identical problems as Michael.  Unselecting Local Norm somewhat corrected the issues but I need to do further processing. For me this started with PI 1.8.9-1 and the two recent WBPP releases. 

 

I would like to understand what's going on.

 

EDIT - I see this on an ASI2600MC Pro CMOS.

Edited by Jim Waters, 08 June 2022 - 12:14 AM.

[sharkmelley]

Or am I not understanding something specific to the Nikon?

Yes, there is a nefarious data compression issue specific to Nikon.

[Jared]

Yes, there is a nefarious data compression issue specific to Nikon.

Well, that would certainly be annoying.

[17.5Dob]

Interesting !!! My last two images, processed last week, doing everything exactly as I always have, were a "What The ...." mess. I'll have to go back and see what's different.

[Jared]

I have what looks like identical problems as Michael.  Unselecting Local Norm somewhat corrected the issues but I need to do further processing. For me this started with PI 1.8.9-1 and the two recent WBPP releases. 

 

I would like to understand what's going on.

 

EDIT - I see this on an ASI2600MC Pro CMOS.

I know there was a new star detection algorithm incorporated into 1.8.9-1 and that would affect local normalization. I'm still on 1.8.9 so can't speak to the newer release. You didn't see this on 1.8.9? It just appeared on 1.8.9-1?

[arbit]

I've had this when using registered files output from the "old" local normalization algos before 1.8.9 with the new algo. I think it can also happen if the smoothing is not quite right, but that's done automatically in WBPP.

 

I've switched to NSG for any LN now. IMO, it has far better fine control in terms of visual feedback for reference frames (though todays WBPP 2.4.5 closes the gap somewhat), gradient smoothing for best fit, ROI for PSF evaluation, and sub rejection, including a visual guide.

[Juno18]

Dang!

 

Thank you Michael!

 

Had the strange color gradient on the last image. I even rolled back PI to a previous version (from a disk image) trying to figure out this issue. When I reinstalled 1.8.9, the problem came back.

 

I too was using WBPP with LN checked and using defaults. I tried deselecting "Allow clustered sources" (my image was M92) without any success.

 

On an image previous to the last one (M92), I went ahead and processed the master produced by WBPP with the messed up color patches using an extremely aggressive DBE technique and it cleaned it up enough to process. 

 

 

Just downloaded the newest WBPP script yesterday (2.4.5). Hopefully, I can use LN without this issue.

 

Thanks a ton for this thread!

Edited by Juno18, 08 June 2022 - 06:27 AM.

[Midnight Dan]

One thing that jumped out in your original statement, by the way... You said that your flats had an average ADU well below 1,000. I don't know anything about the lossy compression issue you mentioned with your Nikon, but generally speaking having flats at 1,000 ADU would be way, way too low. 

Just want to mention that my images were looking exactly like Micheal's and my flats were at 25K.  I'm using an ASI071MC Pro camera

 

-Dan

[arbit]

I've had this when using registered files output from the "old" local normalization algos before 1.8.9 with the new algo. I think it can also happen if the smoothing is not quite right, but that's done automatically in WBPP.

 

I've switched to NSG for any LN now. IMO, it has far better fine control in terms of visual feedback for reference frames (though todays WBPP 2.4.5 closes the gap somewhat), gradient smoothing for best fit, ROI for PSF evaluation, and sub rejection, including a visual guide.

Samples. Test stack of 10 subs each with NSG. Left is at low gradient smoothness (-4), and right with higher (-1).

 

Dont know how visible it is compressed, but the left is more patchy. The impact is much more visible in the full stack.

 

The smoothness graphs are also attached.

Attached Thumbnails

• 
• 

[Scott Badger]

This issue came up in a Pixinsight forum thread and Juan Conejero suggested using a best-of integration as the reference and also upping the scale to 768, or 1024. What made the most difference though, was drizzling. FWIW.

 

Cheers,
Scott

[Michael Covington]

Update:  Juan Conejero confirms what a couple of you said, that Local Normalization has been greatly improved since he advised people in 2020 not to use it.

However, it appears that its default parameters are incorrect, so I stand by my opinion that it should be off by default, leaving it to the user to turn it on and make appropriate settings (which I haven't delved into yet; I know nothing).

 

In this thread I only wanted to make one point, which is that turning Local Normalization on by default was not the right move; it surprised people by making their images come out worse.

 

My low ADUs are to avoid a Nikon data compression issue.  The ADU scale on this camera goes to 16383, not 64k, so 1000 is not super-low.

 

Admittedly those discolorations are appearing in *extremely* stretched, faint images.  Their amplitude may be less than 1 ADU (possible because stacking is done with floating-point numbers). 

I have not yet had time to read everything on this thread closely -- just wanted to give a quick update -- will be back when time permits.

[bulrichl]

Update:  Juan Conejero confirms what a couple of you said, that Local Normalization has been greatly improved since he advised people in 2020 not to use it.

However, it appears that its default parameters are incorrect, ...

Did you reset WBPP choosing the option 'Reset all parameters to factory-default values?' when you tried LocalNormalization?

 

If not, you didn't set the default parameters, instead, the settings of your last WBPP run were used.

 

The default parameters are:

Generate images:              disabled
Reference frame generation:   Integration of best frames
Evaluation critera:           PSF Signal Weight
Normalization scale:          512
Scale evaluation method:      PSF flux evaluation
PSF type:                     Auto
Growth factor:                1.00
Maximum stars:                24576
Minimum detection SNR:        40
Allow clustered sources:      enabled
Low clipping level            4.50e-05
High clipping level:          0.85

You can compare whether these values were used.

 

Bernd