786 replies to this topic

[tkottary]

Yes its not 100%,but it's the best correction I had so far for those files . I am aware of the different compression table per  our previous note.

 

I should also tell that my files were in  14 bit lossless compressed mode . 

 

That is odd.  I wouldn't expect it to have any benefit on Z6 files because the Z6 uses a different compression table.

 

Mark 

[sharkmelley]

Just as I thought we were beginning to understand this, something unexpected happened.

 

I had a bunch of Nikon D5300 NEFs which I converted to DNGs using the Adobe DNG Converter.  When opened in any raw convertor these should be identical.  But they are not!

 

Whether I open them in PixInsight or Rawdigger (both of which use LibRaw) or whether I open them in Photoshop/CameraRaw there are significant differences between the NEF and the DNG.  This should not be happening because DNG is supposed to be a digital negative format suitable for archival.

 

Now the original NEF file contains a compression table and it's quite possible that LibRaw and Adobe have a difference in interpretation of that table.   But it's clear that Adobe is being inconsistent because using Adobe Camera Raw to open the NEF or the Adobe DNG file are not giving the same result.  So there is different processing going on in ACR and the DNG Convertor.

 

This discovery came about because I was obtaining unexpected results from my own coding which I thought must be a bug I had introduced.  But it soon became apparent that the problem was more fundamental.

 

Mark

[Mike in Rancho]

Ah, the old monkey in the wrench.  I should have maybe thought of something like that.  Unlike DSS (that I am aware of), ASTAP not only offers several demosaicing algorithms (though that should come later), but also at least 3 raw conversion schemes, including his own modification on dcraw or libraw.  I wonder what those do differently, or if anybody is already attempting to handle lossy artifacting.  Based on our rings, I presume not.

 

But of course the next logical question is, how pristine was the data that your correction table was derived from?  Put another way, is it reliant somehow in whatever method PI used to open the NEF?

 

And if everyone unpacks NEFs in a slightly different way, absent stuffing corrected data back inside a NEF or perhaps a FITS, this might prove rather difficult to do outside of PI where you know it works.

[bobharmony]

I have a stack running in ASTAP with all converted files.  We'll see how that goes in the morning.  Thaks, Han, for the tip on putting *.* in the file name space to get the .tif file to show up for the flats and dark flats.  I am using bias frames for the flat darks... is that OK?

 

Mark, sorry to hear that the files are handled differently by different programs, this is indeed a setback for us DSS users. 

 

Bob

[bobharmony]

Hmmm... I noticed that ASTAP is creating fits files from the tif inputs during the analysis phase.  Inquiring minds want to know if those fits files can be used as input to DSS.  Something to try after looking at the ASTAP stack.

 

Bob

[sharkmelley]

Hmmm... I noticed that ASTAP is creating fits files from the tif inputs during the analysis phase.  Inquiring minds want to know if those fits files can be used as input to DSS.  Something to try after looking at the ASTAP stack.

Try it.  If it converts 16bit gray TIFFs to 16 bit gray FITS then DSS should be able to load them and debayer them.  There may also be other TIFF to FITS convertors out there somewhere (I haven't looked).

 

Meanwhile, I have built both LibRaw and the DNG SDK so out of interest I'll take a look at exactly how they unpack the lossy compressed NEF and DNG files.  

 

By the way, it goes without saying that because the NEF and DNG are different, the ring pattern is also different!

 

Mark

[Mike in Rancho]

Hmmm... I noticed that ASTAP is creating fits files from the tif inputs during the analysis phase.  Inquiring minds want to know if those fits files can be used as input to DSS.  Something to try after looking at the ASTAP stack.

 

Bob

Good idea.  I forgot that ASTAP littered the hard drive with all new fits files lol.

 

I did a bunch of playing around tonight, without much success at first, in both stackers.  I told ASTAP to use Libraw to convert, since that's what Mark used.  I used the *.* loading of tiff flats and dark flats.  I used the ASTAP-converted fits files in DSS.  Pretty much all the same, if not weirder.  Same checkerboards.  Same wrong colors (though the fits did bring color into DSS finally, so a minor achievement there).  Even ended up with some really strange criss-crossing gradient strips in the image, though I can't remember if that was with averaging or kappa sigma.  Maybe both.  In any event, it was all a bust.  Except just like the first night, I knew something seemed either doubled-up or transposed with the colors.  NGC1499 is not a green nebula.  And why are the blue and red histograms perfectly overlapped into...purple!?

 

So I started fiddling with the Bayer matrix pattern and eventually landed on GBRG.

 

Colors were in the right place, checkerboards and criss-cross gradients went away, all seems good.  This worked in both ASTAP and in DSS using ASTAP fits.

 

I believe Mark's ring reduction worked as well, though that will require additional testing to truly prove and quantify it.  I was using very poor data here, really just for quickie registration and to see if the files would stack and calibrate after being run through Mark's app:  Only 30 minutes integration of a faint object like Cal Neb, with a zoom lens at 300mm, wide open f/5.6, meaning massive noise, vignetting, lossy compression rings, and the need for a really big stretch to see anything.  I also used flats that were known to be in a very bad lossy zone, rather than the low and high zone flats I took later.  And, unlike with the original, I was actually able to process it.  Not well lol, but at least it was something.

 

Further testing, on better data but still with known rainbow ring issues, will require a lot of converting and restacking.  But I have high hopes now that it's going to work.  Additional experimentation could probably also be done on DSS and ASTAP conversion and Bayer settings so that we can settle on the absolute best way to use Mark's tiffs.

[bobharmony]

My results match what Mike sees.  ASTAP doesn't handle the monochrome fits files as undebayered and produces the checkerboard pattern.

 

 

DSS does appear to recognize the files as undebayered (CFA parm is set to "Yes") and you can see from the histogram that red and blue overlap while green if off to the left a bit.

 

 

I think I know why this is and why Mikes change in pattern fixes it.  Looking at the two stacked images it is apparent they are mirror images of each other.  We used RGGB as our demosaic pattern.  On the sensor that looks like this:

 

RG

GB

 

The mirror image of that is

 

GR

BG

 

or GRBG in the stacking software.  I am going to rerun the DSS stack with that pattern and am hoping for good things.

 

One other positive thing: I did a meridian flip while capturing this target, but none of my DSS stacks would include the frames from the opposite orientation from the DSS reference frame.  If I used a pre-flip image as reference, only the pre-flip images would be included in the stack.  The same held true for using a post-flip frame as the reference frame - that stack only included post-flip images.  This wasn't my first experience with stacking pre- and post-flip images and I had never seen that behavior before.  The good news - every stack I have done since applying Marks conversion has included every frame - the conversion is having a good effect already even if we haven't gotten to the final product yet.

 

A note about using ASTAP to do the Tiff to Fits, it reads the file name to restore Exif data about ISO, exposure length, etc.  This is a nice to have feature.

 

I will report further after the next DSS stack (and hopefully some post-processing results).

 

Bob

 

Adding - I note the difference between between Mikes new pattern and mine.  I will try both if needed, one or the other should work!

Edited by bobharmony, 05 February 2021 - 08:40 AM.

[CardBoardBoxProcessor]

Sort of no related but maybe alittle. my Z7 had rings due to flats apparently. following an earlier post in tis thread's advice about varied exposure times I recaptured flats with braketing 0.3 stops difference 9 shots per series. then stacked those for the flat. This eliminated my ring problem fully. just leaving this here. It is likely not related to the compression as the D5300 issue is of this thread but I figure others will find this thread such as I did. 

Edited by CardBoardBoxProcessor, 05 February 2021 - 09:11 AM.

[bobharmony]

Success!  The stack with GBRG (Mike's experiment did better than my logic) yielded a very nice image that shows no rings at all!  Also the HH was red instead of blue, as my GRBG version turned out.  The 1/10" flats undercorrected vignetting and dust motes, but they are underexposed, peaking at about 3000 ADU.  I am going to go through the process one more time using the 1/6" second flats and see where that takes things.

 

Thanks, Mark, for going above and beyond - identifying the cause of the issue, developing a fix, adapting it for PI, then a Windows variant that allows the rest of us to take advantage of it.  You are my hero!

 

Bob

[sharkmelley]

I have some great news - version 2.0 is now ready and I've tested it on both Windows 7 and Windows 10. This version works with DNG files, which is better solution because DNG files can be opened by almost everything:  PixInsight, DeepSkyStacker, AstroPixelProcessor, Photoshop/LightRoom, RawTherapee etc. It also means that important header information is preserved e.g. ISO, exposure length, lens data etc.

 

The breakthrough happened when I discovered the DNG file contains the original lossy compressed NEF data.  So it became possible to uncompress it using the same LUT that LibRaw uses and then apply my table of corrections.

 

You will need to download the Adobe DNG Converter which will convert your NEF files to DNG:

https://helpx.adobe....-converter.html

 

Afterwards run my application to remove the D5300 rings.

 

The interface is the same as before:

 

 

 

The executable is larger because of the larger size of the DNG library which means the zip is too large to be uploaded here.  It's therefore on my Google Drive:

https://drive.google...I4b2zej3fXZYjKG

 

Be sure to read the ReadMe file for instructions and other information.

 

I'm happy to supply the project files on request to anyone who wishes to examine the code or build it for themselves or develop it further.  Please send me a PM.

 

The application should work for all Nikon cameras that use the same lossy compression as the D5300, as long as 14bit lossy compression mode was used:

D5100/D5200/D5300
D500
D600/D610
D7000/D7100/D7500
D3S/D3X
D4
D800/D800E/D850

 

The application does not check that DNG files are lossy compressed; does not check that the DNG files are 14bit and does not check the camera uses the same compression table as the D5300. In each case the application will still apply the correction table to the data, which will have the unfortunate effect of introducing rings where previously none existed.

 

Mark

[bobharmony]

I will download v2 and give it a shot, Mark.  It'll be a quicker process than converting the tiff files to fits in ASTAP!  I like that it preserves all the tags!

 

I did a quick process on the stacked image with the 1/6" flats.  It is way better than where I started with this.  It preserves all the color and detail of the original without the artifacts from the rings.  Still a little vignetting and flat correct still isn't perfect, this was my first outing with the D5300 and I haven't got all that down quite yet.

 

 

Two more things to note here - the yellow glow that I thought was light leak from a local streetlight is completely gone.  It must have been an effect of the rings.  Second, there were several satellite trails in the lights.  With the rings in place they were not eliminated from the stacked image with KS clipping and I had to remove them from the stack to keep them from showing up.  The stack I did with the converted files includes those pesky trails - and there is not a sign of them anywhere!

 

This process has made the purchase of the modded D5300 well worth it for me.  I can do my standard processing and get results that are pleasing.  When I get the final version of this done, I will put it in a new thread in the proper forum, and share a link to this topic.  A lot of people are going to be able to benefit from the work Mark has put into this.  Thanks you, thank you, thank you!

 

Bob

[sharkmelley]

For the D5300 (plus my earlier list of supported cameras), if you use my ring removal application then you can ignore all my previous advice for the D5300 about exposing lights so the back-of-camera histogram peak is less than 1/4 from the left and exposing lights so they are as far to the right as possible.

 

Now let's talk about the D5500 and D5600.  They use a different compression table which is much less of a problem than the D5300. Any lights or flats shot with the back-of-camera histogram peak further to the right than the 1/3 point will potentially contain a big donut because the first histogram gap appears at a raw data value of 1552 but multiple rings don't become a problem until the histogram goes beyond the halfway point. So in practice, only the flats will be a problem on the D5500/D5600.

 

However, it's quite feasible for me to add a D5500/D5600 correction table into my application, if there is any interest.  The app would be able to recognise the camera from the header and apply the relevant correction table.  But first I would need to generate the correction table. 

 

Mark

Edited by sharkmelley, 06 February 2021 - 05:38 AM.

[bobharmony]

A quick question, Mark.  When running the Adobe DNG, I assume I need to check the "Embed original RAW file" checkbox in the Preferences section.  The way I read your description, that original is opened by your conversion program to have the fix applied.

 

I am going ahead with that assumption at the moment.  If it needs to be set differently, I will go back and do that.

 

Bob

[bobharmony]

I went through the process with the Adobe DNG tool as the first step, then De-ring tool v2, DSS stacking and StarTools post-processing.  Results:

 

 

I think the detail is sharper here and the star shapes (despite my focusing and tilt issues) look better.  I messed with the vignetting option in StarTools and overdid just a bit.  I am going to go back and re-process leaving that step out.

 

Note that with the DNG option there is no need to change the demosaic pattern.  Luckily I thought of that before stacking and set it to RGGB in DSS, otherwise I would be stacking again right now.

 

Bob

 

PS - here is the color result from the Wipe module.  Much cleaner than the one I posted earlier in this thread!  This before I played with vignetting adjustments.

 

Edited by bobharmony, 06 February 2021 - 12:57 PM.

[sharkmelley]

A quick question, Mark.  When running the Adobe DNG, I assume I need to check the "Embed original RAW file" checkbox in the Preferences section.  The way I read your description, that original is opened by your conversion program to have the fix applied.

No, there's no need to "Embed original RAW file".  That's not how I was using the DNG Converter.

 

Mark

[limeyx]

Thanks Mark, fantastic work as I just sent my D5300 out to be modded since the budget for an astro-cam is not there yet. Hoping I dont need this with the scope instead of lens but it's awesome to have it as an option

 

I may run some older data through it since it's going to be cloudy for approx 187 days here

[han.k]

I have a stack running in ASTAP with all converted files.  We'll see how that goes in the morning.  Thaks, Han, for the tip on putting *.* in the file name space to get the .tif file to show up for the flats and dark flats.  I am using bias frames for the flat darks... is that OK?

Bias and flat-darks are practical the same. The read noise is only significant, not the dark current of a few seconds.

 

Han

[sharkmelley]

I've now managed to produce correction tables for the D5300 12bit and for the D5500 14bit.  I've plotted them below to show the similarities as well as the differences.  The origin on the D5500 curve has been shifted to the left to line it up with the D5300 curve.  The D5300 12bit curve has been scaled up by a factor of 4 both vertically and horizontally.

 

 

These correction curves will allow me to treat the following lossy compression file types in addition:

• D5300 12 bit
• D500/D600/D610/D7000/D7100/D7500 12 bit 
• D3S/D3X/D4/D800/D800E/D850/Df 12 bit
• D5500/D5600 14 bit
• D3100/D3200/D3300
• D5000
• 1 J1
• 1 V1

Obviously, where possible lossless compression should always be used when available.

 

Mark

 

[Andy Lucy]

Mark,

 

Really great work.

 

Could you comment on the position with regards to concentric rings when lossless compressed or uncompressed files are used? 

 

I have a Nikon Z50, which does not offer any compression options, and the manual does not describe the one option that is used.  Fortunately, Iliah Borg and Bill Claff have determined that the files are always lossless uncompressed (see this thread on dpreview):  https://www.dpreview...s/post/63743192

 

I have seen some evidence of Z50 concentric rings in my images and they have been reported here:  https://www.cloudyni...ing/?p=10158668

 

Regards,

 

Andy

[sharkmelley]

Could you comment on the position with regards to concentric rings when lossless compressed or uncompressed files are used? 

 

I have a Nikon Z50, which does not offer any compression options, and the manual does not describe the one option that is used.  Fortunately, Iliah Borg and Bill Claff have determined that the files are always lossless uncompressed (see this thread on dpreview):  https://www.dpreview...s/post/63743192

 

I have seen some evidence of Z50 concentric rings in my images and they have been reported here:  https://www.cloudyni...ing/?p=10158668

The concentric rings discussed in that thread are unrelated to lossy compression.  They are caused by another form of raw data scaling which I think is probably some kind of colour shading correction.  Unfortunately it cannot be disabled and it wouldn't surprise me if all Nikon mirrorless cameras are affected.

 

Mark

Edited by sharkmelley, 07 February 2021 - 09:54 AM.

[Steve OK]

Mark, I just want to add my heartfelt "Thank You!" for doing this.  You are the epitome of what makes Cloudy Nights such a great community.

 

Steve

[bobharmony]

One last note - I rolled through the process again, this time not embedding the original .nef data inside the .dng file during the Adobe DNG Converter step.  If anything, the data is a little easier to use and shows better retention of the dimmer dusty detail in the area.  It is also possible that after going through processing this data over 40 times, I may finally be hitting the settings in StarTools that do it the most justice.

 

Bob

[sharkmelley]

Version 3.0 for Windows can be downloaded from here:

https://drive.google...Z8TBatLUO_Xs17a

 

Again this version works with DNG files so you will first need to run Adobe DNG Converter to create the DNG files.  I have now removed the reference to the Nikon D5300 because this version has 3 inbuilt correction tables which means it is compatible with many more cameras.

 

The application should now work with the following lossy compressed file types:

D5100/D5200/D5300  14-bit and 12-bit
D500               14-bit and 12-bit
D600/D610          14-bit and 12-bit
D7000/D7100/D7500  14-bit and 12-bit
D3S/D3X            14-bit and 12-bit
D4                 14-bit and 12-bit
D800/D800E/D850/Df 14-bit and 12-bit

D5500/D5600        14-bit
D7200              14-bit
D750/D810          14-bit

D3100/D3200/D3300  12-bit
D5000              12-bit
1 V1/1 J1          12-bit

 

Be sure to read the ReadMe file for instructions and other information.  Let me know if you experience any problems.

 

Mark

[Mike in Rancho]

Over the past few days I had a chance to experiment with the conversions a bit, first v1.0 on some old data, then v2.0 on some new Monkey Head data I just took, both with good results!  I will have to try out v3.0 now.

 

One thing - Adobe's DNG converter is so cool that I forgot to then make new subfolders and run the ring removal converter on it!  I really don't want to admit how many stacks and restacks I examined, looking to find a difference.  Extra thanks to Mark for adding "fix" to the filenames (which you find out once you remember to actually run ring removal!).  There are a lot of extra files here, but well worth it.

 

As a concrete example of what is being helped out here, I loaded both an original Monkey Head NEF stack and a v2.0 ring removed stack into Startools.  The Wipe module displays a fairly strong temporary stretch to reveal noise and flaws.  I took each just barely past the default gradient settings to the same point on the sliders, made screenshots, and did a subtract in Gimp to see the difference.  This is the junk that's being removed:

 

 

At only 2 hours of integration, this is pretty noisy data to begin with, so some of the ringing would have been squashed out anyway just to deal with that.  It sure is nice though to only have to worry about routine gradients and light pollution!  I can see this being super useful with better data, where the ring removal will allow the stretching out of fine detail.  Likewise, the red rings (for me often a large portion of the center) were nearly impossible to differentiate from red nebulosity.  I will not miss those.

 

Just for fun, here was a quickie full processing of my 2-hour MH after Mark's ring remover 2.0.

 

 

Big thanks one more time.