Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Nikon D5300 Flats Problem - Black Clipping Related?

  • Please log in to reply
237 replies to this topic

#1 Jerry Lodriguss

Jerry Lodriguss

    Vendor

  • *****
  • Vendors
  • topic starter
  • Posts: 7,533
  • Joined: 19 Jul 2008
  • Loc: Voorhees, NJ

Posted 17 November 2016 - 07:38 AM

I'm shooting some faint dust in Taurus with a Nikon 180mm f/2.8 lens at f/2.8 on a D5300. The lens has some vignetting wide open.

 

I shot some blue-sky flats during daytime.

 

I tried calibrating the lights with darks, flats and bias in both Images Plus and DeepSkyStacker.

 

The weird thing is the flat calibration is completely different in each - see the attached files.  The DSS stack has a large red halo. The IP stack has a darker center with brighter corners.  

 

These are stretched, sky background neutralized, and then hit with the "equalize" command in Photoshop, which ridiculously exaggerates the contrast, but shows the problems.

 

I've tried calibrating without the darks, and also checking on and off "Set Black Point to Zero" in DSS.

 

The flats are not clipped in any of the channels.

 

All lights and calibration frames are shot at ISO 200 so it's not an ISO mismatch in DSS.

 

In the original stack I have 117 x 4min Lights, 6 darks (ambient temps were very cold), 64 flats and 64 bias.

 

I have put 3 frames each of the lights, darks, flats, and bias here in DropBox:

 

https://www.dropbox....OpfnK_gafa?dl=0

 

I'm only posting 3 of each because it goes a lot quicker if anyone wants to try it and it shows the same problem as the full set.

 

I repeated the tests with blue-sky t-shirt flats and it didn't make any difference, the problem was still there.

 

I also ran a set with the 180mm lens and a Canon and didn't see anything like this.

 

I'm wondering if this could be something related to the black clipping in the Nikons?

 

Any suggested solutions would be greatly appreciated.

 

Jerry

Attached Thumbnails

  • DSS Stack Set BP to Zero OFF Circular Artifact.jpg
  • IP Bad Corners.jpg

Edited by Jerry Lodriguss, 17 November 2016 - 08:03 AM.

  • calypsob likes this

#2 entilza

entilza

    Soyuz

  • *****
  • Posts: 3,785
  • Joined: 06 Oct 2014
  • Loc: Oakville, ON, Canada

Posted 17 November 2016 - 08:57 AM

Hey Jerry, I just stacked these in DSS and PI and both performed perfectly.  So that makes me wonder what setting could be incorrect on yours.

 

I will upload the processed files shortly.

 

For DSS:  I have Set black pt to zero checked.  White balance:  nothing checked.    Bayer Matrix:  Bilinear

 

For PixInsight:  I just used the BatchPreProcessing default options.

 

DSS.jpg

 

PI.jpg

 

Let me know if you'd like me to check anything else.


Edited by entilza, 17 November 2016 - 09:06 AM.

  • BrianThePrimate likes this

#3 Daniel Dance

Daniel Dance

    Eastern Voltage Research, LLC

  • *****
  • Vendors
  • Posts: 1,736
  • Joined: 08 Jun 2016
  • Loc: Eastern United States

Posted 17 November 2016 - 09:15 AM

I just did a quick processing in PixInsight.  No issues here other than those Nikon files take forever to process compared to my Canon 6D files.  There are some gradients, but nothing that can't be fixed.

 

Maybe its time to jump on the PixInsight bandwagon.  :blush:

 

31054276345_62fddcd2e7_b.jpgjerry01 by Daniel McCauley, on Flickr


Edited by Daniel Dance, 17 November 2016 - 09:18 AM.


#4 Herra Kuulapaa

Herra Kuulapaa

    Viking 1

  • -----
  • Posts: 514
  • Joined: 10 Dec 2014
  • Loc: Vantaa, Finland

Posted 17 November 2016 - 09:27 AM


I'm wondering if this could be something related to the black clipping in the Nikons?
 

Don't mind this Jerry: Just user related ;)

Ps. I'm kidding you..

Jerry.jpg

[This was manually calibrated with Pixinsight]


Edited by Herra Kuulapaa, 17 November 2016 - 09:33 AM.


#5 smokey688i

smokey688i

    Lift Off

  • *****
  • Posts: 10
  • Joined: 18 Oct 2014

Posted 17 November 2016 - 10:24 AM

Very curious what the conclusion is here, I've had transient issues with that same red halo on the edge/green center tint in my DSS stacks that I have yet to get to the bottom of... I most recently have been exploring ways to better match dark and light frame sensor temp, but seeing as it appears to be a settings problem I'm in part relieved.

I'm wondering if it might be a bug in DSS, perhaps solved by reinstalling...

 

I use a modded D5100 with the true dark current hack applied.

Chris


Edited by smokey688i, 17 November 2016 - 10:27 AM.


#6 Traveler

Traveler

    Soyuz

  • *****
  • Posts: 3,571
  • Joined: 19 Aug 2007
  • Loc: The Netherlands

Posted 17 November 2016 - 11:30 AM

Hi Jerry,

I am thinking seriously to buy a Nkon D5500 or the newer D5600 therefor i have some questions if i may. Feel free to answer them if you like.

 

This "black clipping in the Nikons", did anybody reported such a thing with a Nikon D5300/D5500/D5600 before?

Is this your first "test" with a Nikon D5300?

What makes you think it is something related with black clipping and not with DSS or even user related?



#7 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 21,494
  • Joined: 27 Oct 2014

Posted 17 November 2016 - 01:04 PM

Hi Jerry,

I am thinking seriously to buy a Nkon D5500 or the newer D5600 therefor i have some questions if i may. Feel free to answer them if you like.

 

This "black clipping in the Nikons", did anybody reported such a thing with a Nikon D5300/D5500/D5600 before?

Is this your first "test" with a Nikon D5300?

What makes you think it is something related with black clipping and not with DSS or even user related?

I'll jump in, I've done a lot of research on this, own a D5500.  Nikons used to set the black point to zero (Canons had it at 600 ADU).  That chopped off the whole lower half of the bias histogram.  Which made camera calibration wonky.

 

But the D5300 has it set at 600 ADU, just like a Canon.  Bias histogram is fine.  Lights are fine.  Darks can be an issue at long exposures and higher temperature, the histogram drops off at 480 ADU.  I've seen a competent knowledgable imager say that's really not an issue.  With the low thermal noise, I don't do darks much.

 

@Jerry.  Can't help, I've never seen this, either in DSS or PixInsight.


Edited by bobzeq25, 17 November 2016 - 01:06 PM.


#8 Jerry Lodriguss

Jerry Lodriguss

    Vendor

  • *****
  • Vendors
  • topic starter
  • Posts: 7,533
  • Joined: 19 Jul 2008
  • Loc: Voorhees, NJ

Posted 17 November 2016 - 01:10 PM

Hi Martin, Dan and Herra,

 

Thank you very much for looking at the data.

 

I guess I have a real mystery now.

 

Martin, I used those same parameters in DSS.

 

I don't have PixInsight, but getting it doesn't seem to be a solution since I have problems in both DSS and Images Plus, and I'm an experienced user with both of those programs.

 

I'm tempted to say that makes it unlikely that it is user error, but maybe it is.

 

I have to say I'm baffled now.

 

Jerry



#9 entilza

entilza

    Soyuz

  • *****
  • Posts: 3,785
  • Joined: 06 Oct 2014
  • Loc: Oakville, ON, Canada

Posted 17 November 2016 - 01:28 PM

Jerry the good news a reproducible error is usually the easiest to corrected.

 

Let's focus with DSS as I do not have the other program.  Are you using DSS 3.3.4?

 

Have you tried clearing your %temp% folder?  (You can see your temp folder in the stacking parameters tab)



#10 Jerry Lodriguss

Jerry Lodriguss

    Vendor

  • *****
  • Vendors
  • topic starter
  • Posts: 7,533
  • Joined: 19 Jul 2008
  • Loc: Voorhees, NJ

Posted 17 November 2016 - 03:34 PM

Jerry the good news a reproducible error is usually the easiest to corrected.

 

Let's focus with DSS as I do not have the other program.  Are you using DSS 3.3.4?

 

Have you tried clearing your %temp% folder?  (You can see your temp folder in the stacking parameters tab)

Yes, 3.3.4 and temp folder is clean.

 

I also uninstalled it, deleted the registry file and reinstalled it.

 

Jerry



#11 entilza

entilza

    Soyuz

  • *****
  • Posts: 3,785
  • Joined: 06 Oct 2014
  • Loc: Oakville, ON, Canada

Posted 17 November 2016 - 03:44 PM

For the weighting algorithm can you try kappa sigma? I used to get errors integrating the larger files. You have experience with integrating D810A images too which are even larger. So I am not sure.

Do you happen to have another computer to try this stack test on?

Also are you using just the sample 3 files you shared to rule out some other bad calibration files?

#12 Jerry Lodriguss

Jerry Lodriguss

    Vendor

  • *****
  • Vendors
  • topic starter
  • Posts: 7,533
  • Joined: 19 Jul 2008
  • Loc: Voorhees, NJ

Posted 17 November 2016 - 04:10 PM

Ok, just figured out why it was producing artifacts in Image Plus...

 

It was having "DSLR Automatic Dark Frame Matching" turned on.

 

I still think this is a black clipping issue because the blacks are clipped in the bias frames and bias is needed to scale the darks.

 

Now to dig around in DSS and see what would be similar... dark optimization is off by default, so it's not that... let me try it with that on... I'll report back

 

Jerry



#13 Jerry Lodriguss

Jerry Lodriguss

    Vendor

  • *****
  • Vendors
  • topic starter
  • Posts: 7,533
  • Joined: 19 Jul 2008
  • Loc: Voorhees, NJ

Posted 17 November 2016 - 04:16 PM

It looks like turning "Dark Optimization" on solved the problem in DSS.

 

Note also that I used "Camera WB" and unchecked "Set Black Point to Zero", which are the settings I recommend for unmodified cameras.

 

Thanks to everyone who helped and tried to help.

 

Jerry



#14 Traveler

Traveler

    Soyuz

  • *****
  • Posts: 3,571
  • Joined: 19 Aug 2007
  • Loc: The Netherlands

Posted 17 November 2016 - 04:24 PM

So to make sure nobody thinks there is an issue with the Nikon D5300:

 

What you had is not an issue with Nikon Black Clipping.



#15 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 21,494
  • Joined: 27 Oct 2014

Posted 17 November 2016 - 04:28 PM

Ok, just figured out why it was producing artifacts in Image Plus...

 

It was having "DSLR Automatic Dark Frame Matching" turned on.

 

I still think this is a black clipping issue because the blacks are clipped in the bias frames and bias is needed to scale the darks.

 

Now to dig around in DSS and see what would be similar... dark optimization is off by default, so it's not that... let me try it with that on... I'll report back

 

Jerry

The bias frames are not clipped, at all.  Mine go from 590 to 610 ADU, in a reasonably symmetric curve.  Nowhere near the clipping point of 480.  If they are in your case, something is wrong.

 

On Nikons earlier than the 5300, with the black point at 0 ADU, rather than 600, it's a different story.


Edited by bobzeq25, 17 November 2016 - 04:32 PM.


#16 entilza

entilza

    Soyuz

  • *****
  • Posts: 3,785
  • Joined: 06 Oct 2014
  • Loc: Oakville, ON, Canada

Posted 17 November 2016 - 04:49 PM

The mean is definitely at 600 on your Bias Jerry, I checked that at the start.

 

My DSS stack did not have Dark optimization on during my stack.

 

I just tried DSS with a variety of options and it worked in all cases with your 3 sample set.. So I hate to say this, but that does not seem to be the issue.

 

I can't reproduce the failed effect.

 

However I am glad yours is working now and it looks to be a fine image by the way :)


Edited by entilza, 17 November 2016 - 04:50 PM.


#17 Jerry Lodriguss

Jerry Lodriguss

    Vendor

  • *****
  • Vendors
  • topic starter
  • Posts: 7,533
  • Joined: 19 Jul 2008
  • Loc: Voorhees, NJ

Posted 17 November 2016 - 05:31 PM

The mean is definitely at 600 on your Bias Jerry, I checked that at the start.

 

My DSS stack did not have Dark optimization on during my stack.

 

I just tried DSS with a variety of options and it worked in all cases with your 3 sample set.. So I hate to say this, but that does not seem to be the issue.

 

I can't reproduce the failed effect.

 

However I am glad yours is working now and it looks to be a fine image by the way :)

Well, this gets stranger and stranger.  Maybe it's because I'm operating on no sleep.

 

But now the problem is back in Images Plus.  I think the one that I thought didn't have the problem was done without any flat frames at all because there wasn't a master flat in the directory when I just went to it.

 

And here's something interesting... I just realized I'm using 3.3.2 now because when I uninstalled it, I forgot to copy 3.3.4 to the DSS directory to overwrite the older version.

 

Maybe it's time to go get some sleep...

 

Jerry



#18 Jerry Lodriguss

Jerry Lodriguss

    Vendor

  • *****
  • Vendors
  • topic starter
  • Posts: 7,533
  • Joined: 19 Jul 2008
  • Loc: Voorhees, NJ

Posted 17 November 2016 - 05:34 PM

So to make sure nobody thinks there is an issue with the Nikon D5300:

 

What you had is not an issue with Nikon Black Clipping.

I'm not sure.

 

The reason I say this is I sent the calibrated images that showed the problem to Wei-Hao and he said that was exactly the problems he was having with his D800 and it was from the black clipping issue.

 

Bob, do you use calibration frames? So you might not have run into this problem?

 

On the other hand, PixInsight seems to handle these files ok.

 

Maybe PixInsight is adding an offset?

 

Too tired to think straight right now.

 

Jerry


Edited by Jerry Lodriguss, 17 November 2016 - 05:37 PM.


#19 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 21,494
  • Joined: 27 Oct 2014

Posted 17 November 2016 - 06:19 PM

 

So to make sure nobody thinks there is an issue with the Nikon D5300:

 

What you had is not an issue with Nikon Black Clipping.

I'm not sure.

 

The reason I say this is I sent the calibrated images that showed the problem to Wei-Hao and he said that was exactly the problems he was having with his D800 and it was from the black clipping issue.

 

Bob, do you use calibration frames? So you might not have run into this problem?

 

On the other hand, PixInsight seems to handle these files ok.

 

Maybe PixInsight is adding an offset?

 

Too tired to think straight right now.

 

Jerry

 

The D800 probably has the black point at 0 ADU.  I think the full frame changeover to 600 ADU happened with the 810.

 

I use bias and flats, always.  They never clip.  Very occasionally darks.  The last time was 120 seconds and 10C (just checking them out).  They didn't clip, the histogram looked like very slightly noisier bias, the boosted autostretch in PixInsight looked like bias (no fixed pattern noise) and there was no noticeable impact on the final (which is why I don't do darks).  I think I got some clipping with darks at 10 minutes at 20C, when the camera was new and I was checking it out.

 

I don't think your issue is black point clipping.


Edited by bobzeq25, 17 November 2016 - 06:32 PM.

  • sharkmelley likes this

#20 sharkmelley

sharkmelley

    Soyuz

  • *****
  • Posts: 3,756
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 18 November 2016 - 02:01 AM

I've had under or overcorrection problems like this in the past where I have flats and master flats lying around and fed the wrong one or wrong ones into the software.  So the bias ends up being either double subtracted from the flat or not subtracted at all.

 

For instance, I think that if you give DSS just a single flat then it assumes it is a master flat with bias already subtracted.

 

Mark



#21 Jerry Lodriguss

Jerry Lodriguss

    Vendor

  • *****
  • Vendors
  • topic starter
  • Posts: 7,533
  • Joined: 19 Jul 2008
  • Loc: Voorhees, NJ

Posted 18 November 2016 - 08:08 AM

I've had under or overcorrection problems like this in the past where I have flats and master flats lying around and fed the wrong one or wrong ones into the software.  So the bias ends up being either double subtracted from the flat or not subtracted at all.

 

For instance, I think that if you give DSS just a single flat then it assumes it is a master flat with bias already subtracted.

 

Mark

Hi Mark,

 

It's definitely not that.

 

I'm definitely giving it 3 separate lights, darks, flats and bias every time.

 

I delete the masters and re-start the program every time.

 

Jerry



#22 starbob1

starbob1

    Vanguard

  • *****
  • Posts: 2,074
  • Joined: 11 Mar 2007
  • Loc: IN

Posted 18 November 2016 - 08:20 AM

I had the same problem also using My 5300 with Images Plus. I posted on this issues when I first got my 5300. I been using Canons and Images Plus forever and the Nikon just does not play well with Images Plus. I finally had to over exposed my flats more than what I use to do with my canon flats.



#23 smokey688i

smokey688i

    Lift Off

  • *****
  • Posts: 10
  • Joined: 18 Oct 2014

Posted 18 November 2016 - 09:13 AM

Jerry,

I also tried stacking your sample files with several of my typical configurations in DSS, and was unable to cause the green/red colored vignette pattern.

I do have an idea...Can you upload the dark/bias/flat master tiff files that your DSS is creating from the 3 sub sample set? I have a hunch that DSS is incorrectly applying or calculating the master flat, perhaps by treating the RGB channels as grayscale or vice versa...the master flat I came up with when I ran your test set of images had a blue tint to it, but in the past when I had similar issues I do recall some of my master flats coming up looking grayscale, when the original subs looked reddish... I'll look more into this this evening.

Chris

#24 entilza

entilza

    Soyuz

  • *****
  • Posts: 3,785
  • Joined: 06 Oct 2014
  • Loc: Oakville, ON, Canada

Posted 18 November 2016 - 09:23 AM

I just setup DSS 3.3.4 on a clean virtualbox computer and did not change a single setting from the defaults after installation and it worked.  Just to create a control test base experiment.

 

So now the question is what is causing this on Jerry's computer...   I will try DSS 3.3.2 just for kicks if it will even load the newer NEF's.

 

EDIT:  Surprisingly DSS 3.3.2 also worked with these files.   The only setting that needs to be changed for this test is the default stacking % from 80% to 100% stack all 3.

 

Did you try deleting all the text files?


Edited by entilza, 18 November 2016 - 10:27 AM.


#25 Jerry Lodriguss

Jerry Lodriguss

    Vendor

  • *****
  • Vendors
  • topic starter
  • Posts: 7,533
  • Joined: 19 Jul 2008
  • Loc: Voorhees, NJ

Posted 18 November 2016 - 03:02 PM

I had the same problem also using My 5300 with Images Plus. I posted on this issues when I first got my 5300. I been using Canons and Images Plus forever and the Nikon just does not play well with Images Plus. I finally had to over exposed my flats more than what I use to do with my canon flats.

Hi Starbob,

 

I can't say it is good to have company on this, and I guess it still doesn't rule out operator error, although at least now I know I'm not totally crazy.   Or maybe I am. :-)

 

Jerry




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics