Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Flats - D5300 vs PixInsight

  • Please log in to reply
16 replies to this topic

#1 Hypoxic

Hypoxic

    Viking 1

  • *****
  • topic starter
  • Posts: 859
  • Joined: 24 Dec 2011
  • Loc: Oise, France

Posted 12 April 2021 - 02:14 AM

So, I took flats last and in doing so I put the histogram on my Nikon D5300 just a hair over 50%

 

When I opened one of those flats in PixInsight this morning, the Histogram is telling me a different story. How do I see an ADU value here? I'd like to see the actual number associated with what I'm seeing so I know how to adjust my settings the next time I'm out. I thought I exposed my flats correctly, but It looks under exposed here in PI. Or am I just reading this histogram incorrectly? I expected to see something close to what I saw on the Nikon last night.

 

I'm in the process of trying to shoot my flats correctly so I'm altering my old routine, which was erroneous. 

 

Here is one RAW flat from last night.

Attached Thumbnails

  • FlatExposure.jpg

Edited by Hypoxic, 12 April 2021 - 10:43 AM.


#2 imtl

imtl

    Gemini

  • *****
  • Posts: 3,487
  • Joined: 07 Jun 2016
  • Loc: Down in a hole

Posted 12 April 2021 - 02:29 AM

Untick ''normalized''
  • Hypoxic likes this

#3 sharkmelley

sharkmelley

    Aurora

  • *****
  • Posts: 4,623
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 12 April 2021 - 04:03 AM

To see the actual ADU values then change the presentation in the Statistics window from "Normalized Real [0,1]" to "16-bit [0,65535]"

 

DSLR raw data values are in the range 0-16384 so they will always be in the bottom quarter of the PixInsight histogram.

 

If you are exposing your flats so the BoC (back-of-camera) histogram peak is halfway then you will be underexposed because the BoC histogram is a histogram of the stretched data, so you need your BoC histogram peak over to the right instead.

 

Mark



#4 Hypoxic

Hypoxic

    Viking 1

  • *****
  • topic starter
  • Posts: 859
  • Joined: 24 Dec 2011
  • Loc: Oise, France

Posted 12 April 2021 - 04:34 AM

To see the actual ADU values then change the presentation in the Statistics window from "Normalized Real [0,1]" to "16-bit [0,65535]"

 

DSLR raw data values are in the range 0-16384 so they will always be in the bottom quarter of the PixInsight histogram.

 

If you are exposing your flats so the BoC (back-of-camera) histogram peak is halfway then you will be underexposed because the BoC histogram is a histogram of the stretched data, so you need your BoC histogram peak over to the right instead.

 

Mark

So for a proper flat, I need to expose for the center of the PI histogram window, ok I'm going to have to play with that a bit.

 

Also, for the ADU value am I just looking at the X axis in the histogram? The Mean value in Statistics?

 

edit: Wow, they look pure white now.


Edited by Hypoxic, 12 April 2021 - 04:42 AM.


#5 sharkmelley

sharkmelley

    Aurora

  • *****
  • Posts: 4,623
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 12 April 2021 - 04:55 AM

The mean value in the statistics window is a good way of doing it.  Aim for 8000, which is half the saturation level of 16384.  But on the BoC histogram it will definitely look quite overexposed.

 

Mark


  • Hypoxic likes this

#6 Hypoxic

Hypoxic

    Viking 1

  • *****
  • topic starter
  • Posts: 859
  • Joined: 24 Dec 2011
  • Loc: Oise, France

Posted 12 April 2021 - 05:01 AM

The mean value in the statistics window is a good way of doing it.  Aim for 8000, which is half the saturation level of 16384.  But on the BoC histogram it will definitely look quite overexposed.

 

Mark

This is very helpful, thank you. 

Going to play with this a bit and find my new camera settings. This one was too much and totally clipped.

Attached Thumbnails

  • Flat.jpg

Edited by Hypoxic, 12 April 2021 - 12:50 PM.


#7 Hypoxic

Hypoxic

    Viking 1

  • *****
  • topic starter
  • Posts: 859
  • Joined: 24 Dec 2011
  • Loc: Oise, France

Posted 12 April 2021 - 08:44 AM

What I've found with my D5300 is that the pink window that I've highlighted in this image represents the min/max range of my camera. So if I land the histogram in the middle of this range I have a mean value of about 8000.

Attached Thumbnails

  • Range.jpg

  • sharkmelley and jonnybravo0311 like this

#8 Hypoxic

Hypoxic

    Viking 1

  • *****
  • topic starter
  • Posts: 859
  • Joined: 24 Dec 2011
  • Loc: Oise, France

Posted 12 April 2021 - 09:03 AM

Closest I can get:

Attached Thumbnails

  • Mean 7491.jpg


#9 jonnybravo0311

jonnybravo0311

    Surveyor 1

  • *****
  • Posts: 1,573
  • Joined: 05 Nov 2020
  • Loc: NJ, US

Posted 12 April 2021 - 09:19 AM

That's absolutely correct, and is a realization I just had a few days ago. Mark stated it above, but it's a subtlety that is not intuitive. The range of numbers PI shows is based on your camera's ADU, but it's presented in the 16 bit scale.

 

In other words, if you've got a 12 bit camera, the data will only go from 0-4095. However, PI displays the 0-4095 on a 16 bit scale (0-65535). That's why you'll never get an ADU value of 32k from your D5300 in PI. It's why I never got it with my Lumix G9. As you discovered with your D5300, completely clipping the flat will get you a min/max value of 16383 on the PI histogram and is what's shown in PI stats. Therefore, you need to get your flats for the D5300 somewhere around the 8k ADU target - which you did.

 

This was one of those things that was irking me for months, and it wasn't until I read a post the other day on the PI forums (linked from another thread here) that it finally clicked :)


  • cybermayberry, Hypoxic and sharkmelley like this

#10 sbharrat

sbharrat

    Viking 1

  • -----
  • Posts: 529
  • Joined: 28 Nov 2020
  • Loc: NJ, USA

Posted 12 April 2021 - 11:33 AM

That's absolutely correct, and is a realization I just had a few days ago. Mark stated it above, but it's a subtlety that is not intuitive. The range of numbers PI shows is based on your camera's ADU, but it's presented in the 16 bit scale.

 

In other words, if you've got a 12 bit camera, the data will only go from 0-4095. However, PI displays the 0-4095 on a 16 bit scale (0-65535). That's why you'll never get an ADU value of 32k from your D5300 in PI. It's why I never got it with my Lumix G9. As you discovered with your D5300, completely clipping the flat will get you a min/max value of 16383 on the PI histogram and is what's shown in PI stats. Therefore, you need to get your flats for the D5300 somewhere around the 8k ADU target - which you did.

 

This was one of those things that was irking me for months, and it wasn't until I read a post the other day on the PI forums (linked from another thread here) that it finally clicked smile.gif

Ok. Now I am confused. I have a Canon 1000d which is 12bit. When I drag a 0.1s flat into PI, it debayers it (my current setting) so I then extra the lum. The image statics on that shows:

post.jpg

 

If what you wrote above is general, then I should never see a value this high. Is it already "fixed up" as part of the debayering? 



#11 jonnybravo0311

jonnybravo0311

    Surveyor 1

  • *****
  • Posts: 1,573
  • Joined: 05 Nov 2020
  • Loc: NJ, US

Posted 12 April 2021 - 12:05 PM

You're debayering, which is altering the raw data. I probably should have clarified, that it is with pure RAW settings where the issue is presented. You had that whole thread a while back where Dynan stated to both you and me that our flats were sorely underexposed. I ran a bunch of tests. John (jdupton) ran a bunch of tests and we all just kind of ended up scratching our heads in confusion. The part of the puzzle we were missing was that PI is presenting the raw data (in my case, 12 bit numbers) on a 16 bit scale. Therefore, my stats could NEVER give me a 25k to 35k ADU value for my flats. Consequently, my histogram could NEVER show a nicely centered value.

 

So, if you are importing a RAW file from some camera and your PI settings are set to "pure raw", then you need to consider the bit depth of your camera's ADC when looking at the statistics / histogram in PI. In other words, with my 12 bit Lumix G9 and your 12 bit 1000d, a "properly" exposed flat would have a mean ADU of around 2k.

 

What this typically means is your back-of-camera histogram would show you are 2-3 stops over-exposed for a "properly exposed" flat.


  • sbharrat likes this

#12 Hypoxic

Hypoxic

    Viking 1

  • *****
  • topic starter
  • Posts: 859
  • Joined: 24 Dec 2011
  • Loc: Oise, France

Posted 12 April 2021 - 12:38 PM

In my case, I'm looking at RAW flats, straight from the camera.



#13 cybermayberry

cybermayberry

    Vostok 1

  • -----
  • Posts: 192
  • Joined: 26 Aug 2010

Posted 12 April 2021 - 04:02 PM

It rears it ugly head again. 

Pixinsight does something odd when propriety raw file are brought into it.

I have a Nikon body that allows uncompressed TIF to be created. I then created two flats one saved in uncompressed NEF, and one saved in uncompressed tiff both saved at 12 bits. Everything was identical except file format. I brought them both in pixinsight the TIFF files histogram showed as expected center of histogram. The NEF showed on histogram pinned against the left side.

When additional flats were taken and master flats were created one from TIFF and one from NEF. histogram results in pixinsight were the same, both also seem to function the same in calibration.



#14 sharkmelley

sharkmelley

    Aurora

  • *****
  • Posts: 4,623
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 12 April 2021 - 04:48 PM

It rears it ugly head again. 

Pixinsight does something odd when propriety raw file are brought into it.

I have a Nikon body that allows uncompressed TIF to be created. I then created two flats one saved in uncompressed NEF, and one saved in uncompressed tiff both saved at 12 bits. Everything was identical except file format. I brought them both in pixinsight the TIFF files histogram showed as expected center of histogram. The NEF showed on histogram pinned against the left side.

When additional flats were taken and master flats were created one from TIFF and one from NEF. histogram results in pixinsight were the same, both also seem to function the same in calibration.

The two files created by your camera are completely different - that's why their histograms are so different.

 

The raw file is just as it says - it is raw data.  But the JPG and TIFF files created by your camera are non-linear processed images.  That's why their histograms differ so much from the raw data.  By the way, the BoC (back-of-camera) histogram is the histogram of a non-linear processed image and that is why it is so different from the histogram of the linear raw data.

 

PixInsight is not doing anything odd but it is important to understand if the image file contains raw data or a fully processed image. 

 

Mark


Edited by sharkmelley, 12 April 2021 - 04:52 PM.


#15 cybermayberry

cybermayberry

    Vostok 1

  • -----
  • Posts: 192
  • Joined: 26 Aug 2010

Posted 12 April 2021 - 06:08 PM

The two files created by your camera are completely different - that's why their histograms are so different.

 

The raw file is just as it says - it is raw data.  But the JPG and TIFF files created by your camera are non-linear processed images.  That's why their histograms differ so much from the raw data.  By the way, the BoC (back-of-camera) histogram is the histogram of a non-linear processed image and that is why it is so different from the histogram of the linear raw data.

 

PixInsight is not doing anything odd but it is important to understand if the image file contains raw data or a fully processed image. 

 

Mark

I don't doubt you, but I am confused as to why when both NEF and TIFF are saved uncompressed, why the TIFF files data size is larger if it contains less data? I am aware that the BOC histogram in NEF is using the baked in 8bit Jpeg contained with in NEF for display of the image and for the histogram. This is what led me to the false conclusion that seeing the TIFF file was larger it contained at least the same amount of recorded image data. I was mistaken when I thought the Tiff was being saved at 12 bit, it was only 8 bit. However this leaves me even more confused as to differences in files size of uncompressed formats. This would explain the PI histogram difference however.


Edited by cybermayberry, 12 April 2021 - 06:39 PM.


#16 sharkmelley

sharkmelley

    Aurora

  • *****
  • Posts: 4,623
  • Joined: 19 Feb 2013
  • Loc: UK

Posted 12 April 2021 - 06:41 PM

I don't doubt you, but I am confused as to why when both NEF and TIFF are saved uncompressed, why the TIFF files data size is larger if it contains less data? I am aware that the BOC histogram in NEF is using the baked in 8bit Jpeg contained with in NEF for display of the image and for the histogram. This is what led me to the false conclusion that seeing the TIFF file was larger it contained at least the same amount of recorded image data

I don't know which camera you are using or what file sizes you are seeing but in simple terms the TIFF file will be large because it is a 3 channel colour image file i.e. it has to store a red, green and blue value for every pixel.  Raw files store only a single value for each pixel.

 

 

Mark


  • Stelios, cybermayberry, limeyx and 1 other like this

#17 cybermayberry

cybermayberry

    Vostok 1

  • -----
  • Posts: 192
  • Joined: 26 Aug 2010

Posted 12 April 2021 - 07:34 PM

Thanks, Mark.

I just found that information and was about to confirm the validity of it with you.

If I followed what I read correctly it appears the major difference between raw (in this case NEF) and tiff files is the unbayerisation or demosaicing phase, which as you described above increase the file size for the tiff files

I image with a dedicated astro-camera and I am glad I do, in the end it seems a bit less confusing. However topic has come up a few times and peaked my interest. I feel I half way almost understand what going on with it now. I really appreciate it.




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics