Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Pedestal...Please explain (in basic terms).

  • Please log in to reply
36 replies to this topic

#1 scopenitout

scopenitout

    Surveyor 1

  • -----
  • topic starter
  • Posts: 1718
  • Joined: 24 Aug 2013
  • Loc: Mt. Belzoni

Posted 07 July 2016 - 11:42 AM

After reading through this post: http://www.cloudynig...ion-challenges/ it occured to me after imaging for 4 yrs now, I know nothing about the "pedestal".

 

I had been using DSS for a long time. Then I began to use PI for calibration and stacking.

First, with the handy BPP script, and then using the individual, calibration and integration modules (thank you, David Auld). The difference in the image quality was substantially positive. I will never return to DSS.

 

I image with a range of cameras, from Canon DSLR to Atik 414EX to the SBIG STF-8300. I have never (yet) paid any attention to setting or adjusting the "pedestal", and the resulting  images look just fine to my eye.

It's probably just as well I did not attempt any adjustments, given I do not know what a pedestal is or what it does to the image. 

 

So, please give me a practical explanation of what Pedestal is and how it works, and why I might or might not need to use it in my processing workflow.

 

TIA to all respondents!



#2 futuneral

futuneral

    Apollo

  • *****
  • Posts: 1004
  • Joined: 27 Dec 2014
  • Loc: Phoenix, AZ

Posted 07 July 2016 - 12:14 PM

Pedestal is just a constant value added to each pixel's value to prevent the value from going negative during calibration.

 

Practical (will only look at bias for brevity): Say you have zero signal on some pixel - zero light falls onto it. The sensor will still measure some value (due to bias). Say 12. After you got this sub, you want to calibrate by subtracting a bias frame from your library. Your bias frame is a combination of many bias images made to reduce the noise. However, your actual pixel value you measured above still has its noise. So potentially, the value of the corresponding pixel in the bias frame could be something like 20 and not exactly 12. In this case when you subtract your bias, you're gonna get a negative value (or rather clipping, because it'll be reset to 0). 

In order to avoid this, at the very beginning we can add say 100 to each pixel. So your pixel becomes 112, you subtract 20 and get a positive number - no information loss.


  • Ken Sturrock, Jon Rista, jlandy and 1 other like this

#3 pedxing

pedxing

    Vanguard

  • *****
  • Posts: 2292
  • Joined: 03 Nov 2009
  • Loc: SE Alaska

Posted 07 July 2016 - 12:27 PM

A pedestal is a small fixed value that is added to all pixels in an image so that a subtraction operation doesn't result in negative values for any of the pixels.

Negative values aren't supported, so they end up all being zero. That is a form of clipping and it results in a loss of information.

In my case, for some reason one or more of the subtraction operations involved in calibrations was causing clipping, so my calibrated images were coming out like this:

clipped.jpg

instead of this:

not_clipped.jpg

Adding a pedestal in Image Calibration seems to have fixed the problem. You can run the PixelMath test that Jon Rista posted in that thread to see if you have any clipping. I've never seen it with my DSLR images.
  • Ken Sturrock, Jon Rista, futuneral and 1 other like this

#4 futuneral

futuneral

    Apollo

  • *****
  • Posts: 1004
  • Joined: 27 Dec 2014
  • Loc: Phoenix, AZ

Posted 07 July 2016 - 12:43 PM

A pedestal is a small fixed value that is added to all pixels in an image so that a subtraction operation doesn't result in negative values for any of the pixels.

Negative values aren't supported, so they end up all being zero. That is a form of clipping and it results in a loss of information.

In my case, for some reason one or more of the subtraction operations involved in calibrations was causing clipping, so my calibrated images were coming out like this:
 

 

Could the original issue have been that you have a pedestal set in all of your calibration frames? So you pretty much need something like 3x the pedestal on your light frames to avoid negatives?



#5 William Mc

William Mc

    Mercury-Atlas

  • *****
  • Posts: 2783
  • Joined: 24 Nov 2007
  • Loc: North East Florida

Posted 07 July 2016 - 12:46 PM

I've never seen it either. Is there a set of circumstances that would  produce this calibration clipping repeatedly, or is this a somewhat rare occurrence?



#6 pedxing

pedxing

    Vanguard

  • *****
  • Posts: 2292
  • Joined: 03 Nov 2009
  • Loc: SE Alaska

Posted 07 July 2016 - 12:57 PM

Could the original issue have been that you have a pedestal set in all of your calibration frames? So you pretty much need something like 3x the pedestal on your light frames to avoid negatives?

I don't know how a pedestal would be getting set, I'm not doing anything unusual, just my normal workflow in SGP/PI.

I tried both with the master darks/biases that BPP produces and with masters I created using David Ault's recommendations and got the same results.

Also, there is definitely not a PEDESTAL keyword in the fits headers.

#7 pedxing

pedxing

    Vanguard

  • *****
  • Posts: 2292
  • Joined: 03 Nov 2009
  • Loc: SE Alaska

Posted 07 July 2016 - 01:01 PM

I've never seen it either. Is there a set of circumstances that would  produce this calibration clipping repeatedly, or is this a somewhat rare occurrence?


I am plagued by clouds so I haven't been able to see if it persists.

It's probably just me, I'm cursed. :smile:

#8 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 23347
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 07 July 2016 - 01:33 PM

A pedestal is simply a linear, noiseless digital offset added to each pixel. You could think of it as a "perfect and ideal" replacement (having a standard deviation of zero and devoid of pattern) for a bias offset, which actually serves the same purpose in analog space. The bias offset bumps up the voltage of each pixel, such that when analog noise is encountered within the analog signal, the voltage never drops to zero, thus giving you "room" to preserve the full signal despite noise. 

 

The problem with a bias offset is, it is a signal itself, and thus has noise. Due to the nature of a sensor, the bias noise tends to be fixed, so we can easily remove the noise within the bias signal by subtracting it. However, if we simply subtract the bias signal without first adding a pedestal, then the benefits of having a bias offset in the first place are negated...any pixels that have a value less than the baseline bias offset in ADU (say 256 ADU) are going to end up clipping to black once we subtract 256 ADU (or around there, accounting for the fixed pattern structure of the bias signal) from each pixel. By adding a pedestal first, we add a NOISELESS replacement for the bias, thus ensuring that once we subtract the bias out, we will not clip anything. 

 

The way PixInsight, and I believe most tools with full FITS compliance, track a pedestal is by adding a PEDESTAL keyword to the FITS headers, with the value of the pedestal you added to the frames. This allows the pedestal to be automatically handled by processing tools from that point forward. PixInsight will take care of removing the pedestal if it ever needs to be (i.e. when doing full precision floating point math where negative values are allowed), as should any other fully FITS compliant editor. 


  • futuneral and Gary Imm like this

#9 jdupton

jdupton

    Surveyor 1

  • *****
  • Posts: 1774
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 07 July 2016 - 02:14 PM

scopenitout, Wmacky,

 

So, please give me a practical explanation of what Pedestal is and how it works, and why I might or might not need to use it in my processing workflow.

 

I've never seen it either. Is there a set of circumstances that would  produce this calibration clipping repeatedly, or is this a somewhat rare occurrence?

 

   I came across a set of circumstances around the beginning of this year where a pedestal is consistently needed for me. (The required pedestal was rather small at ~45 - 50 ADU but real.) I used to have problems with clipping during calibration and sometimes had to band-aid it using the pedestal. (Sometimes, I didn't bother if the clipping was minimal.)

 

   I began to do a detailed analysis of my two CCD cameras. One is an older Orion StarShootPro OSC (Sony ICX413AQ-based) and the other is a StarLight XPress SXVR-H694M mono (Sony ICX694ALG-based). Both showed some similarly odd results as I was building a library of calibration frames (Bias and Dark). What I noticed was that depending on how I gathered my calibration frames, there were sometimes differences between the beginning and ending frames in a sequence.

 

   After going through my Bias frame collection sequentially (for both cameras) and doing some research online, I found that my Bias frames tended to have higher median ADU values at the beginning and end of the run compared to those in the middle. I found that the higher values at the beginning of the run appeared to be due to not letting the TEC "soak" the sensor long enough before starting the run. I would often just start a Bias run after the temperature readout indicated that I had reached my set-point. The slow rise in values for subsequent frames after reaching a minimum was still a bit puzzling.

 

   In my online research, I learned that at least some StarLight Xpress cameras temporarily turn off cooling power to the TEC during image download to reduce noise. I found that if I inserted a delay between Bias frames rather than "letting them rip" when building the library, the results were more consistent. The same was found to be true of the Orion camera even though it simply runs the cooler all out all the time. (The old Orion camera doesn't have set-point cooling -- it's just on at fixed power all the time.)

 

   My extensive testing indicated to me that taking Bias frames back to back as fast as possible seemed to be inducing self-heating of the readout and clocking circuitry on the chip for both cameras. This tended to lead to higher median values in the integrated Master Bias frames than might be expected otherwise. I "fixed" this problem by inserting a one minute delay between frames in SGP when building my library. The delay gives the cameras time to cool back after the download process before taking another frame.

 

   After gathering a new library using the delay between frames, I found that the required ~50 ADU pedestal was no longer routinely required to prevent clipping. Other cameras may not have this problem but I suspect it may be more common than some think. The constant clocking out of data for downloading Bias frames compared to the "exposure time" of the Bias frames does not mirror our normal light frame exposures.

 

   Lights are characterized by long idle time of the clocking circuits on the chip during exposure followed by furious clocking during download. The download time to idle time ratio for a light frame might be something much less than 1% while for library gathered Bias frames, it will be close to 100% download time. That self heating of the readout circuitry during download clocking can contribute to higher Bias signal in the Master Bias frame compared to the Bias signal in the light frame. In such a case, it could be enough that a pedestal is required to compensate.

 

   David Ault and I talked this over one night and he mostly agreed although he had never seen the effect in his camera. I vaguely recall that he later mentioned having done some testing himself and noticed a similar albeit less effect compared to what I found. If he sees this post, maybe he can refresh my memory.

 

   So, in my opinion, a small pedestal may be generally useful depending on how Bias frames are gathered. The same thing might happen for short duration Flats, Flat Darks, or any other frame type where download time of multiple frames becomes a significant percentage of the total imaging cycle.

 

 

John


Edited by jdupton, 07 July 2016 - 02:32 PM.

  • futuneral likes this

#10 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 23347
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 07 July 2016 - 02:23 PM

Very interesting, John. I might have to check out my cameras to see if there is a similar effect. I started using a pedestal with my 5D III because I was getting too many clipped pixels. I've noticed it is often required to avoid too much clipping with KAF-8300 data as well...but, not always. Depends on the data and who it is coming from. I've been using a small pedestal with my ASI1600, but that camera does seem to have some inconsistencies with it's bias signal, and perhaps following your technique will help minimize the issue.


  • futuneral likes this

#11 jerry10137

jerry10137

    Viking 1

  • *****
  • Vendors
  • Posts: 776
  • Joined: 26 Jan 2013
  • Loc: Texas, USA

Posted 07 July 2016 - 02:53 PM

Is there a place in pix insight that will measure this for a single sub and your master bias to see what's happening? Also, where in pixinsight can you add the pedistal?



#12 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 23347
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 07 July 2016 - 03:27 PM

Clipping will often occur when your background sky signal is shallow. If your bias offset is 800 ADU, and your signal mean is 810 ADU, but your standard deviation is 20 ADU, then you will have pixel values that are well below 800 due to the noise.

 

One way to avoid the need to use a pedestal is to expose for longer, and get that mean skyfog level higher above the bias offset level. There are some potential consequences to exposing longer. If you expose for too long, then you can needlessly lose dynamic range, as the higher your average background level, the more of your dynamic range you are consuming with data that will simply be offset during processing (i.e. thrown away, useless). It takes some testing and evaluation to determine the sweet spot, where you have exposed long enough to sufficiently swamp camera noise, but not so long that you are just wasting dynamic range and needlessly clipping stars.

 

Even if you do find that sweet spot, that does not guarantee that you will not clip data. Noise can spread out several standard deviations from the signal peak, and even with a 50 ADU background sky level (which is a very reasonable level), you could still have usable data dropping below 800 ADU, which would still end up clipped when the bias signal was subtracted. 



#13 pedxing

pedxing

    Vanguard

  • *****
  • Posts: 2292
  • Joined: 03 Nov 2009
  • Loc: SE Alaska

Posted 07 July 2016 - 04:39 PM

Yeah, normally I would shoot 1800s Ha subs, but I had limited time and wanted to get as many to stack (4!) as possible.

So it could be underexposure on my part. Also I didn't have any delay between biases, I'll try again with a delay since they are so easy to take.

#14 pedxing

pedxing

    Vanguard

  • *****
  • Posts: 2292
  • Joined: 03 Nov 2009
  • Loc: SE Alaska

Posted 07 July 2016 - 04:44 PM

Is there a place in pix insight that will measure this for a single sub and your master bias to see what's happening? Also, where in pixinsight can you add the pedistal?


Check out this thread: http://www.cloudynig...ion-challenges/

The pedestal value is entered in the Output Pedestal field of Image Calibration.
  • jerry10137 likes this

#15 jdupton

jdupton

    Surveyor 1

  • *****
  • Posts: 1774
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 07 July 2016 - 05:33 PM

Jon,

 

Very interesting, John. I might have to check out my cameras to see if there is a similar effect. I started using a pedestal with my 5D III because I was getting too many clipped pixels. I've noticed it is often required to avoid too much clipping with KAF-8300 data as well...but, not always. Depends on the data and who it is coming from. I've been using a small pedestal with my ASI1600, but that camera does seem to have some inconsistencies with it's bias signal, and perhaps following your technique will help minimize the issue.

 

   To expand on my methodology for investigating this, you can duplicate what I did on existing data you probably already have. I had always expected and assumed that my Bias files would all be very similar to one another with only random variations between. I was wrong for both my cameras. It is easy to check this assumption if you have PixInsight.

 

   Try this out on you Master Bias Library Data:

 

1) In PixInsight, open the Blink Process.

 

2) Load your raw Bias frames that go into your Master Bias. (I had 250 of them.)

 

3) Open up the Blink Process Series Analysis Report Tool (It's the next to last icon at the bottom of the Blink Process window.)

 

4) In the new dialog window that opens, check the "DATE-LOC" field checkbox to add a timestamp to the output data.

 

5) Also check the "Write text file" checkbox.

 

6) Enter an output location for the statistics file that will be created.

 

7) Execute with the OK button.

 

   When the Blink process finishes processing the data, you will now have a file called Statistic.txt in the output file location.

 

8) Load this file into a spreadsheet (taking care to get values into separate columns). 

 

9) Sort by the DATE-LOC field in case your files names / numbers were not alphabetically sequential.

 

10) Select the Filename or Number Column along with the Mean and Median value columns and create a chart from your data. Choose a chart type of X-Y Plot.

 

   For both my cameras, I get a plot that show rapidly decreasing Mean and Median values which reach a minimum value followed by a very, very slow rise as time (and frame number) increases. This was very different from what I expected to see. I had always assumed I would see a flat line with some very small random variations.

 

   Since all the data isn't the same within statistical variation, you don't end up with a Master Bias that represents the bulk of your information. Instead, the greater values (especially at the beginning) tend to increase the Mean of the Master Bias which, for me, was leading to clipping in calibration.

 

 

John


  • Jon Rista and futuneral like this

#16 futuneral

futuneral

    Apollo

  • *****
  • Posts: 1004
  • Joined: 27 Dec 2014
  • Loc: Phoenix, AZ

Posted 07 July 2016 - 05:55 PM

Interesting ... mine seems to be pretty stable (KAF8300)

 

 

Attached Thumbnails

  • Screen Shot 2016-07-07 at 3.54.47 PM.png


#17 buckeyestargazer

buckeyestargazer

    Vendor - Buckeyestargazer.net

  • *****
  • Vendors
  • Posts: 4655
  • Joined: 12 Jan 2008
  • Loc: IN, USA

Posted 07 July 2016 - 06:22 PM

 

The pedestal value is entered in the Output Pedestal field of Image Calibration.

 

I assume this just for light frame output?  When creating a master flat we use ImageCalibration as well.  

 

I'm trying to understand this pedestal thing too and I'm wondering if adding a pedestal might help resolve an issue I'm seeing with flats slightly over-correcting the lights in the corners of an image that has a lot of vignetting in the corners.  



#18 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5057
  • Joined: 14 Jun 2009

Posted 07 July 2016 - 07:34 PM

if a calibration frame is marked with having a pedestal (fits keyword PEDESTAL) then the pedestal will be subtracted off at the end of calibration of a light or a flat or whatever. ImageCalibration itself has no idea what it's calibrating, so its behavior won't differ depending on the input frames.

 

i think if you want to figure out if you need a pedestal, subtract a master bias from a master dark and see if there are any 0 valued pixels in the result. you can do that with pixelmath: $T==0 should give you an image with white pixels wherever there was a 0 value in the resultant image.

 

rob


  • jerry10137 likes this

#19 jerry10137

jerry10137

    Viking 1

  • *****
  • Vendors
  • Posts: 776
  • Joined: 26 Jan 2013
  • Loc: Texas, USA

Posted 07 July 2016 - 08:24 PM

i think if you want to figure out if you need a pedestal, subtract a master bias from a master dark and see if there are any 0 valued pixels in the result. you can do that with pixelmath: $T==0 should give you an image with white pixels wherever there was a 0 value in the resultant image.

 

rob

Rob,

 

Thanks for that great piece of information.  I ran this little test on two cameras.  1 result gave me a black image with only 1 or 2 white pixels.  I'm guessing that is pretty good.  The other gave me a result that had quite a few (a bunch really) white pixels.......I'm guessing that is bad.  All based on what I am reading here.



#20 buckeyestargazer

buckeyestargazer

    Vendor - Buckeyestargazer.net

  • *****
  • Vendors
  • Posts: 4655
  • Joined: 12 Jan 2008
  • Loc: IN, USA

Posted 07 July 2016 - 08:26 PM

Yes, thanks Rob.  I ran this test on two different master darks, a 30min and 10min.  The 10min master dark showed 14 white pixels and the 30min show no white pixels.  


Edited by buckeyestargazer, 07 July 2016 - 08:27 PM.


#21 jerry10137

jerry10137

    Viking 1

  • *****
  • Vendors
  • Posts: 776
  • Joined: 26 Jan 2013
  • Loc: Texas, USA

Posted 07 July 2016 - 08:30 PM

I am understanding this is bad?

Attached Thumbnails

  • 7-7-2016 8-26-58 PM.jpg

Edited by jerry10137, 07 July 2016 - 08:31 PM.


#22 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 23347
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 07 July 2016 - 09:00 PM

Yes, you are clipping quite a lot. I'd say that warrants a pedestal.
  • jerry10137 likes this

#23 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5057
  • Joined: 14 Jun 2009

Posted 07 July 2016 - 09:04 PM

yes that would seem to me that there are very many pixels in the master bias that exceed the ADU value of the corresponding pixel in the master dark.

 

as a sanity check, you can verify by using the Statistics process to see what the mean value in the bias is vs. the mean in the dark. the bias should be slightly higher to be consistent with this result.

 

when i ran into this, it manifested as the ImageCalibration process being unable to scale the master dark, as it turned out to be mostly 0. however, it could be that you'll never see any symptom of this if less than some percentage of the pixels are zeroed out. of course just because the program does not detect a problem does not mean that the calibrated result is correct.

 

so it would seem that it's worth adding an output pedestal as described further up this thread. having said this, i am not sure how it all works out if you simply load uncalibrated dark/bias masters into ImageCalibration and try to calibrate a light, but you specify an output pedestal. in that case the calibrated light itself might get written with the PEDESTAL keyword, which i don't think is what you want. it's likely that the pedestal is not even necessary in this case because the result of the subtraction of both the bias and the dark from the light will not yield a 0 pixel value - assuming that the light's ADU values far exceed the bias and dark, which is generally true. the case where you are calibrating dark subs with a master bias is when a pedestal is necessary - without it the calibrated dark subs gets written out with a bunch of 0 values, and then the integrated master dark also has a bunch of 0s as well.

 

rob


  • jerry10137 likes this

#24 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 23347
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 07 July 2016 - 09:11 PM

I wouldn't say it is necessarily true that the lights background sky level "far exceeds" the dark mean level. That really depends on how much dynamic range you have, what you are imaging, what filters (if any) you may be using, and how much of the full signal range you are trying to preserve. If you don't mind clipping a fair amount of stars, and/or have lots of dynamic range, then it might be true that the background sky level of the lights "far exceeds" the mean level of the darks. However it is just as likely that you are limited in how bright you can make the background sky level, and it may not be possible to make it a standard deviation or greater than the mean of the dark level, and still preserve enough dynamic range/avoid clipping too many stars.

#25 jerry10137

jerry10137

    Viking 1

  • *****
  • Vendors
  • Posts: 776
  • Joined: 26 Jan 2013
  • Loc: Texas, USA

Posted 07 July 2016 - 09:18 PM

Do you know how many people are loading up PixInsight right now and checking this LOL

 

Seriously......great info.  Thanks for the time invested to explain this.  I learned something very valuable today  :waytogo:




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics