Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Bimodal Master Bias Frames

  • Please log in to reply
21 replies to this topic

#1 sws626

sws626

    Sputnik

  • -----
  • topic starter
  • Posts: 46
  • Joined: 31 Aug 2017
  • Loc: Berkeley, CA, USA

Posted 03 October 2017 - 01:59 PM

I mentioned yesterday that I'd seen something odd when reviewing the histograms of some of my master bias frames.  While each individual bias frame shows a normal distribution, the integrated frames are bimodal, with a dip between the two modes that deepens with the integration of larger numbers of frames while the second of the two peaks gradually drops.

 

I thought I'd share the data and see whether anyone else has any ideas on what might be the source of this phenomenon.  I'm not trying to solve any particular problem here.  I'm just curious as to whether anyone has seen this before or has any thoughts on it.

 

These bias frames were shot in SGP with a Zwo ASI1600MM-C at 198 gain, 50 offset, -20degC, for exposure times of 0.5 sec (I shot another set at "0.0" which shows the same characteristics).  Masters were created from individual frames in PixInsight with the following settings (followwwing David Ault's recommendations:

 

gallery_279346_8308_1372.jpg

 

Here are the statistics from a single frame (original histogram at the bottom, stretched at the top):

 

gallery_279346_8308_58123.jpg

 

10 integrated frames:

 

gallery_279346_8308_73243.jpg

 

20 integrated frames:

 

gallery_279346_8308_16534.jpg

 

40 integrated frames:

 

gallery_279346_8308_4785.jpg

 

80 integrated frames:

 

gallery_279346_8308_60805.jpg

 

160 integrated frames:

 

gallery_279346_8308_1220.jpg

 

300 integrated frames:

 

gallery_279346_8308_66078.jpg



#2 happylimpet

happylimpet

    Fly Me to the Moon

  • *****
  • Posts: 5,618
  • Joined: 29 Sep 2013
  • Loc: Southampton, UK

Posted 03 October 2017 - 02:54 PM

Hiya,

 

Like I said in the other thread, its due to the chequerboard pattern created by having two readout registers on the CMOS chip, which have slightly different 'sensitivities' if you will. It shows up in the biases, lights and flats, but only when the SNR is high enough, hence you needing to stack to see it.

 

Its a pain in the arse as while it calibrates out fine, it means you cant shoot uncalibrated frames for planetary, lunar, solar etc. If you do, the grid pattern will show up, and planetary stacking software (ie autostakkert) will align to the grid.

 

Its a **** shame, but it doesnt really affect its use for calibrated DSO work.

 

By way of example, here's a crop of a flat around a dust mote, blown up x4, and contrast stretched.

MasterFlat_ISO0_cropexample-big.jpg

 

ZWO introduced the 'pattern' parameter to deal with this, and it works to some extent, but not for the settings i use (ie high gain).


Edited by happylimpet, 03 October 2017 - 03:00 PM.

  • sws626 likes this

#3 sws626

sws626

    Sputnik

  • -----
  • topic starter
  • Posts: 46
  • Joined: 31 Aug 2017
  • Loc: Berkeley, CA, USA

Posted 03 October 2017 - 03:37 PM

That is really very interesting.  It hadn't occurred to me that the camera has two readout registers with subtly different sensitivities.  It makes perfect sense that these would become more evident with stacking and I've now been able to see this in the stacked dark frames as well.  It looks like it will be straightforward to filter  these out, but it would be a real mess if you couldn't.

 

Here are close ups of the same regions of images stretched and zoomed to the same degree, two randomly selected single bias frames (leftmost column) and the six integrated bias frames I showed data from earlier.  The "checkerboard" pattern is lost in the noise in the single frames, but is clearly evident as soon as even a few more are stacked.

 

gallery_279346_8308_143597.jpg


Edited by sws626, 03 October 2017 - 03:42 PM.

  • happylimpet likes this

#4 happylimpet

happylimpet

    Fly Me to the Moon

  • *****
  • Posts: 5,618
  • Joined: 29 Sep 2013
  • Loc: Southampton, UK

Posted 03 October 2017 - 06:16 PM

Yup. You might find that the 'pattern' parameter helps reduce it, but it didnt for me....i did exhaustive tests and found out that while the pattern changed in intensity across the values for bias and sensitivity (ie y=mx+c) there was no value that particularly reduced its prominence in a useful way for both bias and sensitivity.

 

Still a great DSO camera. Not such a good planetary/lunar/solar camera.



#5 kingjamez

kingjamez

    Vanguard

  • *****
  • Posts: 2,310
  • Joined: 03 Oct 2006
  • Loc: Fairfax, VA

Posted 03 October 2017 - 09:08 PM



Yup. You might find that the 'pattern' parameter helps reduce it, but it didnt for me....i did exhaustive tests and found out that while the pattern changed in intensity across the values for bias and sensitivity (ie y=mx+c) there was no value that particularly reduced its prominence in a useful way for both bias and sensitivity.

 

Still a great DSO camera. Not such a good planetary/lunar/solar camera.

While I understand your reasoning, I don't agree with your conclusion. This is from my ASI1600mm-c and I see zero evidence of the checkerboard despite having stacked over 2000 frames for this image. 

 

Could it be worse on some camera's than others? 

 

get.jpg

 

-Jim



#6 happylimpet

happylimpet

    Fly Me to the Moon

  • *****
  • Posts: 5,618
  • Joined: 29 Sep 2013
  • Loc: Southampton, UK

Posted 04 October 2017 - 05:11 AM

 



Yup. You might find that the 'pattern' parameter helps reduce it, but it didnt for me....i did exhaustive tests and found out that while the pattern changed in intensity across the values for bias and sensitivity (ie y=mx+c) there was no value that particularly reduced its prominence in a useful way for both bias and sensitivity.

 

Still a great DSO camera. Not such a good planetary/lunar/solar camera.

While I understand your reasoning, I don't agree with your conclusion. This is from my ASI1600mm-c and I see zero evidence of the checkerboard despite having stacked over 2000 frames for this image. 

 

Could it be worse on some camera's than others? 

 

get.jpg

 

-Jim

 

It might be! It also depends very much on what settings are being used, ie gain and offset. May I ask what settings you used here?

 

As far as I know its not an intentional offset, its just variability between two different readout systems, so on some cameras this effect might be effectively zero (lucky you!).

 

Did you stack this in autostakkert? NOte that what tends to happen is that where there are high contrast features in the image (ie a lunar mountain peak) stacking works fine and the grid can be lost due to natural dithering, but where there are no strong features to lock onto (lunar plains) it shows up.  It can be reduced by 'preblurring' in 'advanced features' in autostakkert but this messes with the quality estimation.

 

Im not so sure you are free of it either, look at this crop of your image:

 

Capture.PNG

 

You'll see similar across much of it. You'll probably find it everywhere now youre looking.



#7 kingjamez

kingjamez

    Vanguard

  • *****
  • Posts: 2,310
  • Joined: 03 Oct 2006
  • Loc: Fairfax, VA

Posted 04 October 2017 - 11:23 AM

Meh... If it is there, and I have to zoom to 400% to "kinda" see it, I don't think that disqualifies the camera from being a good solar system imager. 

 

Yes, I use autostakkert to stack.

 

-Jim


Edited by kingjamez, 04 October 2017 - 11:25 AM.


#8 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 24,804
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 04 October 2017 - 12:41 PM

Hmm...I'm wondering if it is really just that the checkerboard pattern is causing the bimodality. Stuart, any chance you could share your bias/master bias? 



#9 sws626

sws626

    Sputnik

  • -----
  • topic starter
  • Posts: 46
  • Joined: 31 Aug 2017
  • Loc: Berkeley, CA, USA

Posted 04 October 2017 - 01:23 PM

Hmm...I'm wondering if it is really just that the checkerboard pattern is causing the bimodality. Stuart, any chance you could share your bias/master bias? 

I've uploaded two master bias files (one integrated over 10 frames, another over 300 frames) and the 10 individual bias frames used to make the first master to a Dropbox directory here.  I see the same pattern, though not quite as pronounced, in the dark histograms, so have uploaded a dark master (integrated over 36 frames) and one of its individual dark frame as well.


Edited by sws626, 04 October 2017 - 11:09 PM.


#10 happylimpet

happylimpet

    Fly Me to the Moon

  • *****
  • Posts: 5,618
  • Joined: 29 Sep 2013
  • Loc: Southampton, UK

Posted 05 October 2017 - 05:17 AM

Meh... If it is there, and I have to zoom to 400% to "kinda" see it, I don't think that disqualifies the camera from being a good solar system imager. 

 

Yes, I use autostakkert to stack.

 

-Jim

 

But think about what this actually means for your images.  Autostakkert stacks to subpixel accuracy, meaning the full resolution of the system is preserved.

 

With this grid, autostakkert is 'forced' into aligning to the nearest 2x2 position that matches the grid, like a 'snap to grid' in microsoft office.

 

This means you're smearing your frames by an average of 1 pixel in each axis. If you're very oversampled you might be happy to live with that, but its throwing away resolution needlessly and imposing an ugly grid on your image. I know I'm not happy with that!

 

Think how much better that solar granulation would look without the grid, as well as being sharper......on a big print you'll certainly see it clear as day.


Edited by happylimpet, 05 October 2017 - 05:18 AM.


#11 kingjamez

kingjamez

    Vanguard

  • *****
  • Posts: 2,310
  • Joined: 03 Oct 2006
  • Loc: Fairfax, VA

Posted 05 October 2017 - 10:43 AM

 

Meh... If it is there, and I have to zoom to 400% to "kinda" see it, I don't think that disqualifies the camera from being a good solar system imager. 

 

Yes, I use autostakkert to stack.

 

-Jim

 

But think about what this actually means for your images.  Autostakkert stacks to subpixel accuracy, meaning the full resolution of the system is preserved.

 

With this grid, autostakkert is 'forced' into aligning to the nearest 2x2 position that matches the grid, like a 'snap to grid' in microsoft office.

 

This means you're smearing your frames by an average of 1 pixel in each axis. If you're very oversampled you might be happy to live with that, but its throwing away resolution needlessly and imposing an ugly grid on your image. I know I'm not happy with that!

 

Think how much better that solar granulation would look without the grid, as well as being sharper......on a big print you'll certainly see it clear as day.

 

Again, I understand where you are coming from, but I disagree with your conclusions.

 

I'm quite happy with large prints. This particular one was taken using an ROI of 1700x1700, so not even half the sensor was being used. Zero evidence of artifacts.

Attached Thumbnails

  • sun_print.jpg

Edited by kingjamez, 05 October 2017 - 10:45 AM.

  • happylimpet likes this

#12 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 24,804
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 05 October 2017 - 11:39 AM

 

Hmm...I'm wondering if it is really just that the checkerboard pattern is causing the bimodality. Stuart, any chance you could share your bias/master bias? 

I've uploaded two master bias files (one integrated over 10 frames, another over 300 frames) and the 10 individual bias frames used to make the first master to a Dropbox directory here.  I see the same pattern, though not quite as pronounced, in the dark histograms, so have uploaded a dark master (integrated over 36 frames) and one of its individual dark frame as well.

 

So I did some testing. PixInsight has a pretty cool histogram tool, and when you click with the pixel inspector on a pixel in an image, it draws a vertical line in the histogram where that pixel is. It seems that the checkerboard is indeed the cause of the bimodality, as when I move the pixel inspector around a clean background area (no hot pixels), it jumps back and forth between the two modes based on which part of the checkerboard I'm clicking on (dark or light). Additionally, in the two images that demonstrate bimodality, the checkerboard is actually quite strong. In the other two that don't really exhibit bimodality, the checkerboard is barely visible. 

 

I thought the bimodality might be the result of glows, however that does not seem to be the case. The glows seem to result in a larger shoulder in the bright part of the histogram that leads into an impulsational bump from hot pixels shortly after.

 

Anyway. The strongly bimodal bias definitely appears to be due to the fairly strong checkerboard. 


  • kingjamez and sws626 like this

#13 freestar8n

freestar8n

    Vendor - MetaGuide

  • *****
  • Vendors
  • Posts: 10,227
  • Joined: 12 Oct 2007
  • Loc: Melbourne, Australia

Posted 05 October 2017 - 03:45 PM

I will need to look at what my asi1600 is doing because I don't see any of these issues.

 

The strange thing (or one of them) about this checkerboard issue is that if it were only in the bias frames, and only visible after stacking, it would be a pretty subtle effect when looking at a well exposed image of the sun - where the shot noise would dominate small issues in the bias.  It sure looks like something thinks it is a bayer array camera.

 

If the pattern shows in a well exposed image - that implies it's either PRNU - which wouldn't show in the bias or dark - or some artifact of the calibration process.

 

If it doesn't show in a single bias, does it show in a very dumb and simple average of many bias frames?

 

Are you sure the settings for the camera were correct when the frames were captured?

 

Frank



#14 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 24,804
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 05 October 2017 - 04:19 PM

Frank, it seems to be both DSNU and PRNU. I see the pattern, although weakly, in my flats, and if I integrate a lot of flats it can become strong. I also see it a bit in my master biases. Whatever causes it, it doesn't discriminate between dark signal or photon signal.

Edited by Jon Rista, 05 October 2017 - 04:25 PM.

  • happylimpet likes this

#15 kingjamez

kingjamez

    Vanguard

  • *****
  • Posts: 2,310
  • Joined: 03 Oct 2006
  • Loc: Fairfax, VA

Posted 05 October 2017 - 04:32 PM

Here is a screen shot of my master bias it's a stack of 100, 0.03" frames, Gain 200 offset 50, temp -18c. The only difference from yours is that my sigma clipping was set to 4 for low and 3 for high.

Attached Thumbnails

  • Bias_Histogram.jpg


#16 happylimpet

happylimpet

    Fly Me to the Moon

  • *****
  • Posts: 5,618
  • Joined: 29 Sep 2013
  • Loc: Southampton, UK

Posted 05 October 2017 - 05:30 PM

Heres some tests I did last year when this first came up, and I was bickering with Sam at ZWO about the strength of the effect and trying to make a case for getting my money back. This was because I wanted to use it for uncalibrated planetary imaging. Now I have an ASI290MM for the same job Im not so bothered.

 

As a result of my flagging this up (and possibly others, but Im not aware of that) he introduced a 'pattern' parameter to adjust for it, with settings from 0 to 4.  This is certainly avaiable in Firecapture, not sure about SGP etc.

 

For each setting of 'pattern', I ran tests on biases (note each image has the exact same screen stretch):

bias-pattern-samerange.png

 

and flats:

flats-pattern.png

 

These are all stacks of around 400 frames, I think.

 

(more coming, hang on)


Edited by happylimpet, 05 October 2017 - 05:36 PM.


#17 happylimpet

happylimpet

    Fly Me to the Moon

  • *****
  • Posts: 5,618
  • Joined: 29 Sep 2013
  • Loc: Southampton, UK

Posted 05 October 2017 - 05:33 PM

As well as typical high-gain exposures for Neptune:

 

Nep-pattern-samerange.png

 

I also plotted the standard deviation of the image, a measure of sorts of the bimodality or grid:

 

graph.png

 

This showed that there was no single setting that removed the pattern for me, under all circumstances. Note that this all relates to high gain work (gain~450) whereas DSO folks tend to use much lower gains, where it might work. Others reported success with pattern=3, for example.

 

 



#18 sws626

sws626

    Sputnik

  • -----
  • topic starter
  • Posts: 46
  • Joined: 31 Aug 2017
  • Loc: Berkeley, CA, USA

Posted 05 October 2017 - 06:24 PM

I don't think there's anything unusual about how the images were collected.  I confirmed Jon's observation that adjacent pixels correspond to alternate humps in the histogram.  The pattern is more pronounced as more frames are integrated, but seems to level out around 100 frames.  It's there, though fainter, in as few as 3.  I haven't taken any flats yet, but its present in when integrating either bias or darks.  I did another run with 100 bias frames and the low 4.0, high 3.0 sigma clipping to compare with @kingjamez amd still see it (shown below).

 

The only thing I can think of is that in some cameras, the differences in the two readout registers is negligible and in other not.  I'm not planning on doing any planetary or solar imaging, but I'll be interested to see how this calibrates out of stacked lights.

Attached Thumbnails

  • 100-Frame_Bias_198_50_0p5s_neg20C.JPG

Edited by sws626, 05 October 2017 - 06:28 PM.

  • happylimpet likes this

#19 happylimpet

happylimpet

    Fly Me to the Moon

  • *****
  • Posts: 5,618
  • Joined: 29 Sep 2013
  • Loc: Southampton, UK

Posted 06 October 2017 - 04:39 AM

The only thing I can think of is that in some cameras, the differences in the two readout registers is negligible and in other not.  I'm not planning on doing any planetary or solar imaging, but I'll be interested to see how this calibrates out of stacked lights.

I agree.

 

Ive been keeping a close eye on this, and as far as I can tell, it calibrates out perfectly.



#20 StevenBellavia

StevenBellavia

    Viking 1

  • *****
  • Posts: 657
  • Joined: 11 Mar 2014
  • Loc: New York

Posted 11 May 2018 - 02:19 PM

Not sure this is the right thread, but I am getting bimodal flat frames, on each individual frame, with a huge space between the distributions.

 

ZWO ASI 183MC

0.25 sec exposure

Gain = 0 (max dynamic range, from presets in driver)

Gerd Neumann aurora flat panel with 1/16" thick UHMW plastic to reduce illumination, so I can increase exposure time, to help remove banding from slow PWM

 

 

Attached Thumbnails

  • Flat_Frame_0p25sec_Gain_0_plastic.JPG


#21 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 24,804
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 11 May 2018 - 04:21 PM

Not sure this is the right thread, but I am getting bimodal flat frames, on each individual frame, with a huge space between the distributions.

 

ZWO ASI 183MC

0.25 sec exposure

Gain = 0 (max dynamic range, from presets in driver)

Gerd Neumann aurora flat panel with 1/16" thick UHMW plastic to reduce illumination, so I can increase exposure time, to help remove banding from slow PWM

In this case, it is explicitly because you are using a color camera. A color camera has this by nature in non-debayered data, since each channel is not necessarily going to get the same amount of signal. Depends on the power of the signal in each channel. But this is not a problem for you, as once you demosaic it won't really matter. You will still have different peaks in the histogram, but you'll see that they correspond to distinct channels.



#22 billdan

billdan

    Mariner 2

  • -----
  • Posts: 293
  • Joined: 21 Feb 2012
  • Loc: SE Queensland

Posted 12 May 2018 - 06:16 AM

It would be interesting to know if QHY163 users are getting the same pattern as the ASI1600. This way we could speculate if its an issue with the sensor or the external electronics that's causing it.


Edited by billdan, 12 May 2018 - 06:17 AM.



CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics