Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Fixed Pattern Noise getting worse and worse

  • Please log in to reply
145 replies to this topic

#26 joelin

joelin

    Mercury-Atlas

  • *****
  • topic starter
  • Posts: 2,890
  • Joined: 14 Jan 2008
  • Loc: Saratoga, CA

Posted 13 December 2018 - 02:51 AM

Well you have the original light frames, it should be easy for you to compare the first and the last one to see if there’s any drift.

Ok I tested it with the blink test

Here is a YouTube video 24 seconds long...please play in HD

 

https://www.youtube....eature=youtu.be

 

It seems to show black dots all over the frame...is there a problem with the way I debayered? It looks terrible! 

 

and yes there is drift even when guiding!!!



#27 joelin

joelin

    Mercury-Atlas

  • *****
  • topic starter
  • Posts: 2,890
  • Joined: 14 Jan 2008
  • Loc: Saratoga, CA

Posted 13 December 2018 - 03:01 AM

Also, the following screenshot shows D600 on the left and debayered ASI1600 on the right, zoomed in so 1 pixel is represented by 10 on the screen

 

VG9mVEE.png

 

 

what is that ugly pattern? I used debayer GRGB and Billinear and Multiresolution support



#28 fmeschia

fmeschia

    Vanguard

  • *****
  • Posts: 2,173
  • Joined: 20 May 2016
  • Loc: Mountain View, CA

Posted 13 December 2018 - 11:16 AM

Ok I tested it with the blink test

Here is a YouTube video 24 seconds long...please play in HD

 

https://www.youtube....eature=youtu.be

 

It seems to show black dots all over the frame...is there a problem with the way I debayered? It looks terrible! 

 

and yes there is drift even when guiding!!!

The black dots form a fixed pattern... how were the subs calibrated?



#29 joelin

joelin

    Mercury-Atlas

  • *****
  • topic starter
  • Posts: 2,890
  • Joined: 14 Jan 2008
  • Loc: Saratoga, CA

Posted 13 December 2018 - 11:38 AM

In the blink test they weren’t calibrated, only debayered.


It seems the debayer result looks terrible. Is that how they should come out?

#30 fmeschia

fmeschia

    Vanguard

  • *****
  • Posts: 2,173
  • Joined: 20 May 2016
  • Loc: Mountain View, CA

Posted 13 December 2018 - 11:46 AM

It seems the debayer result looks terrible. Is that how they should come out?

I never use Bilinear. Can you try VNG and see if it’s any better?

Also I don’t know what is the Bayer pattern for those cameras. If you leave it to “Auto” in PixInsight, it should infer it from the EXIF data for the DSLR, but I’m not sure if it can do the same for the ASI1600.



#31 entilza

entilza

    Soyuz

  • *****
  • Posts: 3,831
  • Joined: 06 Oct 2014
  • Loc: Oakville, ON, Canada

Posted 13 December 2018 - 01:44 PM

and yes there is drift even when guiding!!!


Hope this is all starting to make sense. This hobby keeps you busy. That is a wonderful video to show people the entire walking noise issue.

#32 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 6,565
  • Joined: 14 Jun 2009

Posted 13 December 2018 - 01:46 PM

seems to me that the ASI image was debayered with wrong settings. you shouldn't see such a strong 'screen door' look in a correctly debayered file.

 

rob



#33 joelin

joelin

    Mercury-Atlas

  • *****
  • topic starter
  • Posts: 2,890
  • Joined: 14 Jan 2008
  • Loc: Saratoga, CA

Posted 13 December 2018 - 02:38 PM

what settings should I use on PI or DSS? this is the ASI1600MC pro.



#34 Peter in Reno

Peter in Reno

    Hubble

  • *****
  • In Memoriam
  • Posts: 13,067
  • Joined: 15 Jul 2008
  • Loc: Reno, NV

Posted 13 December 2018 - 02:42 PM

There are only four debayer settings, try them all and decide which one looks correct. Not all cameras debayer automatically or correctly with certain software and sometimes you have to guess.

 

Peter



#35 42itous1

42itous1

    Viking 1

  • *****
  • Posts: 920
  • Joined: 21 May 2016
  • Loc: Wilton, CA

Posted 13 December 2018 - 02:55 PM

I have always used GRBG to debayer my asi1600mcc.  Easy to remember "garbage" for GRBG


Edited by 42itous1, 13 December 2018 - 02:56 PM.


#36 FlankerOneTwo

FlankerOneTwo

    Apollo

  • *****
  • Posts: 1,108
  • Joined: 30 Aug 2017
  • Loc: Vegas, baby!

Posted 13 December 2018 - 09:16 PM

 

what is that ugly pattern? I used debayer GRGB and Billinear and Multiresolution support

Agree with above, should be GRBG for the ASI1600MC.



#37 joelin

joelin

    Mercury-Atlas

  • *****
  • topic starter
  • Posts: 2,890
  • Joined: 14 Jan 2008
  • Loc: Saratoga, CA

Posted 14 December 2018 - 12:28 PM

Thanks everyone. I used GRBG and that resolved the screen door pattern.

 

However, a blink test of my uncalibrated lights still shows dark spots that are fixed in the frame. That got me thinking, shouldn't darks calibrate that out?

 

I did a blink test of my darks and I clearly see areas where the pixels are stuck at pure white:

 

https://www.youtube....eature=youtu.be

 

Is that normal? 

 

BTW, the darks are not debayered


Edited by joelin, 14 December 2018 - 12:29 PM.


#38 choward94002

choward94002

    Surveyor 1

  • *****
  • Posts: 1,657
  • Joined: 25 Jul 2016
  • Loc: Central AZ

Posted 14 December 2018 - 12:37 PM

You're conflating "darks" and "dithering" ... both solve different issues (and are discussed in some excellent books out there) ...

 

"Darks" will subtract the dark current accumulated in each pixel well from the total well contents.  Imagine that you've got a bunch of cups laid out in your driveway and someone turns on a lawn sprinker over them for a few minutes.  Some of the cups will have no water in them, some of them will have a little water and some will have a lot of water ... that's "dark current" and is what will result in Fixed Pattern Noise, or "rain" patterns in the images.  Luckily those pixel wells have repeatable levels of dark current accumulation at the same temperature so if we take a "dark" at that temperature we can get a reading of the dark current for each pixel well and subtract that.  Note that this processing is on a SINGLE IMAGE ...

 

Pixels can also either fill completely (white) or never fill (black), and this too is repeatable ... it's like one of the cups on the driveway has a cap covering it or has its bottom cut out.  These pixels too have a repeatable level and we use dithering to fix it ... shifting the image by a few pixels with each image, so the software will recognize that there's a white (or black) pixel that's moving around and removing it from a stack.  This processing is on a STACK of single images ...

 

So, we take images with a dither pattern and take darks to apply to each image to remove the FPN, then we combine those dithered images in a stack to remove the hot/ cold pixels ...



#39 joelin

joelin

    Mercury-Atlas

  • *****
  • topic starter
  • Posts: 2,890
  • Joined: 14 Jan 2008
  • Loc: Saratoga, CA

Posted 14 December 2018 - 01:37 PM

You're conflating "darks" and "dithering" ... both solve different issues (and are discussed in some excellent books out there) ...

 

"Darks" will subtract the dark current accumulated in each pixel well from the total well contents.  Imagine that you've got a bunch of cups laid out in your driveway and someone turns on a lawn sprinker over them for a few minutes.  Some of the cups will have no water in them, some of them will have a little water and some will have a lot of water ... that's "dark current" and is what will result in Fixed Pattern Noise, or "rain" patterns in the images.  Luckily those pixel wells have repeatable levels of dark current accumulation at the same temperature so if we take a "dark" at that temperature we can get a reading of the dark current for each pixel well and subtract that.  Note that this processing is on a SINGLE IMAGE ...

 

Pixels can also either fill completely (white) or never fill (black), and this too is repeatable ... it's like one of the cups on the driveway has a cap covering it or has its bottom cut out.  These pixels too have a repeatable level and we use dithering to fix it ... shifting the image by a few pixels with each image, so the software will recognize that there's a white (or black) pixel that's moving around and removing it from a stack.  This processing is on a STACK of single images ...

 

So, we take images with a dither pattern and take darks to apply to each image to remove the FPN, then we combine those dithered images in a stack to remove the hot/ cold pixels ...

Yes, but the bigger question is: I never dithered before with my 4 other cameras (Sony A7S, Nikon D600, Canon 6D, ASI1600MM) and I got much, much better results than the ASI1600MC? Why?

 

This leads me to think 

1) I made a blunder in processing. I did correct the debayer settings, but the streaks continue. 

2) There is a defect in my camera, such as having an excessive number of hot/cold pixels.

Could I test #2 by comparing darks from my older cameras with this one?



#40 choward94002

choward94002

    Surveyor 1

  • *****
  • Posts: 1,657
  • Joined: 25 Jul 2016
  • Loc: Central AZ

Posted 14 December 2018 - 01:48 PM

Darks will be totally different from camera to camera, even temperature to temperature makes a big difference ... don't do that ...

 

Could be a defect in the camera, but maybe not ... Jon Rista is a go-to guy for the ASI 1600 here, that's a question better directed to him ...

 

Walking FPN, or "rain" in the exposure in my experience is a function of a non-optimal dark processing step (remember, the darks need to be VERY close to the temperature of the image, 1-2C close or you'll get some FPN) and can be minimized by aggressive dithering (10-15 pixel RA and DEC) in the stacking ... do a CN search for "walking rain" and "FPN" and "dither" for some threads on detecting and dealing with that ...



#41 joelin

joelin

    Mercury-Atlas

  • *****
  • topic starter
  • Posts: 2,890
  • Joined: 14 Jan 2008
  • Loc: Saratoga, CA

Posted 14 December 2018 - 03:01 PM

I did use temperature matched darks thanks to a cooler camera.

The DSLRs has darks but not temperature matched and I had better results. Why?

How would I detect a defect in my camera?

Edited by joelin, 14 December 2018 - 03:02 PM.


#42 fmeschia

fmeschia

    Vanguard

  • *****
  • Posts: 2,173
  • Joined: 20 May 2016
  • Loc: Mountain View, CA

Posted 14 December 2018 - 03:04 PM

Are your lights calibrated with a master bias? Are the same hot pixels present in the master bias?

#43 42itous1

42itous1

    Viking 1

  • *****
  • Posts: 920
  • Joined: 21 May 2016
  • Loc: Wilton, CA

Posted 14 December 2018 - 03:24 PM

It is my understanding you should not take bias frames--use dark flats instead.



#44 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 26,034
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 14 December 2018 - 03:40 PM

Yes, but the bigger question is: I never dithered before with my 4 other cameras (Sony A7S, Nikon D600, Canon 6D, ASI1600MM) and I got much, much better results than the ASI1600MC? Why?

 

This leads me to think 

1) I made a blunder in processing. I did correct the debayer settings, but the streaks continue. 

2) There is a defect in my camera, such as having an excessive number of hot/cold pixels.

Could I test #2 by comparing darks from my older cameras with this one?

My guess is your DSLR subs are just much more deeply exposed. People tend to use the "1/3 rule" for determining exposure with DSLRs, which is often much more than enough, yet it can also lead to more clipped stars and more heavily clipped stars (which seems to be apparent in your examples.) 

 

IMO, your ASI1600 data actually looks better from a detail standpoint, as well as from a color accuracy standpoint. Your only issue is the pattern noise. 

 

You have made several mistakes. For one, not dithering, you definitely want to dither, and I would say that is not optional (IMO, with any camera). You also do not appear to be exposing long enough...you said you are using 30 second exposures. Have you tried 60? 90? 120? Aside from dithering and proper calibration, the next best way to deal with DFPN is to bury it with signal, so longer exposures should help. 

 

I am also not quite sure if you are or are not calibrating. Are you? If so, are you sure you are not scaling the darks? Calibration is only effective when it is done properly, and you need to make sure you are both integrating the dark frames properly (without normalization) and subtracting them from the lights without scaling. Good calibration should deal with most of the DFPN, so only any small remnant would require dithering to eliminate. 


  • mikefulb and 42itous1 like this

#45 entilza

entilza

    Soyuz

  • *****
  • Posts: 3,831
  • Joined: 06 Oct 2014
  • Loc: Oakville, ON, Canada

Posted 15 December 2018 - 12:08 AM

Hes using the ASI air and cant dither at moment..

Joelin blink through your old camera data and see how much drift there was then.

I already posted some tips earlier.

#46 joelin

joelin

    Mercury-Atlas

  • *****
  • topic starter
  • Posts: 2,890
  • Joined: 14 Jan 2008
  • Loc: Saratoga, CA

Posted 15 December 2018 - 03:51 AM

Here is the M45 blink using my Nikon D600 data:

 

https://www.youtube....eature=youtu.be

 

The noise pattern seems a bit more random 



#47 joelin

joelin

    Mercury-Atlas

  • *****
  • topic starter
  • Posts: 2,890
  • Joined: 14 Jan 2008
  • Loc: Saratoga, CA

Posted 15 December 2018 - 04:23 AM

My guess is your DSLR subs are just much more deeply exposed. People tend to use the "1/3 rule" for determining exposure with DSLRs, which is often much more than enough, yet it can also lead to more clipped stars and more heavily clipped stars (which seems to be apparent in your examples.) 

 

IMO, your ASI1600 data actually looks better from a detail standpoint, as well as from a color accuracy standpoint. Your only issue is the pattern noise. 

 

You have made several mistakes. For one, not dithering, you definitely want to dither, and I would say that is not optional (IMO, with any camera). You also do not appear to be exposing long enough...you said you are using 30 second exposures. Have you tried 60? 90? 120? Aside from dithering and proper calibration, the next best way to deal with DFPN is to bury it with signal, so longer exposures should help. 

 

I am also not quite sure if you are or are not calibrating. Are you? If so, are you sure you are not scaling the darks? Calibration is only effective when it is done properly, and you need to make sure you are both integrating the dark frames properly (without normalization) and subtracting them from the lights without scaling. Good calibration should deal with most of the DFPN, so only any small remnant would require dithering to eliminate. 

Since I'm using a C8 Hyperstar, my focal ratio is f/2 and according to this: https://www.cloudyni...d-maybe-qhy163/ 30 seconds is plenty long (in fact quite a bit longer than specified) for f/2 and a gain of 139. 

 

In PI, I did not check "Optimize Dark Frames" which I think is what the dark scaling thing you mentioned is. I'm not sure what normalization integration is but I used Average combination and Windsorized Sigma Clipping in the integration. 



#48 ChrisWhite

ChrisWhite

    Cosmos

  • *****
  • Posts: 8,899
  • Joined: 28 Feb 2015
  • Loc: USA

Posted 15 December 2018 - 08:45 AM

Joe,  The ASI 1600 is an FPN machine!  You are pretty much GUARANTEED to get fixed pattern noise if you don't dither frequently and aggressively.  FPN and the ASI 1600 has been discussed ad nauseam here on the forum for the last two years.  You must dither aggressively and frequently with this camera.  One reason is that this camera has ridiculously low thermal noise and read noise.  This is a good thing!  But unlike your DSLR's which "swamp" the building blocks of FPN with deeper exposure and thermal noise, the ASI 1600 requires active management of collection techniques (i.e. Dithering) to prevent the issues you are having. 


Edited by ChrisWhite, 15 December 2018 - 08:45 AM.

  • 17.5Dob, happylimpet, tkottary and 2 others like this

#49 dkeller_nc

dkeller_nc

    Surveyor 1

  • *****
  • Posts: 1,546
  • Joined: 10 Jul 2016
  • Loc: Central NC

Posted 15 December 2018 - 10:00 AM

The reason that your DSLR data doesn't look "as bad" as the ASI1600 is 3-fold.  First, the DSLR data isn't nearly as sharp as it has a couple of filters in front of the sensor that blur the image - blurring an image is one trick we use in Photoshop to mask noise or a fixed pattern.  Secondly, your DSLR data is taken at a much higher image scale than your ASI1600 data - this in and of itself will cause significant blurring of fine details, like correlated FPN.  Finally, the movements of your scope when you were taking the DSLR data was essentially random - that IS dithering.  With your ASI1600, you've got something really weird going on with the guiding, where it appears that the the system doesn't "correct" the pointing of the scope for at least 3 or 4 frames, then brings the guidestar back to its original position.  This is correlated movement, and it's guaranteed to produce a fixed pattern when you stack your images.

 

By the way - since you have PixInsight, you'd be well served to learn the calibration, registration and stacking processes in it and abandon DSS.  Here's a link to the basic steps.



#50 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 26,034
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 15 December 2018 - 03:47 PM

Finally, the movements of your scope when you were taking the DSLR data was essentially random - that IS dithering.  

I have to refute this every time it comes up. tongue2.gif The movements of the scope are not dithering. If the scope moved enough to "dither" you would have stars trailing all over the place in the image. Dithering is not caused by movements of the scope.

 

Further, the drift of the stars across the frame due to imperfect polar alignment or differential flexure, which are also often called "dithering", are not only most definitely NOT dithering...they are  the direct cause of the issue the OP is having: Correlated noise! (Walking noise, raining noise, etc.) 

 

It IS the drift that ultimately allows the FPN to correlate. Once the frames are registered, the stars stop drifting...and now the pattern drifts. When stacked, it is the slight but consistent changes in the position of the pattern that ultimately "correlates" in the stack, giving rise to the streaking in the noise. 

 

So, neither mount movements nor drift are dithering. Mount movements just couldn't be large enough (at a minimum, about 5 pixels, ideally around 10), as if they were then you would have severely blurred data. Drift is the fundamental cause of correlated noise in the first place, and if drift was large enough to "dither" by 5 pixels each frame, then you would have the obvious start of star trails.

 

The idea that there is any such thing as natural dithering is a myth. It has been repeated for years, but it is really not true. The only way to dither such that it can help you randomize the FPN in the registered subs so it averages out in the stack is to actually dither, using a program like PHD or a direct mount dither, and dither by 5-10 pixels every few frames for short (never less often than 3 frames, more often if the frames are longer), and dither RANDOMLY (do not use any kind of pattern dither, like box, or spiral, etc.)


Edited by Jon Rista, 15 December 2018 - 03:48 PM.

  • 17.5Dob, ChrisWhite, choward94002 and 1 other like this


CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics