Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

PI Stacking Oddity

Astrophotography Imaging Software
  • Please log in to reply
31 replies to this topic

#1 Yooper_

Yooper_

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 22 May 2020
  • Loc: Upper Peninsula of Michigan

Posted 03 May 2021 - 07:19 PM

I had a brief night and imaged M101. I've been running through Adam Blocks tutorials on PI so I followed his Quick Start. My stack ended up oddly colored. When I ran PhotometricColorCalibration I see two distinct plots of data. 

 

oSWoSAH.png

rQZ5XBJ.png

 

I've been bouncing my head off of it so I took the same data and plopped it into DSS and ended up with this. (This is AutoStretched in PI)

 

gElfGrn.png

 

The PhotometricColorCalibration looks great and I can totally tweak the stack to get what I'm looking for.

 

The question I have, how can I troubleshoot where I went wrong in my PI process?


  • elmiko likes this

#2 N1ghtSc0p3

N1ghtSc0p3

    Viking 1

  • *****
  • Posts: 508
  • Joined: 19 Feb 2011
  • Loc: Texas

Posted 03 May 2021 - 07:37 PM

If you applied the STF (ScreenTransferFunction) - aka an "auto stretch" in PI, did you unlink the channels first? (Click the chain link button in the upper left of the STF window. You do NOT want that button highlighted or 'active'.)


  • dswtan, pfile and Juno18 like this

#3 Yooper_

Yooper_

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 22 May 2020
  • Loc: Upper Peninsula of Michigan

Posted 03 May 2021 - 07:39 PM

If you applied the STF (ScreenTransferFunction) - aka an "auto stretch" in PI, did you unlink the channels first? (Click the chain link button in the upper left of the STF window. You do NOT want that button highlighted or 'active'.

When unlinked it still has the same odd yellow cast. I guess more then anything the double line on the color calibration has me wondering where I went wrong.



#4 N1ghtSc0p3

N1ghtSc0p3

    Viking 1

  • *****
  • Posts: 508
  • Joined: 19 Feb 2011
  • Loc: Texas

Posted 03 May 2021 - 08:03 PM

Hmmm...can you share the master light that you came up with in PI? Also, what camera was used to collect the subs? Was it a DSLR?



#5 Yooper_

Yooper_

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 22 May 2020
  • Loc: Upper Peninsula of Michigan

Posted 03 May 2021 - 08:30 PM

Hmmm...can you share the master light that you came up with in PI? Also, what camera was used to collect the subs? Was it a DSLR?

https://drive.google...iew?usp=sharing

 

This was an ASI533MC-PRO. 90 second subs, Bortle 2.5 or so. 100 gain, 5 offset.



#6 N1ghtSc0p3

N1ghtSc0p3

    Viking 1

  • *****
  • Posts: 508
  • Joined: 19 Feb 2011
  • Loc: Texas

Posted 03 May 2021 - 08:39 PM

I'll take a look as soon as my WBPP finishes.



#7 Juno18

Juno18

    Mariner 2

  • *****
  • Posts: 252
  • Joined: 07 Dec 2018
  • Loc: Long Beach, Mississippi, USA

Posted 03 May 2021 - 08:46 PM

I downloaded your data and after linear fit and channel combination I see what you are talking about.  A strong green bias. I also use a ASI 533 mcp and use an IDAS LPS D-1 filter which impacts the color bias, but it clears up when I run these processes (earlier). 

 

Are you stacking with WBPP? Are you using the RGGB bayer pattern?



#8 N1ghtSc0p3

N1ghtSc0p3

    Viking 1

  • *****
  • Posts: 508
  • Joined: 19 Feb 2011
  • Loc: Texas

Posted 03 May 2021 - 08:48 PM

Ok, when you debayered your lights, what method did you choose? Was it set to auto, RGGB, etc?

 

The reason I ask is that might account for the overwhelming green cast. I did a little research on the 533MC Pro, and found this:

 

https://forums.sharp...pic.php?t=2503:

 

"I investigated the Bayer pattern further for the ZWO ASI533mc. I placed red electrical tape over the end of the sensor and did a screen capture to a fits file. Then did the same with blue tape. SharpCap was displaying the proper colors. I found that in both Fitswork and Pixinsight the Debayer pattern needed to get the correct color was GRBG. The Auto setting for the Bayer/mosaic pattern in Pixinsight did not work and gave the wrong color."



#9 N1ghtSc0p3

N1ghtSc0p3

    Viking 1

  • *****
  • Posts: 508
  • Joined: 19 Feb 2011
  • Loc: Texas

Posted 03 May 2021 - 08:48 PM

Ha! Juno18 beat me to it by 2 mins!!


  • Juno18 likes this

#10 N1ghtSc0p3

N1ghtSc0p3

    Viking 1

  • *****
  • Posts: 508
  • Joined: 19 Feb 2011
  • Loc: Texas

Posted 03 May 2021 - 08:50 PM

Also, did you check out an individual light frame prior to pre-processing? Could you share one of those as well? That might help us determine the issue.



#11 jdupton

jdupton

    Gemini

  • *****
  • Posts: 3,258
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 03 May 2021 - 10:07 PM

Yooper_,

 

   I downloaded your image and immediately see two issues. 

  1. The image will need to be cropped before application of PCC.
    There are areas of non-overlap of the raw images. This is often normal and is due to a slow drift of the images from frame to frame. Once stacked, some areas of the stacked image have little or no data. (That depends on which frame was used as the Reference Frame when stacking. It is not a problem but does cause weak areas in the stack.)

    Because of the lack of good data in some areas, the two linear fit rows of pixel groups in the PCC output represents the populations of pixels with lots of good data and the populations that are very weak and end up fitting a different set of parameters.

    To fix the PCC calibration, you only need to crop away the areas of non-overlap. They are easy to see in the rejection maps so you could play with cropping those and then apply the same crop to the integrated frame.
     
  2. The second issue is that you need to set the color balance of the camera driver. The native (SDK) device driver for ASI cameras defaults to applying really bad color balance settings to the data read from the sensor. The driver then modifies that data in a way that makes color correcting the images very difficult. 

    In the case of your image, the Blue channel has already been clipped. There is no way to recover the data since the device driver ruined it before your capture software could even write it to a file. That blue channel data is gone.

    To correct this issue, you need to go into the detailed camera settings look for the settings for "Color Balance". There will be two -- one R and one B. They default to 95 and 52 (I think, from memory). They need to changed to 50 and 50. Save the changes and they will be saved in the device driver or camera firmware. Once done, you don't have to do this again.

    This change cannot help the data you have previously gathered. That data is now changed from what the sensor saw and cannot be recovered. After making this change in the device driver settings, you will need to also rebuild any Libraries of Bias, Darks,  and FlatDarks you may have created. 

   For more information on the device driver color balance settings issue, read through the following thread and the links in the selected post in that thread.

 

https://www.cloudynights.com/topic/709866-asi533mc-pro-dark-frame-white-pixels/?hl=%2Bwhite+%2Bbalance+%2Bbernd#entry10225548

 

[EDIT]

   Another very thread on this issue can be found at:

https://www.cloudynights.com/topic/708880-asi533mc-pro-weird-color-balance/

[/EDIT]

 

   To get PCC to work with the data you have, first crop the weak data areas from the integrated image, and then run Background_Neutralization on the image. Finally, you can then run PCC and you should get pretty good results although the Blue channel will still be weak (because of the clipped / lost pixels) and may make the color harder to bring out and get just right.

 

 

John


Edited by jdupton, 03 May 2021 - 10:15 PM.

  • pfile and N1ghtSc0p3 like this

#12 bulrichl

bulrichl

    Ranger 4

  • *****
  • Posts: 347
  • Joined: 27 May 2018
  • Loc: La Palma (Canary Islands)

Posted 04 May 2021 - 05:59 AM

John is correct. Because of the color balance settings in the native camera driver, the data in the subframes are scaled, and due to the number format (16-bit unsigned integer), rounding errors occur. Besides, the blue channel is severely clipped.

 

I don't understand why until now apparently nobody reported to ZWO that the scaled data are not only used for displaying the image, but are written to the FITS file. This is a bug, and it was discovered more than a year ago!

 

I am using the ASCOM camera driver, so this is not relevant for me.

 

Bernd



#13 Yooper_

Yooper_

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 22 May 2020
  • Loc: Upper Peninsula of Michigan

Posted 04 May 2021 - 06:19 AM

Wow, thanks folks, I figured it was an improper check box in PI.

 

I'd never have thought to investigate the camera itself. Thank you very much folks! 

 

So to summarize, I need to set the Debayer to GRBG and set the ZWO Driver Color Balance to R = 50, and B=50?



#14 bulrichl

bulrichl

    Ranger 4

  • *****
  • Posts: 347
  • Joined: 27 May 2018
  • Loc: La Palma (Canary Islands)

Posted 04 May 2021 - 06:55 AM

Regarding the correct Bayer/mosaic pattern, can you please upload one light frame in FITS format?

 

Yes, the color balance must be set to WB_R = 50, WB_B = 50, and these settings must be saved.

 

Bernd



#15 Yooper_

Yooper_

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 22 May 2020
  • Loc: Upper Peninsula of Michigan

Posted 04 May 2021 - 07:21 AM

Regarding the correct Bayer/mosaic pattern, can you please upload one light frame in FITS format?

 

Yes, the color balance must be set to WB_R = 50, WB_B = 50, and these settings must be saved.

 

Bernd

Absolutely Bernd, I'm away from those files until later today.



#16 Yooper_

Yooper_

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 22 May 2020
  • Loc: Upper Peninsula of Michigan

Posted 04 May 2021 - 03:51 PM

Regarding the correct Bayer/mosaic pattern, can you please upload one light frame in FITS format?

 

Yes, the color balance must be set to WB_R = 50, WB_B = 50, and these settings must be saved.

 

Bernd

Here we go! A single frame.

 

https://drive.google...iew?usp=sharing



#17 elmiko

elmiko

    Fly Me to the Moon

  • *****
  • Posts: 5,794
  • Joined: 27 Jul 2009
  • Loc: Arizona

Posted 04 May 2021 - 04:00 PM

I had a brief night and imaged M101. I've been running through Adam Blocks tutorials on PI so I followed his Quick Start. My stack ended up oddly colored. When I ran PhotometricColorCalibration I see two distinct plots of data. 

 

oSWoSAH.png

rQZ5XBJ.png

 

I've been bouncing my head off of it so I took the same data and plopped it into DSS and ended up with this. (This is AutoStretched in PI)

 

gElfGrn.png

 

The PhotometricColorCalibration looks great and I can totally tweak the stack to get what I'm looking for.

 

The question I have, how can I troubleshoot where I went wrong in my PI process?

You have to do a Dynamic Background Extraction first. This will remove the colored gradient. Then do a color calibration. Your stack is totally normal.



#18 jdupton

jdupton

    Gemini

  • *****
  • Posts: 3,258
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 04 May 2021 - 05:01 PM

Yooper_,

 

Here we go! A single frame.

 

https://drive.google...iew?usp=sharing

   Looking at the single frame, I don't see any major issues. All of the color channels appear to be well balanced. The difference between the single frame and the integration is rather striking in terms of how normal the single frame appears.

 

   Based on this, I now suspect that there is an issue in the preprocessing methods you used or else very possibly in the calibration of the raw images. Would it be possible for you to post a link to one each of the calibration frames you used? One Dark, one Bias (if used), one Flat, and one FlatDark (if used) would help us see where things may have gone askew.

 

  I note that you mentioned using GRBG as the Bayer pattern. This sensor is normally RGGB unless the capture software changes the RowOrder Keyword that PixInsight uses. 

 

   I used RGGB, to deBayer your single frame, followed by BackgroundNeutralization, PCC, MaskedStretch, and HT. The following is what I got.

 

Yooper_single_frame.jpg

 

   Since this looks good as a single frame, it might help to both see a sample of the calibration frames and an outline of the processes you used leading up to the integrated image of your fist post here. The single frame shows no signs of the severe clipping of the blue channel seen in the integration so I suspect a Dark frame calibration error in the preprocessing.

 

 

John



#19 Yooper_

Yooper_

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 22 May 2020
  • Loc: Upper Peninsula of Michigan

Posted 04 May 2021 - 05:31 PM

 

 

   Since this looks good as a single frame, it might help to both see a sample of the calibration frames and an outline of the processes you used leading up to the integrated image of your fist post here. The single frame shows no signs of the severe clipping of the blue channel seen in the integration so I suspect a Dark frame calibration error in the preprocessing.

 

 

John

Sure, here's some links to the frames.

 

Flat 

Dark

Bias

 

I just loaded the ASI ASCOM drivers and took a shot a single dark frame for comparison.

 

ASI ASCOM Driver Dark

 

The ASICAP settings were for Auto White Balance, the Red channel was set at 50, and the blue at 1. After seeing multiple folks express issues with the ASI driver I think I'll go with the ASCOM. Unless there's a compelling reason not to?


Edited by Yooper_, 04 May 2021 - 05:32 PM.


#20 Yooper_

Yooper_

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 22 May 2020
  • Loc: Upper Peninsula of Michigan

Posted 04 May 2021 - 05:44 PM

As far as the process, I'm following along with Adam Blocks Fast Track. I downloaded the course summary after watching the videos and used it for a reference. Here's the relevant steps I'm following.

 

 

 

Process – Cosmetic Correction (to remove hot pixels) prep for WBPP below
  Assuming One Shot Color Camera (OSC)
- check CFA
- Select “Use Auto Detect” check “Hot Sigma”
- Create New Instance – drag triangle at bottom of window to desktop

Script – Batch Processing/Weighted Batch Preprocessing (WBPP)
Add Bias files
Add Dark files
Add Flat files
Add Light Files
Middle window panel – check “Calibrate Only”
Select “Lights” tab top left
Cosmetic Correction – check “Apply”
Debayer - Select Bayer/Mosaic pattern found above
Global Options – check the following
“CFA images”
“Separate CFA flat scaling factors”
“Save process log”
“Save frame groups on exit”
“Up-bottom FITS”
“Generate rejection maps”
Enter Output Directory (where to store result)
Click on “Diagnostics” button and resolve any issues
Click on “Run” button
 
Process – Star Alignment
Select Reference image (used to align all other frames)
Target Images header - Add target images
Output Images header - Select Output directory
Click on “Apply Global” icon (circle bottom left)
“Blink” images to verify alignment
 

Process – Image Integration
Input Images header – Add files from Star Alignment process
Image Integration header
Combination – Average
Normalization – Additive with scaling
Additive grid size – 16
Weights Noise evaluation
Scale estimator – BMVM
Check – Generate integrated image
Check – Evaluate noise
Check – Automatic buffer sizes
Check Use file cache
Pixel Rejection (1) header
Rejection algorithm – Winsorized sigma clipping
Normalization – Scale + zero offset
Check all but Clip high range and Report range rejection
Pixel Rejection (2) header
Sigma low – 4.000
Sigma high – 2.200
Winsorization cutoff: 5.000
Click on “Apply Global” icon (circle bottom left)
Review combined image and save

Process – Photometric Color Calibration
Image Parameters – header
May be populated from FITS header
Click on “Apply Global” icon (circle bottom left)
The result is a graph showing measured color indices of stars comparing star catalog data with our data
Graphed results should show many stars centered on sloped lines (good color calibration)
Do a linked screen stretch using Screen Transfer Function window
 

Process – Dynamic Background Extraction
Do a “super stretch” (“nuclear” button + shift key) to see gradients for removal
Click on image (will get cross-hairs on image)
Model Parameters (1) – header
Tolerance – 5.000
Shadows Relaxation – 3.000
Smoothing Factor – 0.600
Sample Generation – header
Default Sample Radius – 20
Samples per row – 10
Minimum Sample Weight – 0.750
Target Image Correction – header
Correction – subtraction
Check – replace target image
Select (click) areas of image that don’t have nebulosity
Select areas that represent sky values (dark areas)
To run process click “check mark” bottom left of window


#21 jdupton

jdupton

    Gemini

  • *****
  • Posts: 3,258
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 04 May 2021 - 06:20 PM

Yooper_,

 

Sure, here's some links to the frames.

 

Flat 

Dark

Bias

 

I just loaded the ASI ASCOM drivers and took a shot a single dark frame for comparison.

 

ASI ASCOM Driver Dark

 

The ASICAP settings were for Auto White Balance, the Red channel was set at 50, and the blue at 1. After seeing multiple folks express issues with the ASI driver I think I'll go with the ASCOM. Unless there's a compelling reason not to?

   The Dark frame from above is bad. Perhaps it had a light leak. It is probably the cause of the issue you were having. The ASCOM Dark frame looks much better. I'll know more once I get to play with the frames some more.

 

   One quick sanity test you can perform when you have calibration issues is to look at the ordering of the average ADU of a sample of each frame type. They should appear in the following order or average brightness. Any deviation can be a sign of a bad set of calibration frames.

  • Bias Frames should the lowest median ADU.
    (Yours is 196.)
  • Dark Frames should have the next lowest median ADU.
    (Your first Dark Frame is 2796 ==> Bad)
    (Your ASCOM Dark Frame is 200 ==> about right for a Bias of 196 ADU.)
  • Your Light Frame should be higher than either the Bias or Dark.
    (Your Light Frame is 280.)
  • Your Flat Frames should have the highest Median ADU.
  • (Your Flat Frame is 23032 ==> a good mid-histogram value.)

   I'll update in another post once I look at how well the calibration frames perform their functions.

 

 

John


Edited by jdupton, 04 May 2021 - 06:44 PM.


#22 Yooper_

Yooper_

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 22 May 2020
  • Loc: Upper Peninsula of Michigan

Posted 04 May 2021 - 06:35 PM

Yooper_,

 

   The Dark frame from above is bad. Perhaps it had a light leak. It is probably the cause of the issue you were having. The ASCOM Dark frame looks much better. I'll know more once I get to play with the frames some more.

 

   One quick sanity test you can perform when you have calibration issues is to look at the ordering of the average ADU of a sample of each frame type. They should appear in the following order or average brightness. Any deviation can be a sign of a bad set of calibration frames.

  • Bias Frames should the lowest median ADU.
    (Yours is 196.)
  • Dark Frames should have the next lowest median ADU.
    (Your first Dark Frame is 2796 ==> Bad)
    (Your ASCOM Dark Frame is 200 ==> about right for a Bias of 200 ADU.)
  • Your Light Frame should be higher than either the Bias or Dark.
    (Your Light Frame is 280.)
  • Your Flat Frames should have the highest Median ADU.
  • (Your Flat Frame is 23032 ==> a good mid-histogram value.)

   I'll update in another post once I look at how well the calibration frames perform their functions.

 

 

John

Excellent, thanks John!

 

Simple question, how do I evaluate the ADU of an image? edit - Got it, Statistics.


Edited by Yooper_, 04 May 2021 - 07:05 PM.


#23 jonnybravo0311

jonnybravo0311

    Surveyor 1

  • *****
  • Posts: 1,552
  • Joined: 05 Nov 2020
  • Loc: NJ, US

Posted 04 May 2021 - 07:27 PM

It is definitely the bad dark frames causing your problem. On the left is the result of manual calibration (light - dark) / (flat - bias) using the dark frame, on the right is manual calibration light / (flat - bias) without.

 

med_gallery_347158_15202_1091537.png

 

Both have the same unlinked channel STF applied.


  • jdupton likes this

#24 KTAZ

KTAZ

    Surveyor 1

  • *****
  • Posts: 1,581
  • Joined: 09 Apr 2020
  • Loc: Scottsdale, AZ

Posted 04 May 2021 - 07:37 PM

As far as the process, I'm following along with Adam Blocks Fast Track. I downloaded the course summary after watching the videos and used it for a reference. Here's the relevant steps I'm following.

As others have stated, even after correcting your darks you ALWAYS should do background extraction and neutralization BEFORE you color calibrate.


  • elmiko likes this

#25 jdupton

jdupton

    Gemini

  • *****
  • Posts: 3,258
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 04 May 2021 - 07:48 PM

Yooper_,

 

   OK, I have initial results of my testing of the calibration frames.

 

   In answer to your question about finding the average ADU of an image, you only have to open up the Statistics Process in PixInsight (Process | Image | Statistics).

 

   I was worried about using the ASCOM driver generated Dark Frame you took. This is not usually recommended. Any time you calibrate Light Frames, you must exactly match Exposure, Gain, Offset, Temperature, Device Driver, and Capture software for all frames. Any mismatch can and often does cause problems. That is the case here. The ASCOM driver Dark frame, while looking to have about the right median ADU, did not work well at all. It led to massive clipping during calibration.

 

   To get around the bad Dark Frame issue without the use of the ASCOM Dark Frame, I cheated. I took the Bias Frame and simply added 4 ADU to it to simulate a Dark frame. When the Light Frame was calibrated with the Bias_Plus_4 frame and also calibrated with a Master Flat made from subtracting the Bias from the Flat, the result looked pretty good but still suffered from clipping where too many pixels were turned to pure black (0 ADU). This was then fixed by doing the calibration again but adding a 900 ADU output Pedestal to the calibrated result. At this point, the Light was well calibrated and only had 2 pixels remaining that were clipped to black. (I strongly feel these would have been cured by using Cosmetic Correction which I did not run.)

 

   After doing the manual calibration, the weak blue channel showed up again. I traced this back to the Flat Frame. A portion of the weak blue channel seen before may be coming from the color balance settings in the driver but another portion seems to be coming from the light source being slightly deficient in blue light. At least there was not the massive clipping in the blue channel seen before.

 

   With more playing, I think I could come up with a manual process for you to follow to recover the data you have but it would not help you with your learning of PixInsight. The main thing to do is to try to find out what happened to your first set of Dark frames and then secondarily investigate the relative color balance of your Flat light source. (You can do that by taking flats with various light sources, deBayering them and examining the RGB Flat to see how the three color channels relate to one another. You want each to be roughly centered in the histogram although they do not need to be perfectly aligned. Just ensure that no color channel is too close to the left or right edge of the histogram and that the "free space" on the left and right is about equal. (In other words, get the average ADU into the 30,000 to 34,000 ADU range.)

 

 

John




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics





Also tagged with one or more of these keywords: Astrophotography, Imaging, Software



Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics