Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Help with Pixinsight calibration

  • Please log in to reply
14 replies to this topic

#1 SambaChoro

SambaChoro

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 07 Sep 2013
  • Loc: Mohegan Lake, NY

Posted 01 August 2020 - 06:45 PM

Hi all,

 

I need a little help if anyone has experienced this before.

This past week I went out to photograph the Veil Nebula, just to be mocked at by the weather gods and ended up with just 14, 2 minute subs of which I am using 13 (high success rate I guess). 

I also took darks, flats and bias. 25 flats and darks and 60 bias.

I'm preprocessing as I have always done, but this time I get a completely dark image with some bright spots after light calibration. They are useless. If I integrate the lights without any calibration I succeed (but that's not how we do things). The only difference now is that I have only 13 subs instead of the 50 or more I usually have.

I tried different algorithms, tried without darks and without flats. The results are always the same. This never happened to me before (I am not a Pixinsight expert).

You can see the photo below the original FIT file (left) and after calibration (right). They are both boost stretched.  I tried re-taking all flats, darks and bias to no avail. 

I have a ZWO 533 Pro, OSC cooled camera.

Any ideas?

Lights taken at 120 secs, gain 100, cooled to -8 c.

 

I thank in advance,

 

Leo

Attached Thumbnails

  • VeilPix.JPG

  • elmiko likes this

#2 pyrasanth

pyrasanth

    Mercury-Atlas

  • *****
  • Posts: 2,918
  • Joined: 08 Jan 2016

Posted 01 August 2020 - 07:40 PM

Have you tried using the weighted batch pre processing script?- put all the raw files you use into this so you effectively create everything from scratch- including your flats, bias & dark. This will eliminate human error provided you add all the files into the script correctly. Have you tried to debayer just one of your subs & see if you can split it into the RGB components?- there is a lot of testing you can do to try & get a handle on the problem. 

 

Let us know how you get on with the script- be careful to make it aware that your using an OSC camera.

 

If you get no joy- bundle up your files- light, darks & bias & put these on dropbox with the debayer matrix your using and we can take a look for you.


Edited by pyrasanth, 01 August 2020 - 07:42 PM.


#3 SambaChoro

SambaChoro

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 07 Sep 2013
  • Loc: Mohegan Lake, NY

Posted 01 August 2020 - 07:49 PM

Have you tried using the weighted batch pre processing script?- put all the raw files you use into this so you effectively create everything from scratch- including your flats, bias & dark. This will eliminate human error provided you add all the files into the script correctly. Have you tried to debayer just one of your subs & see if you can split it into the RGB components?- there is a lot of testing you can do to try & get a handle on the problem. 

 

Let us know how you get on with the script- be careful to make it aware that your using an OSC camera.

 

If you get no joy- bundle up your files- light, darks & bias & put these on dropbox with the debayer matrix your using and we can take a look for you.

Yes,I tried WBPP. It won't even process it. I forgot what the msg said, should've copied it. Debayer works fine. I did integrate without calibrating them and I got a decent color image considering no calibration and only 13 subs.

 

Leo



#4 pyrasanth

pyrasanth

    Mercury-Atlas

  • *****
  • Posts: 2,918
  • Joined: 08 Jan 2016

Posted 01 August 2020 - 07:51 PM

Yes,I tried WBPP. It won't even process it. I forgot what the msg said, should've copied it. Debayer works fine. I did integrate without calibrating them and I got a decent color image considering no calibration and only 13 subs.

 

Leo

The clue would be in the error- if WBPP won't process the files then there is something fundamentally wrong with the files or your workflow for this batch of files.



#5 SambaChoro

SambaChoro

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 07 Sep 2013
  • Loc: Mohegan Lake, NY

Posted 01 August 2020 - 08:09 PM

The clue would be in the error- if WBPP won't process the files then there is something fundamentally wrong with the files or your workflow for this batch of files.

It must be me then. I took the calibration files all over again and got the same results. Gotta figure this out. 

 

Thank you all

 

Leo



#6 elmiko

elmiko

    Aurora

  • *****
  • Posts: 4,834
  • Joined: 27 Jul 2009
  • Loc: Arizona

Posted 01 August 2020 - 08:21 PM

Have you tried to stretch the image? Use Histogram Transformation to stretch the stack. You also need several more subs than 13. Try to get a couple of hours minimum. Then when you stretch the image it will have more signal.


Edited by elmiko, 01 August 2020 - 08:24 PM.


#7 SambaChoro

SambaChoro

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 07 Sep 2013
  • Loc: Mohegan Lake, NY

Posted 01 August 2020 - 08:42 PM

Have you tried to stretch the image? Use Histogram Transformation to stretch the stack. You also need several more subs than 13. Try to get a couple of hours minimum. Then when you stretch the image it will have more signal.

I'm only doing this because I want to know what I can get from 13 subs. The problem is with calibration. As mentioned I integrated the lights without calibration and I have a photo. I have to figure out what the problem is, as I checked and rechecked my workflow. See the image I got by integrating without calibration below.

 

Leo

Attached Thumbnails

  • Veilmaster.JPG

  • elmiko likes this

#8 SambaChoro

SambaChoro

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 07 Sep 2013
  • Loc: Mohegan Lake, NY

Posted 01 August 2020 - 08:44 PM

I tried WBPP again and it does not integrate. See snapshot below

 

Leo

Attached Thumbnails

  • WBPP.JPG


#9 Peregrinatum

Peregrinatum

    Apollo

  • *****
  • Posts: 1,263
  • Joined: 27 Dec 2018
  • Loc: South Central Valley, Ca

Posted 01 August 2020 - 09:33 PM

I dunno, it just seems like you have so much more control and it's easier to identify problems if you skip WBPP and just learn how to do each step individually, and I don't think it really is all that much more time:

-integrate flat darks

-integrate darks

-calibrate flats

-integrate flats

-cosmetic correction

-debayer

-subframe selector, identify reference frames for registration, weight frames

-register and integrate reference frame

-register subs using the reference frame

-integrate, optimize clipping during integration


  • elmiko likes this

#10 jdupton

jdupton

    Mercury-Atlas

  • *****
  • Posts: 2,581
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 01 August 2020 - 09:41 PM

Leo,

 

   Can you look at a sample of each frame type using the PixInsight Statistics process? I suspect that you somehow have ended up with a major mismatch between the Lights and your calibration masters.-- most likely the Dark frame.

 

   If you look at each frame type, should have ADU values ranked in the following order.

  • Bias Frame -- lowest average ADU. (If you have them)
  • Flat-Dark Frame -- next lowest average ADU. (If you used these.)
  • Dark Frame -- greater ADU than either of the above.
  • Light frame -- greater average ADU than any of the preceding frame types.
  • Flat Frame -- the greatest average ADU of all frame types.

   If you find any frame type out of the above order, you probably have a mismatched Offset from when the frames were taken. This is the most common error.

 

   This can often be caused by using different programs to capture the frames. An example might be taking lights using one program and taking the calibration frames with another. It can also be caused by using different device drivers for some of the frames. For example, you might accidentally use native drivers for some frame types and the ASCOM driver for others.

 

   Check the mean / average ADU value of some raw frame samples to see if anything is amiss. If it is, look back at how you captured all the frames. Were they all taken with the same computer, program, and driver types? Finally, double check the FITs Headers using the FITs Header process in PixInsight. It usually records temperature and camera gain. Make sure those were matched between all the frame types you use also.

 

 

John


  • elmiko likes this

#11 SambaChoro

SambaChoro

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 07 Sep 2013
  • Loc: Mohegan Lake, NY

Posted Yesterday, 05:26 AM

I dunno, it just seems like you have so much more control and it's easier to identify problems if you skip WBPP and just learn how to do each step individually, and I don't think it really is all that much more time:

-integrate flat darks

-integrate darks

-calibrate flats

-integrate flats

-cosmetic correction

-debayer

-subframe selector, identify reference frames for registration, weight frames

-register and integrate reference frame

-register subs using the reference frame

-integrate, optimize clipping during integration

I never use WBPP. Always manual. I only used it this time to see if it would process it as I could not get anything out of it myself.

 

Leo



#12 SambaChoro

SambaChoro

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 07 Sep 2013
  • Loc: Mohegan Lake, NY

Posted Yesterday, 05:30 AM

Leo,

 

   Can you look at a sample of each frame type using the PixInsight Statistics process? I suspect that you somehow have ended up with a major mismatch between the Lights and your calibration masters.-- most likely the Dark frame.

 

   If you look at each frame type, should have ADU values ranked in the following order.

  • Bias Frame -- lowest average ADU. (If you have them)
  • Flat-Dark Frame -- next lowest average ADU. (If you used these.)
  • Dark Frame -- greater ADU than either of the above.
  • Light frame -- greater average ADU than any of the preceding frame types.
  • Flat Frame -- the greatest average ADU of all frame types.

   If you find any frame type out of the above order, you probably have a mismatched Offset from when the frames were taken. This is the most common error.

 

   This can often be caused by using different programs to capture the frames. An example might be taking lights using one program and taking the calibration frames with another. It can also be caused by using different device drivers for some of the frames. For example, you might accidentally use native drivers for some frame types and the ASCOM driver for others.

 

   Check the mean / average ADU value of some raw frame samples to see if anything is amiss. If it is, look back at how you captured all the frames. Were they all taken with the same computer, program, and driver types? Finally, double check the FITs Headers using the FITs Header process in PixInsight. It usually records temperature and camera gain. Make sure those were matched between all the frame types you use also.

 

 

John

I have not done it and I will try when I get back from work later. I've used the same computer and APT to acquire everything.

 

Thank you all,

 

Leo



#13 SambaChoro

SambaChoro

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 07 Sep 2013
  • Loc: Mohegan Lake, NY

Posted Yesterday, 08:43 PM

Leo,

 

   Can you look at a sample of each frame type using the PixInsight Statistics process? I suspect that you somehow have ended up with a major mismatch between the Lights and your calibration masters.-- most likely the Dark frame.

 

   If you look at each frame type, should have ADU values ranked in the following order.

  • Bias Frame -- lowest average ADU. (If you have them)
  • Flat-Dark Frame -- next lowest average ADU. (If you used these.)
  • Dark Frame -- greater ADU than either of the above.
  • Light frame -- greater average ADU than any of the preceding frame types.
  • Flat Frame -- the greatest average ADU of all frame types.

   If you find any frame type out of the above order, you probably have a mismatched Offset from when the frames were taken. This is the most common error.

 

   This can often be caused by using different programs to capture the frames. An example might be taking lights using one program and taking the calibration frames with another. It can also be caused by using different device drivers for some of the frames. For example, you might accidentally use native drivers for some frame types and the ASCOM driver for others.

 

   Check the mean / average ADU value of some raw frame samples to see if anything is amiss. If it is, look back at how you captured all the frames. Were they all taken with the same computer, program, and driver types? Finally, double check the FITs Headers using the FITs Header process in PixInsight. It usually records temperature and camera gain. Make sure those were matched between all the frame types you use also.

 

 

John

What am I looking at? Sorry, I'm not a PI expert as I mentioned. Which one of those fields is the ADU?

 

Thank you all for your help.

 

Leo

Attached Thumbnails

  • masterbias.JPG
  • masterdark.JPG
  • masterflat.JPG
  • light.JPG


#14 jdupton

jdupton

    Mercury-Atlas

  • *****
  • Posts: 2,581
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted Yesterday, 09:17 PM

Leo,

 

   Those are the right statistics. They are just in the default normalized format of 0 to 1. You can either display them in 16 bit ADU by clicking on the "Normalized Real" dropdown and selecting 16 bit or just multiply the Means by 2^16 or 65,536. Your numbers translate to:

  • Bias = ~2,796
  • Dark = ~2,848
  • Light = ~197  (Is this really a single raw light frame or a calibrated light frame?)
  • Flat = ~24,924

   And that shows the problem you are encountering. When you subtract your Master Dark at 2848 ADU from one of your lights at 197 ADU, you end up severely negative. That means all of your data in the lights has been clipped to black. All that will remain after calibration is the hot pixels in your light frames.

 

   So, if your light frame shown above is a raw light frame that has not been calibrated, you need to find out why it does not match your other frame types. Once that is figured out, your calibration will no longer "eat" all of your light frame data. I would have expected a value of something in range of 3,000 to 4,000 ADU for your raw light frames.

 

 

John


Edited by jdupton, Yesterday, 10:03 PM.


#15 SambaChoro

SambaChoro

    Explorer 1

  • -----
  • topic starter
  • Posts: 58
  • Joined: 07 Sep 2013
  • Loc: Mohegan Lake, NY

Posted Today, 05:37 AM

Leo,

 

   Those are the right statistics. They are just in the default normalized format of 0 to 1. You can either display them in 16 bit ADU by clicking on the "Normalized Real" dropdown and selecting 16 bit or just multiply the Means by 2^16 or 65,536. Your numbers translate to:

  • Bias = ~2,796
  • Dark = ~2,848
  • Light = ~197  (Is this really a single raw light frame or a calibrated light frame?)
  • Flat = ~24,924

   And that shows the problem you are encountering. When you subtract your Master Dark at 2848 ADU from one of your lights at 197 ADU, you end up severely negative. That means all of your data in the lights has been clipped to black. All that will remain after calibration is the hot pixels in your light frames.

 

   So, if your light frame shown above is a raw light frame that has not been calibrated, you need to find out why it does not match your other frame types. Once that is figured out, your calibration will no longer "eat" all of your light frame data. I would have expected a value of something in range of 3,000 to 4,000 ADU for your raw light frames.

 

 

John

Hi John,

 

I understand what you are saying.

That was the statistics of a single raw light frame. I attach below the statistics of the integrated master light I did without calibration in 16-bit mode. Now I have R/G/B channels instead of the just K channel. How to compare?

And how to fix it? I don't understand how the darks ended up with higher ADU numbers than the lights. Is it a simple matter of quantity? Is it because I have more darks than lights? I'm sorry, but the light bulb did not go off in my head yet.  

The only difference between this session and any other I ever had is the short amount of lights I have due to changing weather. My workflow is the same and this never happened before.

 

Thank you,

Leo

Attached Thumbnails

  • masterveil.JPG



CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics