Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Subs Won't Calibrate in PI, Files Corrupted?

  • Please log in to reply
18 replies to this topic

#1 Sparks-M16

Sparks-M16

    Explorer 1

  • -----
  • topic starter
  • Posts: 91
  • Joined: 28 Oct 2020

Posted 30 November 2021 - 12:44 PM

This happened to me once before several months ago, and it's happened again.  I collected 6 hours of data two nights ago and the files don't seem to want to calibrate in Pixinsight.  I imaged again last night, and no problems.  I've tried everything I can, but either I'm doing something dumb, or maybe the files are somehow corrupted.  Before tossing the data, I thought I'd at least see if someone can help or provide insight.  Two fits files are linked below, the second one below calibrates fine, but the first fails to give anything, it's like an un-stretched image or something.  Any help is appreciated!

 

https://www.dropbox....60.00.fits?dl=0

 

https://www.dropbox....80.00.fits?dl=0



#2 OldManSky

OldManSky

    Fly Me to the Moon

  • *****
  • Posts: 5,967
  • Joined: 03 Jan 2019
  • Loc: Valley Center, CA USA

Posted 30 November 2021 - 01:24 PM

I had no problem opening your "problem" FITS file in ASTAP and PI.  ASTAP screenshot shown below.

Data looks fine (it looks a bit noisy because of the way ASTAP stretches, but that's normal).

I would suggest looking at what you're calibrating WITH, and what error PI puts up -- that might help understand what's going on.

 

 

Attached Thumbnails

  • openfits.jpg


#3 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 27,006
  • Joined: 27 Oct 2014

Posted 30 November 2021 - 01:26 PM

There's nothing at all wrong with the first sub.  It's quite nice.

 

The problem is no doubt either with your processing or with the calibration frames.

 

One thing to try.  Instead of using BPP or WBPP, use the individual steps.  I've had that solve problems, and, in any event, it can be quite diagnostic.

 

The following is absolutely true (from Mastering PixInsight):

 

"The first time we go through the process of executing each step individually takes forever.  This has happened to everyone on Earth who's gone through this.  After we've done it a few times it makes no difference (I'd say little difference) which way we do it.  This too has happened to  everyone on Earth who's had the patience."

 

<smile>


Edited by bobzeq25, 30 November 2021 - 01:32 PM.


#4 Sparks-M16

Sparks-M16

    Explorer 1

  • -----
  • topic starter
  • Posts: 91
  • Joined: 28 Oct 2020

Posted 30 November 2021 - 01:44 PM

Right, the subs both seem fine.  But when I try to calibrate the first one, it doesn't seem to want to.  Here are the calibration files, a master dark and master flat.  I forgot to include them with the first post.  I have no clue why the first sub won't calibrate.

 

https://www.dropbox....0.00s.xisf?dl=0

https://www.dropbox...._Mono.xisf?dl=0



#5 jdupton

jdupton

    Soyuz

  • *****
  • Posts: 3,941
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 30 November 2021 - 01:47 PM

Sparks-M16,

 

   As Paul and Bob have said, there is nothing wrong with either of the subs you uploaded. There is a difference between the two. The first sub shows as being about 600 ADU darker than the second for some reason. Depending on how you are performing image calibration, that me may be making a big difference. 

 

   You should upload your Master Dark frame that is being used to calibrate those lights. Also tell us more about whether you are using BPP, WBPP, or manual ImageCalibration to calibrate the Light frames. What exactly is failing? Is the PI Process Console reporting some subs failed or are they just not working well in later stages of processing?

 

   If the Master Dark has a median ADU value of greater than about 1048 ADU, then the much of the first sub will be clipped whereas the second may calibrate just fine. Seeing the Master Dark will help us help you.

 

 

John


Edited by jdupton, 30 November 2021 - 02:07 PM.

  • bobzeq25 and OldManSky like this

#6 Sparks-M16

Sparks-M16

    Explorer 1

  • -----
  • topic starter
  • Posts: 91
  • Joined: 28 Oct 2020

Posted 30 November 2021 - 01:51 PM

I tried WBPP and manual ImageCalibration, same result with both techniques.


  • bobzeq25 likes this

#7 jdupton

jdupton

    Soyuz

  • *****
  • Posts: 3,941
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 30 November 2021 - 02:00 PM

Sparks-M16,

 

   OK, as I suspected, the Master Dark is bad with respect to calibrating the first sub you uploaded. The Master Dark has a median ADU value of about 1283. That is greater than the median ADU of 1048 for the first sub. Subtracting that Master Dark will remove most of the data from the sub.

 

   There are a few possibilities as to where things went wrong. The Dark frames may have been shot at a different Offset. (The eGain does match the Lights.) Uploading a single original Dark frame might help verify the reason the Darks don't match the first sub from your prior session. The generation of the Master Dark could also have gone awry. And finally, something about the first session of taking Lights may have been different although, at the moment, I cannot think of anything that might cause them to be darker than the subsequent session.

 

 

John


Edited by jdupton, 30 November 2021 - 02:03 PM.

  • psandelle, bobzeq25 and OldManSky like this

#8 Sparks-M16

Sparks-M16

    Explorer 1

  • -----
  • topic starter
  • Posts: 91
  • Joined: 28 Oct 2020

Posted 30 November 2021 - 02:06 PM

John, so I don't have the original individual dark frames, I deleted them once I created the master dark, which was several months ago.  Note that the two subs were shot two nights ago, and last night.  Perhaps I did something in NINA two nights ago with that night's shooting, but I don't know what it would have been.



#9 jdupton

jdupton

    Soyuz

  • *****
  • Posts: 3,941
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 30 November 2021 - 02:14 PM

Sparks-M16,

 

   I understand. In that case, can you simply shoot a single new Dark frame with NINA set up just as you did for last night's session? Upload the fresh new Dark so we can see what it should look like.

 

   If the new single Dark frame resembles the Master Dark, then the earlier Light frame session deserves further investigation. If the new Dark frame does not resemble the existing Master, then that may lead down a different path.

 

 

John



#10 Sparks-M16

Sparks-M16

    Explorer 1

  • -----
  • topic starter
  • Posts: 91
  • Joined: 28 Oct 2020

Posted 30 November 2021 - 02:24 PM

I'll take a dark today and upload it.  I'm suspicious I must have done something shooting the lights two nights ago, since the master dark has been working fine for months, including last night.  Anyway, single fresh dark coming...



#11 Sparks-M16

Sparks-M16

    Explorer 1

  • -----
  • topic starter
  • Posts: 91
  • Joined: 28 Oct 2020

Posted 30 November 2021 - 03:22 PM

Here's a fresh dark, just taken.  300s, -10C, bin 2x2.

 

https://www.dropbox.....00s_.fits?dl=0



#12 jdupton

jdupton

    Soyuz

  • *****
  • Posts: 3,941
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 30 November 2021 - 04:10 PM

Sparks-M16,

 

   Your new single Dark frame is a very good match for the Master Dark you have been using. That means that the problem is with the session of Light frames from several nights ago.

 

   I cannot think of anything that would cause that one set of Light frames to be darker than those taken the next night. The usual suspect would be that the Offset had changed. However, the FITs Header shows that both Gain and Offset match between the sessions and yet one session's subs came out dark versus the next night's session where the levels are where they would be expected.

 

   The only differences between the two nights of shooting appears to be ambient temperature and (as a result) focus position. Neither of those parameters should cause a difference in average brightness. You might think that a more transparent night could cause the overall image to be darker but that cannot be the cause here since the Lights dropped in brightness enough to be darker than a matching Dark frame. A dark sky background cannot be less bright than a Dark where no light hits the sensor at all.

 

   Is there a possibility that you used a different connection method to the camera on those two nights? It is a wild outlandish thought but I am wondering if you used the ASCOM driver on the earlier session and then went back to the native ZWO (SDK) driver for the other nights? For that to cause this, it would also require that a subtle bug (or bugs) be present in the version of NINA you are using. ASCOM does not report the Offset for the camera to the capture software. If the ASCOM driver were set for a lower Offset, --AND-- your version of NINA defaulted to the last used / set native driver Offset for recording the FITs header data, then the darker subs might result. However, NINA would normally have reported the camera as "ASI Camera (1)" or similar when the ASCOM connection is used. This line of thought doesn't sound feasible since it would require multiple bugs in just the right places. Such a set of coincidences is very highly unlikely -- akin to stacking a bunch of BBs one atop the other. 

 

   How this could have happened is a mystery for me. Does anyone else have a way to explain how one set of subs could have a 600 ADU darker median value even though the FITs Header shows the key parameters as matching? It has me stumped.

 

   I did notice one more thing you should modify going forward. You have been using a USB setting of 40 for the Light frames but appear to be using a USB setting of 80 for the Dark frames. All frames should ideally use the same setting. This should not account for the Light frame differences since they appear to be the same in both Light sessions. However, for the best calibration, you should standardize on one setting (I use 40) for everything -- Light, Dark, Flat, Flat-Dark (or Bias). It has been reported that is can sometimes make a slight difference in the amount of Dark Current in the frames.

 

 

John



#13 Sparks-M16

Sparks-M16

    Explorer 1

  • -----
  • topic starter
  • Posts: 91
  • Joined: 28 Oct 2020

Posted 30 November 2021 - 04:41 PM

Thanks John, I'll keep an eye out here for any further insights by folks.  As I mentioned this happened one other time on a night's shoot like a month ago, I may go back and find that night, and see what I can deduce. 

 

Regarding connections, it may not be a coincidence that over the last several nights I was changing connections associated with my new filter wheel, which houses 2" filters.  It took me two nights of trouble-shooting to finally figure out that putting only three 2" filters in a 7-filter wheel (4 empty slots) makes the EFW do all kinds of weird things from the physical imbalance, like disconnecting, working for a while, going to the wrong filter, etc.  During my trouble shooting I tried connecting the filter wheel USB direct to the ZWO camera, and a bunch of other things, none of which worked.  Problem solved by putting the filters in positions 1, 3 and 5 instead of 1, 2 and 3.  I have no idea what changes I made when, there were so many trials. 

 

Anyway, I won't be making any changes from last night, where all the files calibrated fine.



#14 OldManSky

OldManSky

    Fly Me to the Moon

  • *****
  • Posts: 5,967
  • Joined: 03 Jan 2019
  • Loc: Valley Center, CA USA

Posted 30 November 2021 - 04:55 PM

Regarding the median difference John noted...is it possible the first FITs file has already been dark-subtracted?



#15 jdupton

jdupton

    Soyuz

  • *****
  • Posts: 3,941
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 30 November 2021 - 04:57 PM

Sparks-M16,

 

   There is one last thing I would like you to check. Report back once you have tried this.

 

   Connect the USB to the camera as usual. In NINA, select the ASCOM driver but do not press the connect (Power) button yet. With the camera physically connected, select the NINA set-up (Gear) icon. This will bring up the ASCOM driver set-up dialog box. Make sure the "Advanced" option is checked. Look at (but don't change) the Offset parameter.

 

   Let us know what the Offset is set to in the ASCOM driver. If it is set to 10 rather than the 20 you have been using, that might explain a difference of 600+ ADU between Light sessions.

 

 

John



#16 Sparks-M16

Sparks-M16

    Explorer 1

  • -----
  • topic starter
  • Posts: 91
  • Joined: 28 Oct 2020

Posted 30 November 2021 - 05:26 PM

John, see below.

Attached Thumbnails

  • screencap.jpg


#17 Sparks-M16

Sparks-M16

    Explorer 1

  • -----
  • topic starter
  • Posts: 91
  • Joined: 28 Oct 2020

Posted 30 November 2021 - 05:28 PM

Regarding the median difference John noted...is it possible the first FITs file has already been dark-subtracted?

It's a fits file, straight from the camera, zero processing, calibration, etc.



#18 jdupton

jdupton

    Soyuz

  • *****
  • Posts: 3,941
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 30 November 2021 - 05:31 PM

Sparks-M16,

 

   OK, thanks. That answers the two questions I had above. First, the ASCOM Offset is the same as the native driver Offset you were using. It also confirms that NINA is picking up the correct camera name to include in the FITs Header. NINA was not using the ASCOM driver for the oddball subs.

 

   I got nothing, now... We know why the earlier subs don't calibrate but no have explanations as to why the subs were so dark.

 

 

John


Edited by jdupton, 30 November 2021 - 05:32 PM.

  • OldManSky likes this

#19 jdupton

jdupton

    Soyuz

  • *****
  • Posts: 3,941
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 30 November 2021 - 09:10 PM

Sparks-M16,

 

   I have one more follow-up regarding your wayward subs.

 

   If you wish to go to the effort, you may be able to recover the data from those subs to add to your project. I played with the two Light subs you had uploaded and found that the key difference between them seems to be a simple offset in brightness. If anything, the "bad" sub has higher contrast than the "good" sub. If you want to play with the full set, here is something you can try.

  • Create an Image Container and load all of the "bad" subs into it.
  • Set the output template to be "&filename;_adj&extension;"
  • Set the output directory as desired to hold the modified subs.
  • Drag the New Instance icon (lower left sideways triangle) to the PI workspace.
    You can then close the Image Container window.
     
  • Open the PixelMath Process.
  • Enter the following equation in the RGB/K section.
    $T + 608/65535
  • Drag the New Instance icon of PixelMath to the PI workspace and close the PixelMath window.
     
  • Now drag the Image Container and drop it onto the PixelMath Process icon on the PI workspace.
    This will add 608 ADU to each and every one of the "bad" subs and write a new version into the output directory you specify. They will have their original names with an "_adj" suffix.
     
  • Run these subs through Image Calibration just as you would do for all other subs in this project. (I would use a high Output Pedestal (like 800 ADU) in the Image Calibration process for all of your subs in the project.)
  • Inspect a few samples to see what they look like. You can even do an ImageIntegration of these alone just to see how well they stack.
     
  • If all goes well, just add these "adjusted" subs in with all the other "good" subs as part of your overall project.

   I did notice that after adjusting the "bad" sub, that it appeared to have a higher contrast under an STF screen stretch. The statistics showed the the adjusted sub had a smaller / tighter standard deviation than the "good" subs. I feel like that may have some meaning and may be a clue as to what happened to these. However, I don't fully understand the ramifications of the decreased standard deviation of the "bad" subs.

 

 

John


Edited by jdupton, 30 November 2021 - 10:08 PM.



CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics