Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Pixinsight Image Calibration - darks as flats or what?

  • Please log in to reply
11 replies to this topic

#1 txo

txo

    Explorer 1

  • -----
  • topic starter
  • Posts: 65
  • Joined: 10 Jan 2019
  • Loc: EU, Prague

Posted 15 August 2019 - 03:12 PM

Hi there,

 

I always had problems calibrating out the starburst typical of 183M chips. Finally I managed to master it by playing with offsets during frame capturing of frames and manually tweaking pedestal during calibration itself. But now something very strange happened. Dark subtraction suddenly stopped working completely or it subtracts very little of the starburst. I tried literally every possible combination of both dark integration settings and calibration settings with no avail. Then I for giggles tried to use master dark as master flat and voila, it calibrated the image correctly. Any idea what might be going on? I'm completely lost. thx.

 

Camera: Altair Astro Hypercam 183M Cooled

 

Darks and Lights:

700s exposure time

32 black offset

378 gain

8C temperature - I couldn't go cooler because of dew buildup

Ha filter

Darks Integration:

Combination method: average

Weights: don't care

No Normalization

Rejection - all combinations and no rejection

I even tried only one dark subframe as master dark and it behaves the same as integrated one.

Callibration:

Calibrate off

Optimize on, threshold 3.0

When I use dark as flat I have to set pedestal to 50 in order to get it work properly. Everything has been shot with offset 32 however. Bu I'm already used to manual pedestal tweaking.



#2 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5291
  • Joined: 14 Jun 2009

Posted 15 August 2019 - 05:38 PM

yeah that doesn't make sense (dark as flat)... what you want is dark optimization turned off if you have amp glows. the dark duration must exactly match the light duration when optimization is off.

 

rob


  • RossW likes this

#3 txo

txo

    Explorer 1

  • -----
  • topic starter
  • Posts: 65
  • Joined: 10 Jan 2019
  • Loc: EU, Prague

Posted 16 August 2019 - 01:04 AM

yeah that doesn't make sense (dark as flat)... what you want is dark optimization turned off if you have amp glows. the dark duration must exactly match the light duration when optimization is off.

 

rob

When I turn optimization off the resulting image is completely dark with a few contrasty specks where the most bright stars were. The darks and flats are taken with exactly same parameters.



#4 txo

txo

    Explorer 1

  • -----
  • topic starter
  • Posts: 65
  • Joined: 10 Jan 2019
  • Loc: EU, Prague

Posted 16 August 2019 - 08:10 AM

yeah that doesn't make sense (dark as flat)... what you want is dark optimization turned off if you have amp glows. the dark duration must exactly match the light duration when optimization is off.

 

rob

And btw. Dark as flat does make sort of sense. Both are supposed to subtract errors superimposed onto the image by two separate processes. In fact I do not understand why pixinsight even differentiate between them they are both calibration frames each to correct a different independent problem.

 

This doesn't explain my problem however. 



#5 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5291
  • Joined: 14 Jun 2009

Posted 16 August 2019 - 12:35 PM

no - sorry - darks as flats makes absolutely no sense at all. the image under calibration is *divided* by the flat, while the bias or dark are *subtracted* from the image under calibration.

 

every single program that does calibration of digital images distinguishes between darks and flats. this is not a pixinsight thing.

 

almost always, problems like you are having are caused by improper dark acquisition - this is compounded by these new CMOS cameras having user control over the offset and gain. most likely for some reason your dark offset/gain does not match the offset/gain of the lights.

 

some common ways this happens: user error; driver upgrade between light and dark acquisition; capture software upgrade between light and dark acquisition. all of these can result in the darks and lights having different gain/offset.

 

also possible are calibration frames which are too old relative to the lights, or double-bias subtraction during calibration, meaning you used a master bias and a non-calibrated master dark when calibrating the lights. if you have a non-calibrated master dark, then don't specify a master bias.

 

rob


  • kingjamez, Jon Rista and txo like this

#6 MattC867

MattC867

    Mariner 2

  • -----
  • Posts: 223
  • Joined: 18 Jun 2016

Posted 16 August 2019 - 12:49 PM

Can you post the files on dropbox or google drive?



#7 MattC867

MattC867

    Mariner 2

  • -----
  • Posts: 223
  • Joined: 18 Jun 2016

Posted 16 August 2019 - 01:14 PM

Agree with Rob. From your description it does sound like your darks are not exactly the same settings as your lights.

 

For CMOS calibration, due to the amp glow, the darks need to be the exact same gain/offset/length/temperature. A simple subtraction without any pedestal adjustment or scaling should work well if the calibration subs are matched to the lights properly.

 

*Post edited for clarity.


Edited by MattC867, 16 August 2019 - 01:16 PM.


#8 jdupton

jdupton

    Vanguard

  • *****
  • Posts: 2094
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 16 August 2019 - 01:20 PM

txo,

 

   I think I can guess a way to reconstruct your problem. I think the whole issue you are experiencing is due to vastly mismatched calibration frames.

 

   Some points to consider:

  • I may have missed it but I didn't see where you have mentioned the exposure length for your Light Frames.

    You say you are using 700 sec Dark Frames with an Offset set to 32. If your Light Frames are anything other than 700 seconds long and used an Offset other than 32, then Dark Frame calibration will fail. If you subtract a 700 sec Dark Frame from a 15, 30, or 60 second Light Frame, all that will remain is a few of the brighter stars in the field. Everything else will be clipped to zero and lost. This also explains why you see something when you use the Optimize Dark Frame option during calibration. Optimize will adjust the Dark Frame to better match your Light Frame. However, since Amp Glows do not scale well, you are left with residual glow following the optimized Dark Frame subtraction.

    You also say that using a your Dark Frame as a Flat Frame results in no such clipping. This to to be expected. The Dark Frame is reasonably flat. When you divide the Light Frames by a flat Dark Frame, you are essentially dividing by 1 so the frame is unchanged. You would probably notice very little difference between a Light Frame calibrated with a Dark as Flat when compared to an uncalibrated Light Frame.
     
  • To fix the mismatched calibration frame issues above, you need to do the following:

    1) Decide on a Gain and Offset to use with the camera. Set it in the capture program and then stick with it. Do not try to vary either parameter "on the fly".

    2) Take a new set of Dark Frames at the same Gain and Offset Values as above. Do not just take arbitrarily long Dark Frames. Shoot the Dark Frames at the exposure you plan to use. If you always use 30, 60, or 120 seconds exposures, shoot a set of Dark Frames for just those values. You must have Dark Frames that match Your Light Frames exactly in terms of Gain, Offset, Exposure, and Temperature.

    3) When you calibrate the frames in PixInsight, do not use the Optimize Dark Frame option. It cannot be used on a plain Dark Frame from a CMOS camera such as yours. 

      Try picking a Gain and Offset that seems to work for you, shoot a new set of calibration frames, and make sure that your Light Frames are shot at the same Gain, Offset, Exposure and Temperature. Doing those things should get you working again.

 

 

John


Edited by jdupton, 16 August 2019 - 01:23 PM.


#9 txo

txo

    Explorer 1

  • -----
  • topic starter
  • Posts: 65
  • Joined: 10 Jan 2019
  • Loc: EU, Prague

Posted 16 August 2019 - 02:23 PM

Thank you all guys.

 

Darks and lights were taken with exactly the same parameters except very few lights were take at 12C temp instead of 8C. 

 

Only two differences between lights and darks cross my mind. Lights were taken in APT because it has superior framing capabilities and darks were taken in Sharpcap since I love the interface more. Second difference was that light were taken without active cooling (fan only) due to fogging issues. The temperature was pretty stable at 8C however the whole time. When I was taking darks I used cooling because I was doing it at room temperature and I wanted to do them at 8C as well.

 

I attach master dark, few dark subs (I took 40 of them) and few lights.

 

master dark - https://1drv.ms/u/s!...8DTOg?e=r2XmLZ 

dark subs - https://1drv.ms/u/s!...SFSwUg?e=2D5dBj

lights - https://1drv.ms/u/s!...eIByTw?e=9oyCup



#10 jdupton

jdupton

    Vanguard

  • *****
  • Posts: 2094
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 16 August 2019 - 03:40 PM

txo,

 

   The problem of mismatched calibration frames is immediately obvious from the frames you posted. I note:

  • Your Master Dark has a median ADU value of 546.400
  • Your Light Frames you are trying to calibrate have a median  ADU value of around 91.000

   Any time you subtract 546 from 91, you are going to get mostly black frames. 

 

   Using two different programs (or even two different driver types like native vs ASCOM) for acquisition of image data is a bad idea. That is where your problem lies. There was another thread similar to this a couple of months ago. The user was using two, and at one point three, different programs to gather frames and calibration was a disaster.

 

   Re-shoot new calibration frames using the same program as the Light Frame capture and you should see improved results. Be sure to match all other parameters.

 

 

John

 

PS: Your temperature control is not working well either. The temperature of the Light Frames you posted ranged from 7.9° to 11.6°. Are you running at near full TEC Power? If so, it is usually better to back off a bit so that control of the temperature during capture is tighter.


Edited by jdupton, 16 August 2019 - 03:44 PM.

  • pfile and txo like this

#11 txo

txo

    Explorer 1

  • -----
  • topic starter
  • Posts: 65
  • Joined: 10 Jan 2019
  • Loc: EU, Prague

Posted 17 August 2019 - 02:58 AM

txo,

 

   The problem of mismatched calibration frames is immediately obvious from the frames you posted. I note:

  • Your Master Dark has a median ADU value of 546.400
  • Your Light Frames you are trying to calibrate have a median  ADU value of around 91.000

   Any time you subtract 546 from 91, you are going to get mostly black frames. 

 

   Using two different programs (or even two different driver types like native vs ASCOM) for acquisition of image data is a bad idea. That is where your problem lies. There was another thread similar to this a couple of months ago. The user was using two, and at one point three, different programs to gather frames and calibration was a disaster.

 

   Re-shoot new calibration frames using the same program as the Light Frame capture and you should see improved results. Be sure to match all other parameters.

 

 

John

 

PS: Your temperature control is not working well either. The temperature of the Light Frames you posted ranged from 7.9° to 11.6°. Are you running at near full TEC Power? If so, it is usually better to back off a bit so that control of the temperature during capture is tighter.

 

You're right! I was getting little desperate by now. I retook darks using APT and all my problems magically disappeared smile.gif. For almost about a year now (since I'm in this hobby really) I was randomly using different capturing software and had random inexplainable calibration problems. 

 

I however don't understand what's happening here. Exposure times, gains, black levels, temps are technical parameters and should not be influenced by the capturing software. Nor the image itself since we use raw data - or are we? Fits is not a native camera format. Does that mean that different SW encodes RAW data differently?

 

Regarding temperature variation - I didn't use TEC because it was 100% humidity and It would fog the sensor window. But retaken dark does not have any problem calibrating the frames regardless. 


Edited by txo, 17 August 2019 - 04:02 AM.


#12 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5291
  • Joined: 14 Jun 2009

Posted 17 August 2019 - 12:55 PM

well the problem is, the "camera" is not just the hardware, it's the hardware, the camera firmware, and the driver software. and then of course there may be variables/settings that the driver exports to the capture application that some applications change and others leave at defaults.

 

if all your capture applications use the same driver, for instance if there is an ASCOM driver for this camera, then it's more likely that two capture programs configured with the same temperature, gain, offset and frame duration will produce data that's largely the same. however, it's not really a guarantee.

 

if APT uses the ASCOM driver and sharpcap provides its own (this is just a hypothetical, as i have no idea what the driver situation is for Altair), then there's very little hope of your calibration frames matching your lights if you capture each with a different application... as you've found.

 

it is possible for 2 programs to write out different FITS files - one big problem with the FITS standard is that the 0th byte in the file might refer to the top left pixel on the sensor, or it might refer to the bottom left pixel on the sensor. so even if two capture applications configure the driver/camera exactly the same way, if the fits writing direction is reversed between the two programs then the frames will be incompatible with one another. you'd have to read in and then write out the images flipped to make them compatible. pixinsight can handle either fits direction but it is a global setting. there is the possibility to pass a fits reader direction "format hint" to ImageCalibration but i think it would universally apply to all frames being read by ImageCalibration; you can't specify a format hint for just the master dark or master flat or whatever.

 

rob




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics