Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

Strange pattern after Calibration (ML16200)

  • Please log in to reply
25 replies to this topic

#1 Redarm

Redarm

    Lift Off

  • -----
  • topic starter
  • Posts: 9
  • Joined: 23 Apr 2019

Posted 15 May 2019 - 06:01 PM

Hi there,

 

 

after calibrating some images taken with a 17" CDK, these images gave me headache

because there is still a strange nebulosity pattern in it, on all images the same.

Can anyone tell me where they propably come from, i think its a problem with the protective layer of the filter.

 

 

Attached Thumbnails

  • M51 neu-0001L_c1.jpg


#2 BenKolt

BenKolt

    Surveyor 1

  • -----
  • Posts: 1740
  • Joined: 13 Mar 2013

Posted 15 May 2019 - 06:23 PM

Hi, Redarm:

 

Sorry to hear about your troubles.  I've been there before with similar issues with my FLI ML16200 in the past.

 

It's difficult to tell just from the image where the nonuniform background is being introduced.  Let me ask a few questions please for clarification:

  • What do you mean by "protective layer" on the filter?
  • What filter are you using?
  • What are your skyfog conditions?
  • What is your calibration procedure?
  • Did you have thin clouds during acquisition?
  • Can you see this pattern in a single calibrated frame?
  • Have you applied any gradient removal, and if so, what did you do?

There are some steps you can take to help isolate where the pattern is being introduced, whether right at the acquisition of the raw frames, dark subtraction, flat fielding, registration, combination, gradient removal, etc.  This can be done systematically by combining the frames at each of these steps.  But, if you could first provide some more details I think you'll find many on this forum will be able to offer more precise suggestions.

 

Best Regards,

Ben



#3 Redarm

Redarm

    Lift Off

  • -----
  • topic starter
  • Posts: 9
  • Joined: 23 Apr 2019

Posted 15 May 2019 - 06:48 PM

Hi Ben,

 

 

- meant the anti-reflex-layer

- Baader LRGB

- no fog, no clouds in this moment when images where taken

- integrating bias

  integrate darks (same time as lights)

  calibrating flats with bias

  calibrating lights with dark/flat

- this is a single calibrated frame, same pattern in every frame

 when integrating the structure becomes more obvious

- Backgroundextraction with DBE Pixinsight forms pattern

Attached Thumbnails

  • integration.jpg
  • integration_DBE.jpg


#4 jhayes_tucson

jhayes_tucson

    Fly Me to the Moon

  • *****
  • Posts: 6772
  • Joined: 26 Aug 2012
  • Loc: Bend, OR

Posted 15 May 2019 - 09:12 PM

This stuff came come from a few different things so here are some comments and questions:

 

1)  High, thin clouds, but you say that it was perfectly clear so that's probably not the cause.  Sky fog from city lights could cause this sort of problem.  How dark are your skies?

 

2)  Variations across the flat data.  How did you create your flats?

 

3)  Stray light.  It only takes a very small amount of reflection from an off-axis source to create this sort of problem.  It's highly unlikely that the AR coatings would produce this sort of pattern.  Stray light is the number one problem with flats, but it rarely looks like a random, minor variation in illumination.

 

4)  A light film or minor condensation on your camera window could cause this sort of problem; although you would usually see symptoms of it in the star images as well.  Is everything perfectly clean in all of your components--including the field lenses in the CDK?  Dirty field optics could definitely lead to this sort of issue.

 

5)  You say that DBE forms the pattern.  Can you post a stretched version before DBE is applied?

 

John



#5 BenKolt

BenKolt

    Surveyor 1

  • -----
  • Posts: 1740
  • Joined: 13 Mar 2013

Posted 15 May 2019 - 09:16 PM

I was about to ask the same as John's #5.



#6 pfile

pfile

    Aurora

  • -----
  • Posts: 4871
  • Joined: 14 Jun 2009

Posted 15 May 2019 - 10:55 PM

does this particular FLI camera have an iris shutter? if the flats are too short the shutter can leave a starburst pattern in the calibrated lights. it's hard to tell if that's what's happening here but there do seem to be some radial components to the pattern.

 

rob



#7 Redarm

Redarm

    Lift Off

  • -----
  • topic starter
  • Posts: 9
  • Joined: 23 Apr 2019

Posted 16 May 2019 - 09:07 AM

Hi John,

 

to1) the sky was not perfectly clear, blinked through the images, high clouds came through

       but they didn't affect the pattern, just washing them out a bit.

 

to2) flats were taken with a flat field generator from artesky 460mm

 

to3) stray light would also affect other filters, but that didn't happen

 

to4) no film or condensation, temperature difference is only 2-5K to outside the dome

       various dust donuts are calibrated well

 

to5) wasn't clear enough, the right image above is with DBE, the left is without, DBE is bringing the pattern more out

 

to rob) the shutter problematic was already recognized after taking only 2 seconds flats, so we take another set

           with >15 seconds

 

Kind regards,

Kevin



#8 BenKolt

BenKolt

    Surveyor 1

  • -----
  • Posts: 1740
  • Joined: 13 Mar 2013

Posted 16 May 2019 - 10:45 AM

Redarm:

 

Thanks for clarifying your earlier images were with and without DBE.  That rules out DBE as the cause.  That process can introduce variations if it's abused.

 

In my experience, as well as gathering information from other users of this camera, 5s exposures are the bare minimum one ought to use for flats and lights, and even that may be too short to minimize the iris shutter effect.  I tend to go for at least 8s exposures, so your 15s should not be at issue here.

 

The flat panel is not likely the cause of this pattern, however it's easy enough to check its uniformity by taking some test flats with it, rotating the panel or scope systematically and, if possible, shifting it transverse to the aperture.  Look at frame differences of the various raw flats to satisfy yourself that the flats are adequately uniform.  This is an easy enough check to do and will at least give you assurance that the panel is uniform.  I only suggest it because it's an easy check to do and I'm sure that you want to be thorough.

 

I'm now more curious what's happening with your acquisition and calibration.  Starting with the dark frames.  Can you see this nonuniformity in your master darks?  Maybe that's not being properly removed.  Look at your master dark, stretch it hard to look at nonuniformities and see if you spot the pattern there.  I've had this issue in the past.

 

Best Regards,

Ben

 

P.S.  Oh, and maybe this is getting ahead of things, but I have come to calibrate my flat frames with dark subtraction using dark frames of equal exposure to the flats (and equal sensor cooling temperature).  In other words, dark-flats instead of bias subtraction.  This is likely a minor issue and may not be at all contributing to your issue, but my reasoning is that I want to match as exactly as possible the dark signal accumulation that my flats are experiencing by subtracting off the dark offset that also experienced that same duration of dark signal accumulation.  Again, I recognize this is likely not a big deal in comparison with instead subtracting bias signal from the flats, but flat fielding can be tricky and subtle as we are often working to eek out small signal in our images that is just above background.  It is possible this difference has some bearing.  Furthermore, when taking flats for NB filters, my exposure times are often much longer, and using matching dark-flats makes even more sense in those situations.  Just throwing out something more for you to consider, and you are welcome to take it or leave it.


Edited by BenKolt, 16 May 2019 - 10:54 AM.


#9 Jon Rista

Jon Rista

    ISS

  • *****
  • Posts: 22885
  • Joined: 10 Jan 2014
  • Loc: Colorado

Posted 16 May 2019 - 10:48 AM

I am curious...can you take a visual look at the actual sensor? Do you see anything physically with or on the sensor that might correlate with the patterns you are seeing in your data?



#10 Redarm

Redarm

    Lift Off

  • -----
  • topic starter
  • Posts: 9
  • Joined: 23 Apr 2019

Posted 16 May 2019 - 12:01 PM

The observatory is not near, so i have to pick up as many advices as possible.

Flatfield was also checked, we did several flat-sessions afterwards, to find possible failures,

thats why we didn't take the camera and filterwheel off. But next time we will, to find an end of the riddle.

 

Btw i double checked now other filters, to see if the patterns are there, conclusion = nonexistant.

So next time we will look at the filterwheel, corrector lenses and camera.



#11 Peter in Reno

Peter in Reno

    Voyager 1

  • *****
  • Posts: 10257
  • Joined: 15 Jul 2008
  • Loc: Reno, NV

Posted 16 May 2019 - 12:07 PM

Do you also see same strange pattern on original subs before calibration? If not, try experiment:

 

1. Calibrate only with dark

2. Calibrate only with bias

3. Calibrate only with uncalibrated flat

4. Calibrate only with calibrated flat

 

See which ones show strange pattern.

 

Peter



#12 BenKolt

BenKolt

    Surveyor 1

  • -----
  • Posts: 1740
  • Joined: 13 Mar 2013

Posted 16 May 2019 - 12:19 PM

The observatory is not near, so i have to pick up as many advices as possible.

Flatfield was also checked, we did several flat-sessions afterwards, to find possible failures,

thats why we didn't take the camera and filterwheel off. But next time we will, to find an end of the riddle.

 

Btw i double checked now other filters, to see if the patterns are there, conclusion = nonexistant.

So next time we will look at the filterwheel, corrector lenses and camera.

 

Aha!  This helps narrow down the scope of the problem by quite a bit.  So, you're looking at either a physical problem with this particular filter (look at it carefully, of course, for defects, smudges, etc.), or with the flat fielding using this filter.

 

Peter just laid out the steps I was going to suggest last night but hesitated without having more information at the time.  (You'll notice that he also mentioned this difference between bias and dark-flat subtraction, so I'm not the only one thinking about it.)  I suggest you follow his steps, particularly with the filter in question.

 

Ben



#13 AIP

AIP

    Vostok 1

  • -----
  • Posts: 133
  • Joined: 14 Feb 2019
  • Loc: Madrid, Spain

Posted 16 May 2019 - 01:14 PM

If you rotate the camera, is the pattern the same or does it change?



#14 Redarm

Redarm

    Lift Off

  • -----
  • topic starter
  • Posts: 9
  • Joined: 23 Apr 2019

Posted 16 May 2019 - 02:58 PM

Pattern is in Lights and Flats visible, but in Flats "abit more"

obvious donuts are well flattened but it seems there are huge scale donuts

causing this pattern

Attached Thumbnails

  • Flat.jpg


#15 Redarm

Redarm

    Lift Off

  • -----
  • topic starter
  • Posts: 9
  • Joined: 23 Apr 2019

Posted 16 May 2019 - 08:42 PM

As mentioned before, the pattern is visible in flats and lights,

the donuts well presented are corrected in a good way

the donuts far away from focus (you can see barely one the right upper corner)

are overcorrected and this is causing the pattern afterwards.

 

I'm not convinced that my way of calibration is without failure

thats why i uploaded all raw-files for further suspection.

 

https://drive.google..._ch?usp=sharing

 

Thanks to all, who want to help or already helped.

 

Best regards,

Kevin


Edited by Redarm, 16 May 2019 - 08:43 PM.


#16 jhayes_tucson

jhayes_tucson

    Fly Me to the Moon

  • *****
  • Posts: 6772
  • Joined: 26 Aug 2012
  • Loc: Bend, OR

Posted 16 May 2019 - 10:48 PM

Kevin,

That is a very "funky" looking flat.  Is this the linear master flat data or did you enhance it somehow?  You've got a lot of spatial variations and some sort of rectilinear "weave" pattern going on--as well as a sharp edge in the lower left corner.  I'm not familiar with the "flat field generator from artesky".  Is that a "normal flat box?  Are you using a t-shirt with it?  I've never seen a flat that looks like this.  Have you ever been able to get flats to work correctly?  I'm wondering if this is a new problem or if this is a problem that has always existed.  You've either got an equipment problem or you are doing something wrong but from the information you've provided, I can't figure out what's going on.  So, let's start with the flats.  Exactly how are you taking and processing the flat data?

 

I've attached a re-sampled flat from my ML16200 camera on my C14 so you can see what a good flat often looks like.

 

John

 

 

PS  If I mess around with stretching my ML16200 master flat data to enhance contrast, I can see a similar faint "weave" pattern.  That same pattern does not exist in my ML16803 master flat data.  So, did you stretch the flat data that you posted?

Attached Thumbnails

  • _16200_Master_Flat_25C_L_14subs_062318_clone.png

Edited by jhayes_tucson, 16 May 2019 - 10:59 PM.


#17 rockstarbill

rockstarbill

    Fly Me to the Moon

  • *****
  • Posts: 5660
  • Joined: 16 Jul 2013
  • Loc: Snohomish, WA

Posted 16 May 2019 - 10:51 PM

Which camera are you using? How long are the flat exposures? 



#18 BenKolt

BenKolt

    Surveyor 1

  • -----
  • Posts: 1740
  • Joined: 13 Mar 2013

Posted 16 May 2019 - 11:14 PM

Redarm:

 

Your flat and its weave pattern actually look typical to me, assuming your image of it is under a big stretch.  That weave pattern should not be present in your calibrated lights if your flat-fielding is working as it should.

 

There are some oddities in the flat, however, in addition to the strange shading.  There's also a diagonal line in the lower right quadrant.  What do your other flats look like from the other filters?

 

Please give details about your flat-fielding process.

 

Ben



#19 HydrogenAlpha

HydrogenAlpha

    Vostok 1

  • -----
  • Posts: 104
  • Joined: 02 Oct 2014
  • Loc: Singapore

Posted 17 May 2019 - 03:38 AM

I've had a similar issue with my Baader HA, and the gradients were terrible. I sent it back to Baader and they replaced it FOC. Since you're using LRGB, is the issue present only with one filter or with all of them? If all of them have the same issue, it is unlikely to be a filter problem. 



#20 Redarm

Redarm

    Lift Off

  • -----
  • topic starter
  • Posts: 9
  • Joined: 23 Apr 2019

Posted 17 May 2019 - 10:01 AM

Redarm:

 

Your flat and its weave pattern actually look typical to me, assuming your image of it is under a big stretch.  That weave pattern should not be present in your calibrated lights if your flat-fielding is working as it should.

 

There are some oddities in the flat, however, in addition to the strange shading.  There's also a diagonal line in the lower right quadrant.  What do your other flats look like from the other filters?

 

Please give details about your flat-fielding process.

 

Ben

it is stretched to point out the problem, the lines are present in every flat, light, i assume that it comes from the filterwheel

 

- pointing the 17" CDK to 90°

- put on top the artesky flat field box (open truss ring with secondary)

- dimm the light

- taking flats ( first ones where 2 seconds>shutter in picture)

(second ones where 7 seconds, full well 30%)

(third where 20 seconds full well 40% with more dimming)

 

- tried Masterflats, without Calibration, with DarkCalibration, with Dark&Bias, with Bias only (even tried normal Masterbias and Superbias on all those steps)

- tried also with multiple ways to calibrate the lights (Darksubstraction, Biassubstraction, Bothsubstraction)

 

- normal Calibration is  (flat - optimized Dark)-Bias

((Light-Dark)-Bias) * mean(flat) / flat

 

 

if you want to try out the several ways to calibrate an image, i already uploaded all the "raw" files

there you can see that when you correct it in a "good" way it will end with my result (except the left structure and one donut beneath m51, result of flats taken 2 weeks later)

 

Kevin,

That is a very "funky" looking flat.  Is this the linear master flat data or did you enhance it somehow?  You've got a lot of spatial variations and some sort of rectilinear "weave" pattern going on--as well as a sharp edge in the lower left corner.  I'm not familiar with the "flat field generator from artesky".  Is that a "normal flat box?  Are you using a t-shirt with it?  I've never seen a flat that looks like this.  Have you ever been able to get flats to work correctly?  I'm wondering if this is a new problem or if this is a problem that has always existed.  You've either got an equipment problem or you are doing something wrong but from the information you've provided, I can't figure out what's going on.  So, let's start with the flats.  Exactly how are you taking and processing the flat data?

 

I've attached a re-sampled flat from my ML16200 camera on my C14 so you can see what a good flat often looks like.

 

John

 

 

PS  If I mess around with stretching my ML16200 master flat data to enhance contrast, I can see a similar faint "weave" pattern.  That same pattern does not exist in my ML16803 master flat data.  So, did you stretch the flat data that you posted?

yes it is stretched for better visualization, its a normal flat box with dimmer, 560mm in diameter, no t-shirt, we moved the box in circles between some flats

to see wether the structure is coming from it, we cant saw that its was changing, as a bonus :  we took some sky flats, before the flat box arrived, and the structures were there (but shutter problematic)

this is nearly our first light, so we don't have experience with this setup

 

Which camera are you using? How long are the flat exposures? 

fli ml16200 20seconds



#21 Redarm

Redarm

    Lift Off

  • -----
  • topic starter
  • Posts: 9
  • Joined: 23 Apr 2019

Posted 17 May 2019 - 10:15 AM

on monday i have the chance to find out where the dustmotes are, and maybe what is causing the problem

as mentioned, the observatory is not near by me

 

img_0462-1.jpg?w=640

 

 

 

54514850_2271312093093721_80820765835164


  • bmhjr likes this

#22 BenKolt

BenKolt

    Surveyor 1

  • -----
  • Posts: 1740
  • Joined: 13 Mar 2013

Posted 17 May 2019 - 10:30 AM

Redarm:

 

Your equation is a bit confusing, leaving me wondering if you are subtracting the offset twice.

 

I recommend that you try the following and see if there is improvement:

 

Calibrate the Flat Files

 

CalFlat = RawFlat -DarkFlatMaster

 

DarkFlatMaster is a master dark made from combination of dark frames of equal exposure and temperature as the RawFlats.  Make your DarkFlat master without any optimization, just use the raw dark frames.  Also do not use any optimization for calibrating the flat frames.  When combining the calibrated flats into a master flat, it is recommended that you use multiplicative normalization, equal weights and equalize the fluxes under pixel rejection.

 

Calibrate the Light Files

 

CalLight = (RawLight - DarkMaster) mean(CalFlatMaster) / CalFlatMaster

 

DarkMaster is a master dark made from combination of dark frames of equal exposure and temperature as the RawLights.  Again, don't use any optimization with making this.  When calibrating the RawLights, also don't use any optimization.

 

You'll notice that this process doesn't use any bias frames at all.  The offsets ought to be matched between RawFlat and DarkFlatMaster as well as between RawLight and DarkMaster.  I'm not sure what you have done, but at first glance it looked like you were removing the offset twice in some manner or at least in danger of doing so.  This ought to be a check that you can do with your existing data.

 

When you do this, do you see any improvement in your CalLights?  Are the underlying patterns and dust motes better removed?  It may not solve your issue, but it may at least help to figure out if there's something happening with your processing.

 

Good luck!

 

Best Regards,

Ben

 

P.S.  I just added the mean(CalFlatMaster) in the second equation just to be clear that you don't want to possibly throw off the scale of your calibrated lights.  Using ImageCalibration, you populate the dark frame subtraction and flat field sections, leave the bias section unchecked, and turn off any calibration.


Edited by BenKolt, 17 May 2019 - 10:48 AM.


#23 Redarm

Redarm

    Lift Off

  • -----
  • topic starter
  • Posts: 9
  • Joined: 23 Apr 2019

Posted 17 May 2019 - 10:51 AM

Taking Darks with 20 seconds of exposure will be the first and the last thing we will do next monday

until we take away camera and filterwheel, but i think it will barely change anything.

The recommandations for integrating flats are clear.

 

https://www.lightvor...pixinsight.html

 

this tutorial describes exactly the way how i did it in the first way

 

https://www.cloudyni...g/#entry7873089

 

this quotation describes nearly the same process with pixelmath

which generates the same outcome



#24 jhayes_tucson

jhayes_tucson

    Fly Me to the Moon

  • *****
  • Posts: 6772
  • Joined: 26 Aug 2012
  • Loc: Bend, OR

Posted 17 May 2019 - 07:15 PM

Redarm:

 

Your equation is a bit confusing, leaving me wondering if you are subtracting the offset twice.

 

I recommend that you try the following and see if there is improvement:

 

Calibrate the Flat Files

 

CalFlat = RawFlat -DarkFlatMaster

 

DarkFlatMaster is a master dark made from combination of dark frames of equal exposure and temperature as the RawFlats.  Make your DarkFlat master without any optimization, just use the raw dark frames.  Also do not use any optimization for calibrating the flat frames.  When combining the calibrated flats into a master flat, it is recommended that you use multiplicative normalization, equal weights and equalize the fluxes under pixel rejection.

 

Calibrate the Light Files

 

CalLight = (RawLight - DarkMaster) mean(CalFlatMaster) / CalFlatMaster

 

DarkMaster is a master dark made from combination of dark frames of equal exposure and temperature as the RawLights.  Again, don't use any optimization with making this.  When calibrating the RawLights, also don't use any optimization.

 

You'll notice that this process doesn't use any bias frames at all.  The offsets ought to be matched between RawFlat and DarkFlatMaster as well as between RawLight and DarkMaster.  I'm not sure what you have done, but at first glance it looked like you were removing the offset twice in some manner or at least in danger of doing so.  This ought to be a check that you can do with your existing data.

 

When you do this, do you see any improvement in your CalLights?  Are the underlying patterns and dust motes better removed?  It may not solve your issue, but it may at least help to figure out if there's something happening with your processing.

 

Good luck!

 

Best Regards,

Ben

 

P.S.  I just added the mean(CalFlatMaster) in the second equation just to be clear that you don't want to possibly throw off the scale of your calibrated lights.  Using ImageCalibration, you populate the dark frame subtraction and flat field sections, leave the bias section unchecked, and turn off any calibration.

Ben,

I completely agree that no optimization should be used.  If the flat data is made with a flat panel (not the sky) I recommend avoiding any normalization altogether.  Simply average the stack using a Winsorized Sigma filter.

 

John



#25 BenKolt

BenKolt

    Surveyor 1

  • -----
  • Posts: 1740
  • Joined: 13 Mar 2013

Posted 17 May 2019 - 07:33 PM

Ben,

I completely agree that no optimization should be used.  If the flat data is made with a flat panel (not the sky) I recommend avoiding any normalization altogether.  Simply average the stack using a Winsorized Sigma filter.

 

John

John:

 

Yup.  This is what I've learned through hard-earned experience!  I hope all my underlined text will drive the point home to even the most causal of readers.  And I've also lately shifted to the Winsorized Sigma filter as well over the Percentile Clipping filter.

 

Ben




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics