23 replies to this topic

[kisstek]

OK, I finally got around to creating a set of L, R, G, and B flats using APT’s CCD Flat Aid tool. That was easier than I expected. (Other than trying to see the laptop’s screen in broad daylight!)

 

I then processed the flats in PixInsight per “Inside PixInsight” (2nd edition) to generate master flats for each of L, R, G, and B. The R, G, and B versions are very similar in their small, sharp dust spot pattern. L has all of the same ones as R, G, and B’s but with a couple (5-6) larger, fainter, fuzzier ones added. I’m guessing the common, small ones are on the camera itself while the larger L ones are on the L filter. My reasoning is that dust farther from the sensor will be bigger and less sharp. Make sense?

 

So then I calibrate my raw L subframes of M101 from three nights ago using the new master flat L and my master dark from that night. Inspecting the calibrated L subframes reveals that the small common dust spots have been pretty effectively removed. However, the larger L dust spots remain!

 

Any idea why the flat isn’t removing the larger spots?

 

Since it's 12MB, I’ve put a screenshot of my PI screen at

 

http://www.mikemac.c...-dust-spots.png

 

From top to bottom, left to right the images are:

 

1) the raw L subframe

2) the master L flat

3) the calibrated L subframe

4) the stacked L images of all 20 subframes

 

All of the images have had Auto Stretch applied so we can see the details.

 

So what am I doing wrong?

[jerahian]

What is the mean ADU value for your master L flat?  I ask because it's looking on the dim side to me.  Your flats should be approximately 35-45% of your max ADU value.  If your value is lower, I would look at that as the prime suspect first, causing the flat to not be fully effective in doing its job.

 

-Ara

[kisstek]

What is the mean ADU value for your master L flat?  I ask because it's looking on the dim side to me.  Your flats should be approximately 35-45% of your max ADU value.  If your value is lower, I would look at that as the prime suspect first, causing the flat to not be fully effective in doing its job.

 

-Ara

APT's CCD Flat Aid tool picks 20,000 as its target ADU. It then creates the imaging plan for each filter I specified. I actually ran the plan twice and used all 30 of the subframes for the individual masters.

 

Thinking that my problem might be caused by a filter alignment error, I then went back and processed each batch of 15 separately. It made no difference in the calibrated subframes of M101.

 

Then I used PixelMath in PI to subtrace the 15 frame L flat master from the 30 frame L flat master, thinking if the EFW didn't put the filter back exactly in the same spot, the positioning error should show up in the difference. There was no signs of a misalignment.

Edited by kisstek, 19 May 2019 - 03:55 PM.

[Ballyhoo]

What is the mean ADU value for your master L flat?  I ask because it's looking on the dim side to me.  Your flats should be approximately 35-45% of your max ADU value.  If your value is lower, I would look at that as the prime suspect first, causing the flat to not be fully effective in doing its job.

 

-Ara

How does one measure ADU value?

[pfile]

the "embossed" look to the dust spots in the single calibrated light says to me that the dust motes moved inbetween light and flat acquisition. even if its not a filter positioning error, the motion of the filter wheel might have caused the dust to shift.

 

i'm not sure anything can be done about this, short of heroic measures involving making synthetic flats and then pixelmath to try to repair the spots by hand.

 

rob

[jerahian]

How does one measure ADU value?

Most image capture software, like SGP (never used APT though), show mean, median, avg ADU values (0-65535), when each frame is captured.  Also, with flat creation wizards, one targets a particular ADU value and the software determines the length of exposure.  Lastly, you can get the value from the Statistics process from PixInsight.

 

-Ara

[kisstek]

the "embossed" look to the dust spots in the single calibrated light says to me that the dust motes moved inbetween light and flat acquisition. even if its not a filter positioning error, the motion of the filter wheel might have caused the dust to shift.

 

i'm not sure anything can be done about this, short of heroic measures involving making synthetic flats and then pixelmath to try to repair the spots by hand.

 

rob

I had noticed the embossed look too. That's why I initially thought it might be the EFW not being consistent in its positioning. Shifting dust is not much better than an imprecise EFW wheel.

 

Thanks for looking at it!

[kisstek]

Most image capture software, like SGP (never used APT though), show mean, median, avg ADU values (0-65535), when each frame is captured.  Also, with flat creation wizards, one targets a particular ADU value and the software determines the length of exposure.  Lastly, you can get the value from the Statistics process from PixInsight.

 

-Ara

Except PI doesn't label it as ADU.

 

Here's my stats:

 

Master_Flat_2019_05_17_L
            K
count (%)   100.00000
count (px)  16389120
mean        19555.409
median      19656.973
avgDev      1297.784
MAD         1419.061
minimum     14470.272
maximum     23092.155

 

So 19,600 is probably a bit low.I'll up the target ADU to 30,000 next time. Assuming the weather ever clears so there is a next time.!

[pfile]

ADU just means 'analog-digital [converter] unit'; you'll also see "DN" or data number mentioned here and there. but no matter what you call it if you're looking at a 16-bit integer fits file, you're looking at ADUs. PI normalizes everything from 0.0-1.0 internally and the default readout mode is the same. but as you know, setting the stats process to 16-bit integer then represents the 0.0-1.0 values as 16-bit integers. you can also change the readout cursor to read in i16.

 

actually, thinking about this more, my statement above about heroic measures is not 100% true. if you can get new subs for which the flats match well, then make a reasonably clean master light, you can then try using LocalNormalization on the bad frames. they will need to be registered to the same reference as the good lights in order for this to work. LN might be able to fix the dust spots depending on the scale parameter used.

 

rob

[Ballyhoo]

Still, the numbers in the stat process that i ran look different.  What should one be correlating these numbers with respect to the camera specs? It is the full well capacity?

 

Edit, my camera, being an ASI 294 pro, has a ful well capacity of 64000. Is that the number I need to consider?

Attached Thumbnails

• 
• 

Edited by Ballyhoo, 20 May 2019 - 12:03 AM.

[jerahian]

Except PI doesn't label it as ADU.

 

Here's my stats:

 

Master_Flat_2019_05_17_L
            K
count (%)   100.00000
count (px)  16389120
mean        19555.409
median      19656.973
avgDev      1297.784
MAD         1419.061
minimum     14470.272
maximum     23092.155

 

So 19,600 is probably a bit low.I'll up the target ADU to 30,000 next time. Assuming the weather ever clears so there is a next time.!

In hindsight, I'm going to +1 pfile's assessment of the situation.  The embossed effect does hint to a shift, but I would add it might be a shift in your Lum filter instead of the dust particle itself, since it seems all (?) the motes associated to that filter seem embossed.  Is/was your Lum filter screwed in well?  Just throwing that out there as a possibility...

[jerahian]

Still, the numbers in the stat process that i ran look different.  What should one be correlating these numbers with respect to the camera specs? It is the full well capacity?

 

Edit, my camera, being an ASI 294 pro, has a ful well capacity of 64000. Is that the number I need to consider?

The full well capacity is how many electrons have to land into each pixel before that pixel is considered "full" or saturated (like buckets of water in a well).  So, for the 294MC Pro, that capacity is a whopping 63700 electrons...nice and deep.

 

The 294MC Pro has a 14-bit analog-to-digital converter (ADC), so when each pixel is read out, the number of electrons in the pixel (an analog value) is converted to a digital 14-bit value, so between 0 to 16,383.

 

But wait, there's more...most software will display the pixel value as a 16-bit number instead, so that 14-bit value of 0 - 16,383 gets mapped to a value between 0 - 65,535.  This is the precision in which the analog to digital "unit" (ADU) is often referenced.

 

So, you see, the value can be mapped to any range, including a normalized 0 to 1 range.  In the screenshot you attached, PI is showing you the stats, indeed, normalized to 0-1.  If you change the "Normalized Real [0,1]" dropdown, you will see other options, including [0,65535] which will display the numbers in the 16-bit range often referenced by astrophotographers whenever ADU or DN (data number, another generic name for it) is mentioned.

 

That said, this sidebar has somewhat derailed the OP's original thread.  So if you have further questions about this, you should probably start a new thread.

 

-Ara

[kisstek]

In hindsight, I'm going to +1 pfile's assessment of the situation.  The embossed effect does hint to a shift, but I would add it might be a shift in your Lum filter instead of the dust particle itself, since it seems all (?) the motes associated to that filter seem embossed.  Is/was your Lum filter screwed in well?  Just throwing that out there as a possibility...

 

Since the L filter didn't shift when the EFW moved to the R, G, and B filters and back, I'd assume yes, it's in tight. There are three screws holding each filter in.

 

Your observation that there's no movement between the dust spots does seem to eliminate the dust shifting.

 

So the wheel must not reposition exactly under some circumstances. The question then is what are those circumstances. The EFW had been powered off after the light frames were taken and back on the morning I took the flats. Could be some jitter in the startup. Or a thermal effect? Sun was shinning on the scope in the morning. Could cause some expansion?

 

Yet another puzzle to solve!

[kisstek]

That said, this sidebar has somewhat derailed the OP's original thread.  So if you have further questions about this, you should probably start a new thread.

 

-Ara

Until today, I had never run the Image->Statistics process. So I'm glad he asked about how to determine the ADU of an existing image in PI.

[jerahian]

Until today, I had never run the Image->Statistics process. So I'm glad he asked about how to determine the ADU of an existing image in PI.

Glad to hear! Just wanted to be respectful of your post and your problem statement

[jdupton]

Mike,

 

So the wheel must not reposition exactly under some circumstances. The question then is what are those circumstances. The EFW had been powered off after the light frames were taken and back on the morning I took the flats. Could be some jitter in the startup. Or a thermal effect? Sun was shinning on the scope in the morning. Could cause some expansion?

 

   For some reason, this has come up several times in several threads recently here on CloudyNights. (It may be an epidemic of gremlins running around.)

 

   Check the driver options for your filter wheel. If there is an option for either "Bidirectional" or "Unidirectional" movement of the filter wheel, change it so that wheel will always turn the same direction when changing filters. (If Bidirectional is your option, turn the option off. If Unidirectional is your option, turn it on.)

 

   When you see a filter that appears to have slightly moving dust motes where they all move in unison, it is often found that the filter wheel supports and is using bidirectional motion. A filter wheel may settle in a sightly different final location when approaching the filter from a clockwise direction compared to where it settles when the filter is approached from a counterclockwise direction. You may be seeing this effect.

 

   Try looking in the driver options for your filter wheel and make sure it is told to always turn the same direction when changing filters. See if that might fix your issue.

 

 

John

Edited by jdupton, 20 May 2019 - 10:58 AM.

[kathyastro]

I agree with the above assessments that it seems to be a filter wheel positioning issue.

 

I also have to add, though, clean your filters!!!

[Ballyhoo]

The full well capacity is how many electrons have to land into each pixel before that pixel is considered "full" or saturated (like buckets of water in a well).  So, for the 294MC Pro, that capacity is a whopping 63700 electrons...nice and deep.

 

The 294MC Pro has a 14-bit analog-to-digital converter (ADC), so when each pixel is read out, the number of electrons in the pixel (an analog value) is converted to a digital 14-bit value, so between 0 to 16,383.

 

But wait, there's more...most software will display the pixel value as a 16-bit number instead, so that 14-bit value of 0 - 16,383 gets mapped to a value between 0 - 65,535.  This is the precision in which the analog to digital "unit" (ADU) is often referenced.

 

So, you see, the value can be mapped to any range, including a normalized 0 to 1 range.  In the screenshot you attached, PI is showing you the stats, indeed, normalized to 0-1.  If you change the "Normalized Real [0,1]" dropdown, you will see other options, including [0,65535] which will display the numbers in the 16-bit range often referenced by astrophotographers whenever ADU or DN (data number, another generic name for it) is mentioned.

 

That said, this sidebar has somewhat derailed the OP's original thread.  So if you have further questions about this, you should probably start a new thread.

 

-Ara

With the above said, how do I determine what are optimum values of the stats, and does my histogram pic look okay? I understand if you prefer that I start another thread.

[BenKolt]

Still, the numbers in the stat process that i ran look different.  What should one be correlating these numbers with respect to the camera specs? It is the full well capacity?

 

Edit, my camera, being an ASI 294 pro, has a ful well capacity of 64000. Is that the number I need to consider?

Ballyhoo:

 

Using PI's Statistics process window as I see you have done, you can change the displayed values from "Normalized Real" to 16-Bit (or whatever is appropriate) so that you can see those numbers in ADU units.

 

Ben

[Peregrinatum]

PI Clone Stamp tool is your friend here.

[spokeshave]

the "embossed" look to the dust spots in the single calibrated light says to me that the dust motes moved inbetween light and flat acquisition. even if its not a filter positioning error, the motion of the filter wheel might have caused the dust to shift.

 

i'm not sure anything can be done about this, short of heroic measures involving making synthetic flats and then pixelmath to try to repair the spots by hand.

 

rob

I agree - the embossed effect is a clear indication of a lack of registration between the lights and the flats. There can be a number of reasons for this including a filter wheel that isn't landing in the same spot every time. dust moving around is another possibility, but the fact that the embossing is by about the same amount in the same direction for each dust donut probably rules that out. One last thing to consider is that the camera could be rotating in relation to the filter wheel.

 

Tim

[kisstek]

Mike,

 

 

   For some reason, this has come up several times in several threads recently here on CloudyNights. (It may be an epidemic of gremlins running around.)

 

   Check the driver options for your filter wheel. If there is an option for either "Bidirectional" or "Unidirectional" movement of the filter wheel, change it so that wheel will always turn the same direction when changing filters. (If Bidirectional is your option, turn the option off. If Unidirectional is your option, turn it on.)

 

   When you see a filter that appears to have slightly moving dust motes where they all move in unison, it is often found that the filter wheel supports and is using bidirectional motion. A filter wheel may settle in a sightly different final location when approaching the filter from a clockwise direction compared to where it settles when the filter is approached from a counterclockwise direction. You may be seeing this effect.

 

   Try looking in the driver options for your filter wheel and make sure it is told to always turn the same direction when changing filters. See if that might fix your issue.

 

 

John

 

Ah! That might be it. Before I did my imaging run for the lights, I tell APT to move to filter 8 (dark) and then back to filter 1 (L) to make sure it hadn't shifted since the last time I used the scope. I did NOT do that before I took the flats. And my EFW does do bidirectional movements and APT is using that ability. I'll turn that off from future use.

 

In the meantime, I might be able create a new, matching master L flat if I control the approach before I take the subframes.

 

Thanks for the idea!!

Edited by kisstek, 20 May 2019 - 03:40 PM.

[kisstek]

Ah! That might be it. Before I did my imaging run for the lights, I tell APT to move to filter 8 (dark) and then back to filter 1 (L) to make sure it hadn't shifted since the last time I used the scope. I did NOT do that before I took the flats. And my EFW does do bidirectional movements and APT is using that ability. I'll turn that off from future use.

 

In the meantime, I might be able create a new, matching master L flat if I control the approach before I take the subframes.

 

Thanks for the idea!!

I was able to create a new master flat after approaching L from the same side I do when I first start imaging. The master matches a LOT better. Most of the big dust spots are gone and the remaining are pretty subtle although they do show up if I really push the contrast.

 

Too bad my data is bad. M101 is too close to the streetlight in my backyard. So I'm getting a light bleed from that side of the frame.

 

Thanks for the help!!

[kisstek]

Well, that almost works.

 

I shot both M13 and NGC6888 last night using the same imaging plan. Both objects took a very similar track across the sky so the mount's orientation didn't change much between the two. After collecting the second set of data, the scope was parked but left powered up. This morning, I took a set of flat exposures for all four filters (LRGB). I just processed both sets of data using PI's BatchPreprocessing script. I don't have any Bias frames yet so I left that blank. Everything else was pretty much the defaults.

 

The image of NGC6888 came out nice. (Needs more R data.) All of the dust was removed. Unfortunately, the M13 image still has the "embossed" L dust spots! Argh!  

 

So I'm at a loss for what I'm doing wrong. Or more specifically, at the inconsistency of the dust removal.