70 replies to this topic

[sharkmelley]

Here is a recent image I took of Lynd's Dark Nebula 1251 on a reasonably dark night (SQM 21.4 i.e Bortle 3) with an unmodified Nikon Z6 camera on a Tak Epsilon 180ED.  I used 2min subs at ISO 3200.  The image has been calibrated, stacked.  Pixinsight ABE (background extraction) with function degree 1 was applied to subtract the background light pollution, followed by ArcsinhStretch.

 

Note the purple colour cast towards the edges:

 

 

 

Here is an in-camera JPG of a single exposure:

 

 

 

Here is an in-camera JPG of a single dawn-sky flat taken a few hours later:

 

 

A master flat from the dawn-sky flats was used to process the above image.  Surprisingly, the reason for the purple color cast is that the colour and intensity of the flats do not match the colour and intensity of the lights. I will prove this in the next post (successive posts because of the Cloudy Night 500kB limit on images in a single post)

Edited by sharkmelley, 02 November 2019 - 08:19 AM.

[sharkmelley]

Using a coloured light source (a laptop screen showing a flat colour background) I generated some flats whose colour and intensity was a much closer match to the lights.  Here is an in-camera JPG of one of those flats:

 

 

 

Repeating exactly the same processing but using a master flat generated from these carefully created flats showed no purple cast:

 

 

 

I've now done some further investigation into this - see next post.

Edited by sharkmelley, 02 November 2019 - 08:33 AM.

[sharkmelley]

Again using a laptop to provide a coloured light source and with an opaque diffuser on the scope I was able to generate flats of arbitrary colour under well-controlled conditions.  Here are the in-camera JPGs of two examples side by side which approximately match my lights and my dawn sky flats.

 

 

 

I created a master for each from the raw data, bias subtracted them, then divided one by other followed by white balancing so the centre was more or less white white.  Here is the result with a PixInsight STF stretch applied:

 

 

 

The (bias subtracted) master flats do not properly calibrate each other and the purple cast is again obvious. The reason for the purple cast is some kind of non-linearity in the raw data, possibly a non-linearity in the ADC (analogue digital converter).  The non-linearity can be quantified: between the image centre and the corners, the green intensity decreases by 0.4% while the red and blue intensities increase by 0.46% and 0.35% respectively.  So the divergence between the red/blue and green channels is nearly 1% in this particular example.

Edited by sharkmelley, 02 November 2019 - 08:16 AM.

[james7ca]

I've always assumed (and have stated so here on CN) that blue-sky flats probably shouldn't be used with a one-shot-color camera (i.e. something with a Bayer pattern or CFA). The problem is that with such a strong color (blue) in the light source the green and red pixels are probably exposed to a much lower light intensity than the blue pixels. Or, said another way the blue pixels are exposed to a much higher intensity which could lead to possible over saturation of the blue pixels. I guess if you balanced the exposure correctly so that all of the pixels were still operating in their linear range then the only effect might be to introduce a bit more noise in the signal from the green and red pixels. In general, there should be several stops of latitude before you ran into problems with either too low of an exposure (in the green and red) versus too high of an exposure in the blue.

 

I don't know what the exact distribution of colors is in a blue sky, but it probably isn't more than those same few stops in exposure latitude (since two stops would be a ratio of 1:4 between the green and red and the blue). So, if you exposed for blue at 50% full well then the red MIGHT be 50 / 4 or about 12% full well). In fact, I measured a blue sky image and found about a 1:5 ratio between the blue ADU and the red.

 

Mono cameras would be a little different, because all of the pixels could see the same intensity (if you had a perfectly uniform light source). So, in that case any variation would have to be from differences in the individual pixel response or a variation introduced by the optical system (e.g. vignetting).

 

In any case, I always use a near-white source (at least visually) when taking flats.

 

Here is a discussion from 2014 about this issue (blue sky flats with a DSLR). It seems that the initial conclusion was that it should NOT matter (the color of the light source). Then, it was suggested that there might be some minor issues, but I think there was no certain conclusion one way or the other.

Edited by james7ca, 02 November 2019 - 10:58 AM.

[sharkmelley]

 

In any case, I always use a near-white source (at least visually) when taking flats.

 

Thanks for your comments! 

 

I agree with you about avoiding blue sky flats.  The dawn flat I used was shot just as night was ending and dawn was breaking so it was not overly blue.  I did try a white light source for my flats but the non-linearity was hardly different from the dawn sky colour and it resulted in an almost identical purple colour cast in the final image.  I should have mentioned this.

 

Here are my (bias subtracted) raw pixel values for the night sky, dawn sky and white light sources, scaled to give G=1000:

 

Night sky:  RGB = ( 911, 1000,  405)

White:        RGB = ( 513, 1000,  730)

Dawn sky:  RGB = ( 521, 1000,  953)

 

Mark

[whwang]

Hi Mark,

 

Thank you for posting this.

 

Do you have an idea about how good the match has to be in terms of brightness and color?  Any idea what causes the flat field error in the first place?

 

Personally I think controlling the color of the flat is more challenging than controlling the brightness (although you have shown that it's totally achievable).  My idea is that one can shoot flats of many different exposures using a convenient light source.  Even if the color of that light source may not be matched to the night sky (and the color of the night sky changes anyway), for a specific channel among R,G,B, one should be able to find flats whose R or G or B brightness is similar to the R or G or B brightness in the light, and calibrate R, G, B separately.

 

Cheers,

Wei-Hao

[ChristopherBeere]

Great research Mark !

 

On a somewhat tangential note I found that i had to ISO match my flats or i would get poor results when calibrating.

 

I was always under the impression that flat division was an ISO agnostic operation so i shot super low ISO to mitigate noise and optimise flat field illumination but the data proved otherwise.

 

Flats at 100 where useless for my shots at 1600.

 

I now ISO match my flats.

[sharkmelley]

Do you have an idea about how good the match has to be in terms of brightness and color?  Any idea what causes the flat field error in the first place?

 

Personally I think controlling the color of the flat is more challenging than controlling the brightness (although you have shown that it's totally achievable).  My idea is that one can shoot flats of many different exposures using a convenient light source.  Even if the color of that light source may not be matched to the night sky (and the color of the night sky changes anyway), for a specific channel among R,G,B, one should be able to find flats whose R or G or B brightness is similar to the R or G or B brightness in the light, and calibrate R, G, B separately.

 

Unfortunately I don't yet know how good the match needs to be and I don't know the cause of this apparently non-linear behaviour.

 

Your suggestion of taking the R, G & B channels separately from different brightness white flats is a good one.  It ought to work unless something really strange is going on.  I'll give it a try when I get the chance.

 

Mark

[t_image]

Thanks Mark for your continued efforts in AP to solve such complex challenges utilizing your knowledge!

Finding ways to assess and then counter the hiccups in consumer cameras is so useful to the AP consumer camera user crowd!!

 

I wonder if there is any way for you to capture a faint source for long exposure 'lights' with your system[calibrated/completely known source],

then notice the same issues with it in processing,

but then devise a reverse-engineered workflow for a flat that would be matched.....

Maybe ask a neighbor to hold a color checker card and a flashlight a mile away while you image it????

[AgilityGuy]

Mark,

This is very interesting. I’ve also found that sky flats won’t work all that well with my Nikon D800 full spectrum camera. Until recently I had always used an iPad set to white at a bright intensity and exposed to about 50 percent. When using sky flats at the same intensity the color was quite different and surprisingly the vignette pattern was not as well defined. I’ve gone back to the iPad.

[dayglow]

Is there actually a way to apply different flat masters to R, G, B channels respectively of OSC camera RAW ? 

How would one benefit from taking different light sources for each channel ?

 

= = = = =

My imaging is done primarily with astro-modified D5300s (2 of them).

 

I sometimes have difficulty with flats causing uneven color changes across the field of my images and I am still not consistently happy with my results.  To a small extent the lossy compression employed by Nikon in D5300 may contribute some imperfections but I think the essence of the problem lies in the large inherent differences of pixel values between R, G, B channels.

 

I investigated some different processes of getting flats that would not bias the color of my OSC images beyond my ability to subtract out via PixInsight ABE or repair with color balance later .  In all of these cases, a translucent white plastic diffuser is placed on the end of the telescope's dew shield.

 

1a)  daytime sky flats with white diffuser.

 

1b)  daytime sky flats with pale yellow T-shirt instead of diffuser.

 

2)  early dawn or post sunset sky flats with white diffuser.

 

3a) computer or laptop screen displaying what appears to be a clean (no spots) and uniform white image (not a calibrated monitor).

 

3b) computer screen displaying a uniform image whose color is chosen by me from a palette.  I adjust the color in the program until histograms (R, G, B) of the RAW images is as balanced as I can get it.  Often the shape of the 3 color curves differ so complete alignment is not achieved.  The screen does not appear white to the eye but RGB pixel values are not grossly different for making the flat.

 

4)  back to daytime sky flats made with white diffuser but after producing the averaged master, I apply a tight gaussian blur (radius about 2 pixels) to even out all the RGB values locally.  This, of course, negates the ability to compensate for pixel response variations and makes dust correction a little less perfect.  Not all image processing software supports this approach. 

 

Presently I cannot advocate for any of these methods, merely say that I have tried them.

 

-- David F.

[whwang]

Mark,

This is very interesting. I’ve also found that sky flats won’t work all that well with my Nikon D800 full spectrum camera. Until recently I had always used an iPad set to white at a bright intensity and exposed to about 50 percent. When using sky flats at the same intensity the color was quite different and surprisingly the vignette pattern was not as well defined. I’ve gone back to the iPad.

My D800 suffered severely from this, much worse than Mark’s case on Z6 and other cases from D810A that I have seen. However, the problem is completely gone after I use the firmware hack for D800. The flat fielding is always as good as it should be and it’s insensitive to the exposure of the color of the light source.

Just for your reference.

Cheers,
Wei-Hao

[sharkmelley]

Is there actually a way to apply different flat masters to R, G, B channels respectively of OSC camera RAW ? 

How would one benefit from taking different light sources for each channel ?

 

To do this requires the master flat to be artificially constructed from 4 separate channels.  The PixInsight processes SplitCFA and MergeCFA are ideal for this.

 

The benefit of using different flats for each channel is that you only need a single white light source to create them, as long as it has adjustable intensity of course.

 

Mark

[whwang]

Adjustable intensity? How about just changing exposure time?

[sharkmelley]

Adjustable intensity? How about just changing exposure time?

Yes either will work but changing exposure time won't allow fine adjustment to image brightness.  Fine adjustment might be necessary but on the other hand it might not.

 

Mark

Edited by sharkmelley, 03 November 2019 - 05:06 AM.

[whwang]

Hmmm.... I know you are not sure, but you sort of suggested that a 1/3 stop change in exposure may not be enough. If it really require such accurate match in the light and flat, it will be difficult to do in some cases, since the brightness and color of the night sky can be quite variable.

[sharkmelley]

I've done a new experiment to see if 3 white flats will work.

 

The brown master flat used earlier has the following raw values (bias-subtracted) in the image centre:  (R,G,B) = (600, 656, 265)

 

I then created 3 flats (approximately white) with the following sets of RGB raw values in the image centre:

• (R,G,B) = (595, 1171,  874)
• (R,G,B) = (330,   653,  488)
• (R,G,B) = (175,   347,  262)

Taking the red channel from the first, green from the second and blue from the third gave me a synthetic flat with the following values in the image centre:  (R,G,B) = (595, 653, 262)

 

So the flat synthetically created from the white flats matches the values in the original brown flat pretty well.

 

Dividing one by the other gives the following with a PixInsight STF stretch applied:

 

 

In other words, the flat created synthetically from 3 white flats does not correct for purple colour cast.  So it appears to be necessary to create a flat from a coloured light source in order to remove the purple colour cast at the edges.  But it also suggests that this non-linearity is caused by some internal data processing.  The fact that Wei-Hao says that a similar problem on the D800 disappeared when the firmware was hacked also supports this idea.

 

Certainly, something very weird is going on.

 

Mark

Edited by sharkmelley, 03 November 2019 - 04:06 PM.

[whwang]

Now things are getting weird.....

[sharkmelley]

Now things are getting weird.....

My initial hypothesis was a non-linearity in the response of the ADU but the above evidence does not fit that theory and now I really don't know what is going on.

 

By the way, an alternative to creating flats from a coloured light source is to use synthetic flats.  Either take images of an "empty" part of the night sky so that stars can be later removed or put a diffuser over the scope.  There are plenty of online tutorials for this.  The trouble is, this will eat into valuable imaging time because quite a number of exposures will be required to produce a master flat that doesn't add noticeable noise to the stack.

 

Mark

Edited by sharkmelley, 04 November 2019 - 02:33 PM.

[whwang]

Based on your latest finding, it seems the behavior of R, G, and B is not independent from each other.  (Otherwise the method I proposed should work.)  Even from the internal camera processing point of view, this is weird.  I think some more tests are needed to understand the exact behavior.  I will see if I can find time to do some experiments on my D800 (with hacking disabled).  The problem seems worse on D800, meaning that it might be easier to catch what's really going on.

 

The sky flat method is what I constantly do when I use large telescopes for near-IR (1 to 2 micron).  The difference is that our typical targets are point-like distant galaxies and we took lots of dithered exposures.  So, synthetic flats can be very easily created from the dithered images themselves, without spending extra time on empty spots of sky.  Unfortunately this will not work on amateur images, since we almost always image targets that occupy large portions of images.  However, there might be a workaround.

 

Some 15 years ago, I developed a flat-field method for film images.  The first step is to just shoot a normal flat on daytime sky using the same optics.  For the problem we want to solve here, this flat can be a regular flat taken with whatever appropriate exposure time and light source (doesn't have to match the night sky).  

 

The second step is to go to the to-be-flattened night sky image and sample a good number (20 to 40) of "blank" sky areas, just like what we do in DBE in PixInsight.  Then we can use the RGB brightness and the locations of these blank sky areas to "twist" the flat, using a 2-D surface determined by those blank areas, to force the flat to look like the sky background in the celestial image.  

 

This way, the flat image taken in the first step provides the overall shape of the illumination pattern, including vignetting from the optics and camera and even the pixel-to-pixel response variation on the sensor.  The twisting in the second step corrects for whatever nonlinear effect in the sensor response.  

 

There are two potential problems of the this method.  The first one is that in a perfectly linear system, the 2D surface found in the second step should be just a flat surface.  In reality, even if the system is perfectly linear, the surface will not be flat because of the existence of sky gradient.  In some sense, this can help to correct for the gradient, but this isn't really the right thing to do, as gradient removal should be subtraction, not division, and to model the gradient it can sometimes require many more sample points.

 

The second drawback is that unlike film images, we took many dithered subs using digital sensors.  This means the locations of the sample points for blank sky move from one sub to another.  This is where the synthetic sky flat can come to rescue.  We can just take a few exposures on the empty parts of the sky during the imaging session with the same exposure setting.  Then from these few exposures, using median stacking plus rejection, we can create a low-S/N sky flat, mostly free from celestial objects.  This low-S/N flat can be used to construct the surface for nonlinearity correction and then be applied to the high-S/N flat we took in the first step.  Here, probably the entire image can be used, instead of just using a few sample points.  This way, it doesn't matter if our real subs are dithered around, and it doesn't matter if the sky-flat has low S/N.

 

Anyway, I gave the synthetic sky flat a lot of thoughts when I encountered my D800 problem.  If we try hard, we should be able to find a solution, but I rather prefer this to be solved at the raw image level.  

 

I really hope I can have a chance to sit down with a Nikon engineer in charge to talk about this face to face in details.  I don't believe writing to their customer service can be of any help.

 

Cheers,

Wei-Hao

[AgilityGuy]

 

I really hope I can have a chance to sit down with a Nikon engineer in charge to talk about this face to face in details.  I don't believe writing to their customer service can be of any help.

 

Cheers,

Wei-Hao

I hope you get the chance as well.  Maybe they could provide a bit more clarity.  

 

Obviously it's a very small group of Nikon camera owners that use their cameras for astrophotography but it seems it would be simple enough to bury a toggle somewhere in the camera menu that would turn off image processing entirely.  I have to think it's either intellectual property concerns or economics or a combination of the two that prevent such a move.  It sure would be nice to be able to use a camera equipped with the latest and greatest sensor technology in a RAW sensor mode without image processing.  Many of the dedicated astrophotography CMOS cameras now available that provide unprocessed images are using sensor technology that is 6-7 years old (Sony IMX-094, IMX-193).  It's not that many of them don't still perform well but it seems strange to imagine that a dedicated astrophotography camera with the Z6 or Z7 sensor won't be available until...2025?

[otoien]

My D800 suffered severely from this, much worse than Mark’s case on Z6 and other cases from D810A that I have seen. However, the problem is completely gone after I use the firmware hack for D800. The flat fielding is always as good as it should be and it’s insensitive to the exposure of the color of the light source.

 

Since no one has mentioned it: Could the issue in this case be with the bias? I am very far from an expert, but I seem to recall that Mark and others have shown that the unhacked D800 has zero clipping (opposed D800a and later 8x00 models). A problematic bias could lead to problems with scaling of the flat, perhaps affecting color channels differently? It does not explain the Z problems though.

[whwang]

It's hard to say.  D800 has two problems: severe zero clipping, and dark subtraction based on the optical dark pixels.  Both lead to wrong zeros in the files.  The firmware hack removes both and D800 files calibrate perfectly after the hack.

 

On later generation Nikon cameras, from D810, the zero clipping is reduced.  It's not gone, but reduced to a point that most (not all) of the noise histogram can be seen in a bias image.  In principle, this is good enough.  As long as the clipping level remains the same in all files, the bias can be correctly subtracted and the files will remain linear for subsequent corrections (flat fielding).  However, I don't think anyone can confirm (at least so far no one has) that the clipping is a constant one.  What if the "smart" Nikon engineers decide to clip the data based on the exposure levels (or the overall brightness of the image)?  Then we are terribly screwed.  This can lead to what Marks saw, especially if the clipping is different in R, G, and B and changes from image to image, but this doesn't mean this is the right explanation.

 

Also, the dark subtraction based on optical dark pixels should be still there.  Mark's comparison between two flats in post #17 sort of shows that this either is not the cause, or at least is not the full picture.  In #17, Mark divided two flats that are supposed to be taken with relatively short exposure times.  The dark signal should be very small in both the active pixels and the optical dark pixels.  And yet the two flats still cannot correct each other.  So something else other than the optical dark pixel subtraction must be going on.

 

In short, I have no slightest idea about what causes the problem.

 

Cheers,

Wei-Hao

[AtmosFearIC]

I have both an ASI094 and a D810 so when I get some time I could do a little testing although getting any kind of clear skies at the moment seem to be a bit of a premium.

With my Mewlon 250 and ASI094 I do daylight flats with exposures around .01s and it seems to work quite well at the moment but I've only done it a hand full of times.

 

I have extensively used a D810 coupled to a 130mm F/5 refractor and don't really remember seeing any kind of purple around the edges. I've created my flats over time with two different flat boxes, one a custom made by a local Aussie and another is an Artesky Flat panel.

 

It has been quite a while since I used my D810 with my RH200 but when I was doing some testing in my Bortle 7-8 back yard I was getting serious flat fielding issues but I put that down to extreme vignetting caused by both the M48 connection at 55mm and the mirror box within the DSLR (I was getting to 90% vignetting at the extreme corners).

 

When I've used my ASI094 on the RH200 with M54 connections all the way to the chip vignetting has been below 25% out to the edges and I haven't noticed any purple vignetting. I think some of the purple in the image below comes from the lack of a UV/IR filter and from what I can tell, there is FAR too much IR leaking though the bayer. Much of the purple hue comes from purple stars which I think are blue stars with IR bleed. I have a ZWO UV/IR filter and a way of attaching it to the camera but without getting some different attachments made (shorter FLI Atlas to M54) I cannot get it to focus on the RH200 at the moment for further testing.

https://www.astrobin.../full/0h9d5q/0/

 

The following three images have been shot under dark skies and using a flat box. Possibly have a run of ABE with a scale of 1 for gradient removal and most likely have SCNR green removal. I don't remember seeing any purple so if it's a firmware issue is hasn't been in the D810 or D7200 as far as I can tell.

ASI094

D810

D7200

[sharkmelley]

I've had a new idea about a potential cause of the issue here i.e. that flats taken with different colour light sources fail to calibrate one another.  In particular white flats or dusk sky flats fail to exactly calibrate orange/brown light polluted exposures. 

 

Could it be sensor crosstalk?  This is something that occurs in the colour filter array (CFA) where, for instance, the electron generated by a photon passing through the filter for a green pixel ends up in the adjacent red or blue photo-site.  And vice versa of course.

 

The Nikon Z6 uses back side illuminated sensors which should have lower crosstalk than front side illuminated sensors.  However electrical crosstalk still exists, which is the diffusion of electrical charge between adjacent photo-sites.

 

Mark

Edited by sharkmelley, 06 December 2019 - 06:23 AM.