304 replies to this topic

[gezak22]

I've made a little bit of progress on the "coloured bands" issue.  The breakthrough was finding the same artefacts in my PixInsight integration rejection maps.

 

Here is the stretched image alongside a rejection map which I processed to enhance the artefacts:

 

WeirdGradients2.jpg

[click on image to see it full size]

 

 

If you don't know where to find them, I've highlighted the coloured bands in the image below:

 

WeirdGradients3.jpg

 

 

There are two big swirls of red/blue marked by the red pen.  These are the bands I first noticed in the image.  In addition there are some yellow concentric rings marked by the blue pen.  Knowing the position of the yellow rings I was able to find them in the image, though I hadn't noticed them previously.

 

These coloured bands do not exist in the calibration frames but only in the lights.   However, knowing where to look I've gone back to my previous images and found the same coloured bands in the same place in the rejection maps and highly stretched integrations of 3 out 4 of the images I've taken so far.  Each image was 60x 2min at ISO 800 giving 2 hours of integration time.  In the 4th image the position of the bands had moved - the swirls and concentric rings were more widely spaced and fainter.  The first 3 images were taken under more or less identical sky conditions (SQM around 20.8) with the back-of-camera histogram peak at the 1/5 to 1/4 position.  The 4th image was taken nearer the horizon and had brighter light pollution background.

 

This 4th image might provide a clue. Moving the histogram peak further to the right in the back-of-camera histogram might overcome the issue.  This was certainly the case when I encountered colour banding in my Sony A7S images.  Before I can be certain, it will be necessary to set up some indoor test conditions where I can reliably reproduce these effects.

 

In general, with very faint artefacts such as these, they only begin to appear when stacking a large number of images because this reduces the noise level down to the point at which they are potentially visible with heavy stretching.  Many users will never encounter them because they are not stacking large numbers of exposures and are not stretching images sufficiently to show, or instance, the colour in faint clouds of background dust.

 

Mark

The Sony A7R3 has a history of giving colored bands on certain lenses, even when shooting uncompressed raws with all lens compensation settings turned off. It's only obvious on certain lenses and even then only when the background is mostly white or black. The only solution I have found so far was to tape up the electrical contacts on the lens, as demonstrated in this post. I am assuming your camera makes no electrical contact with the optics (as you are using a telescope)?

 

 

Newton's rings? 

From where?

[sharkmelley]

The Sony A7R3 has a history of giving colored bands on certain lenses, even when shooting uncompressed raws with all lens compensation settings turned off. It's only obvious on certain lenses and even then only when the background is mostly white or black. The only solution I have found so far was to tape up the electrical contacts on the lens, as demonstrated in this post. I am assuming your camera makes no electrical contact with the optics (as you are using a telescope)?

 

Ah yes - those "Sony A7xxx Posterization and Colored Banding" threads examining the crude lens compensation applied to raw files.  I contributed a lot to those threads but some folk find it so difficult to accept evidence and blame the user instead 

 

I'm using the Nikon Z6 on a telescope and therefore no electrical lens contacts.  The problem is therefore nothing to do with lens compensation.

 

Newton's rings? 

I wish it were.  Unfortunately I will shortly show conclusive evidence it is not Newton's rings.

 

Mark

Edited by sharkmelley, 02 September 2019 - 03:10 PM.

[sharkmelley]

I've managed to set up a preliminary experiment to reproduce the problem.  Inside my closed and dimly lit observatory I put a diffuser over the telescope and shot 30 exposures at ISO 800 firstly at 0.25sec then 0.5sec then 1.0 sec. 

 

Here's a typical flat frame produced:

 

 

 

No artifacts are visible because this integrated image has not been flat fielded, background subtracted and stretched.  However the PixInsight rejection maps are very interesting:

 

[click on image to see it full size]

 

From left to right are the rejections maps for the 0.25sec exposures, the 0.5sec exposures and the 1.0 sec exposures.  The spacing between the coloured bands halves each time.  But what is this telling us?  I don't know yet.

 

Mark

Edited by sharkmelley, 02 September 2019 - 05:28 PM.

[sharkmelley]

Here's the integration of 30 "flat field" exposures of 0.5sec:

 

 

 

Here's the rejection map:

 

 

 

For the integrated flat, what I've now done is to divide both the (bias subtracted) red channel and the (bias subtracted) blue channel by the (bias subtracted) green channel:

 

[click on image to see it full size]

 

The discontinuities align precisely with the coloured bands in the rejection map.  They characterise the tonal variations and they conclusively show that one set of bands is related to the red channel and the other set of bands is related to the blue channel.

 

But what internal data processing is going on to cause these discontinuities?  Why should there be obvious steps in the red and blue channels relative to the green?  I don't have an answer yet.

 

Mark

Edited by sharkmelley, 02 September 2019 - 05:45 PM.

[whwang]

Let me guess.

 

Are the blue and red channels that suffer from the data scaling?  That introduces discontinuities in the histogram, right?  In the flat image, since pixel brightness is a strong function of location in the image, the tonal discontinuities also occur in certain locations.  And when you change the exposure, the locations also change.  Even when you don't change exposure time, there is a small frame-to-frame brightness variation, so the locations of the discontinuity also slightly shift, and this shows up in the rejection map.

Edited by whwang, 02 September 2019 - 08:00 PM.

[sharkmelley]

Let me guess.

 

Are the blue and red channels that suffer from the data scaling?  That introduces discontinuities in the histogram, right?  In the flat image, since pixel brightness is a strong function of location in the image, the tonal discontinuities also occur in certain locations.  And when you change the exposure, the locations also change.  Even when you don't change exposure time, there is a small frame-to-frame brightness variation, so the locations of the discontinuity also slightly shift, and this shows up in the rejection map.

What you have described was also my first guess.  It's exactly what happens on the Sony A7S and explains the coloured concentric rings that were a big issue for me on the Sony (the Sony scales all 3 colour channels).  But here on the Nikon Z6 it doesn't explain one very important feature - why are the rings (the red/green in particular) so eccentric?  Why don't they align with the overall changes in brightness (i.e. the vignetting) which are more or less circular and definitely not eccentric?

 

That doesn't mean it's the wrong explanation though.  It might simply mean I don't fully understand the chain of consequences caused by the histogram gaps. 

 

I have thought about this further and have designed another interesting experiment I can run tonight.  I have a method to remove the histogram gaps - i.e. to undo the scaling of the red and blue channels that has been applied by the firmware.  I can apply this repair to every individual raw file.  Will it also remove these anomalies from the stack?

 

By the way, I have a hunch that what I'm seeing may well be related to Jerry's problem with the Nikon D5300 here:

https://www.cloudyni...ipping-related/

Again this was a problem that appeared when pushing the camera to its limits - not a problem that everyone would encounter.  But we never fully understood what was going on.

 

Mark

Edited by sharkmelley, 03 September 2019 - 12:54 AM.

[sharkmelley]

I didn't think the cause of the coloured bands was the histogram gaps because the 2 sets of bands didn't align with each other nor with the brightness contours in the image.  Anyway I ran a script that removes the histogram gaps from the raw data.  The easiest way to think about what I do is that an inverse scaling is applied to the red and blue channels but it's a bit more complicated than that.

 

In any case, this failed to remove the coloured bands, so it confirms that the histogram gaps are not the problem.  For now, I've run out of further ideas as to what may be causing this.

 

I've already shown that the position of the bands moves around depending on the recorded light intensity.  So if the light conditions (i.e. the background light pollution brightness) gradually change during an imaging session then this would tend to smear the bands and make them less obvious.  I guess I was unlucky that my sky brightness didn't change at all during my imaging period.

 

The intensity of the bands is very low indeed.  They are only seen after stacking i.e. when the noise floor has dropped to the point where they are visible. So I'm thinking about taking the same approach I did with the Sony A7S i.e. to shoot with back-of-camera histogram up near the hallway point.  Yes this kills the dynamic range but when you are imaging very faint dust clouds, the colour of the dust matters more than saturating the brighter stars.

 

Further experiments are required ...

 

Mark

Edited by sharkmelley, 04 September 2019 - 03:56 PM.

[whwang]

I am thinking it is possible that the band does get smeared out in the stacked light because of the change in sky brightness if all you do is to stack the lights.  On the other hand, it doesn't get smeared out enough in the stacked flat, so this leaves a fixed pattern in the calibrated lights which doesn't get stacked away.

[sharkmelley]

I am thinking it is possible that the band does get smeared out in the stacked light because of the change in sky brightness if all you do is to stack the lights.  On the other hand, it doesn't get smeared out enough in the stacked flat, so this leaves a fixed pattern in the calibrated lights which doesn't get stacked away.

Yes that's a potential problem.  The pattern can certainly exist in either the lights or the flats.  My flats were taken with the back-of-camera histogram just at the point of hitting the right hand side. In the master flat here are the red and blue channels divided by the green:

 

[click image to see full size]

 

There are no rings in the blue channel and only a slight hint of rings in the red channel - both displayed with PixInsight STF AutoStretch.  In the case of my image of the Iris Nebula NGC 7023, it was definitely the lights that contained the obvious pattern of rings.

 

Luckily tonight is clear so I am re-imaging NGC 7023 with 2min exposures at ISO 3200 instead of ISO 800.  The back-of-camera histogram is now at the 1/2 position.  Let's see what happens ...

 

Mark

[tkottary]

Could this be due to  green led that blinks during long exposures?  

[sharkmelley]

Could this be due to  green led that blinks during long exposures?  

It doesn't fit the observed symptoms. How would it cause a pattern of rings whose positions and spacings change with length of exposure?

 

Mark

Edited by sharkmelley, 05 September 2019 - 02:17 AM.

[maxmir]

Internal reflections, try a different scope or a lens.

[sharkmelley]

Internal reflections, try a different scope or a lens.

It doesn't fit the observed symptoms. How would it cause a pattern of rings whose positions and spacings change with length of exposure?

 

Mark

[sharkmelley]

Shooting at ISO 3200 instead of ISO 800 has made a huge difference.  It resulted in the back-of-camera histogram peak being just a tiny bit beyond the halfway point.

 

Here's the before and after - as far as possible I've applied identical post-processing to both:

 

[click image to see full size]

 

The sharply defined coloured bands have disappeared and the left-right discontinuity has almost gone.

However, there is a now a smooth green to purple gradient from the middle to the edges that I don't fully understand. Note that the only background subtraction I have performed is a PixInsight linear ABE - no higher order functions were used and no DBE.

There is also a purple margin top and bottom which seems to correspond to the area of the sensor with no PDAF (autofocus pixel) rows.  This needs some further analysis.

 

So, although it's not perfect, gradient removal will be much easier to perform on this second image.

 

Mark

Edited by sharkmelley, 05 September 2019 - 06:26 PM.

[whwang]

Now it looks more like the nonlinearity I saw in my D800 images.

[sharkmelley]

Now it looks more like the nonlinearity I saw in my D800 images.

Interesting that you have already encountered similar behaviour in a Nikon camera.

 

To me it looks like something "interesting" is taking place in the bias frames.  For instance, a possible systematic deviation of the red/blue channels away from the green channel would entirely explain the purple/green behaviour in the ISO 3200 image above. 

 

I've already discovered something very odd occurring in the histograms of the bias frames but no-one was able to explain it:

https://www.dpreview...s/post/62937140

 

I also seem to remember reading that some CMOS astro-cameras have weird bias behaviour for which the solution is to take longer exposure bias frames.  Other astro-cameras have a problem where the Sony CMOS sensor "recalibrates" after being switched off, giving a different bias frame each time:

https://www.cloudyni...w-cmos-cameras/

 

I'll do some more experimentation ...

 

Mark

Edited by sharkmelley, 06 September 2019 - 12:58 AM.

[whwang]

BTW, not just on an unhacked D800, D810A also has such a behavior.  I found this on a D810A borrowed from a friend.  Then another friend who owned a D810A told me that he sees similar things, all the time.  I encouraged him to dig out his old Canon 5D2 files and do a proper processing.  And guess what?  It's there too.  Among all these, the unhacked D800 is the worst, because of its severe black clipping, I believe.  It's mild on D810A and 5D2, but is nevertheless noticeable as long as the calibration is done properly for this to show up.

 

My theory is that this is caused by the in-camera subtraction of the optical dark pixels.  The over-subtraction (since those pixels should be warmer) introduces a shift in the zero in the light frames.  This makes the file nonlinear.  The firmware hack disables this, so images from a hacked D800 is entirely free of this, while other cameras still suffer.  There may be ways to test this theory, if we know under what condition the in-camera subtraction will engage.

[sharkmelley]

I agree with what you are saying but there is something additional happening on the Nikon Z6 - partly due to the PDAF banding and partly due to the internal treatment of the region near the vertical split in the sensor.

 

I've taken the ISO 3200 data and performed a direct sum of the dithered exposures then displayed both the red and blue channels divided by the green:

 

[click image to see larger version]

 

You can clearly see the top and bottom margin where there are no PDAF pixels.  You can also clearly see the left-right discontinuity. 

 

[continued on next post ...]

Edited by sharkmelley, 07 September 2019 - 02:16 AM.

[sharkmelley]

[continued from previous post]

 

Unfortunately the effects of the PDAF banding never entirely calibrate out, as discussed earlier in this thread.  You can see this in the calibrated version:

 

[click on image to see larger version]

 

There is still a slight top and bottom margin and the left-right discontinuity has not completely disappeared.  Note that I'm being very picky here - it will be very rare to stretch the stacked data to the point where these effects become visible and even when you do so, gradient removal strategies can be used to fix it in post-processing.  But it's good to be aware of where these issues might occur so they can be targeted for post-processing treatment.

 

I don't think there is much more I can say on this subject.  The important thing is to shoot the data so the sharply defined concentric coloured bands do not appear because they would be a complete nightmare to remove in post-processing. 

 

Specifically my recommendation is the following:

• Shoot lights with the back-of-camera histogram peak at the halfway point
• Shoot flats with the histogram well over to the right
• Don't use 12bit raw file format.  Although it makes the PDAF banding disappear, it makes the coloured banding problem much more visible

If you attempt to use the Nikon Z6 to image those really faint inter-stellar dust clouds at low ISO (with the histogram over to the left) in an effort to preserve your dynamic range then you risk encountering issues with concentric coloured bands.  This is especially the case if the brightness of the background sky is fairly constant during the imaging session because the position of the coloured bands will end up being identical in every exposure.

 

Mark

Edited by sharkmelley, 07 September 2019 - 02:36 AM.

[sharkmelley]

I went away and I thought about the problem(s) a bit more.

 

We are seeing 3 problems:

• PDAF banding that does not calibrate away
• A distinct pattern of coloured bands which changes according to the colour and intensity of the background (I still don't understand their cause).
• A left-right discontinuity whose intensity and associated coloured fringing also changes with background level

So what would happen if I shot flats whose background is the same colour and intensity as the lights?  In order to find out, I managed to do this by using a diffuser on the scope and illuminating the ceiling of the observatory with coloured light from a laptop screen.

 

Here is an example ISO 3200 light from the previous imaging session, side by side with one of the new flats - both are out of camera JPGs:

 

[click on image to see larger version]

 

They are not a perfect match but are close enough.

 

Here is the resulting stacked image using this carefully created master flat:

 

 

 

 

The image has been white balanced and stretched in an identical manner to the earlier ISO800 vs ISO 3200 comparison, so it is directly comparable to the images in that earlier post: https://www.cloudyni...ting/?p=9622305

 

The new master flat perfectly calibrated away the PDAF banding, removed the residual left-right vertical discontinuity and it also removed the radial green to purple gradient. 

 

I'm quite astonished by the success of this approach.  It remains to be seen if this is just a lucky coincidence or if it is repeatable on future images.

 

Mark

Edited by sharkmelley, 11 September 2019 - 07:08 PM.

[Jerry Lodriguss]

Did you try the same illumination at lower ISO?

 

Jerry

[AgilityGuy]

That’s an unexpected success. I’ve had a similar problem with my D7200 on various targets and just stopped using it in favor of a full spectrum D800e which has not had this problem.

What were the exposure times for the flats taken against the top of the roof and were they exposed to approximately 50 percent intensity as recommended?

Still mentally processing the results. Likely to have more questions soon.

[sharkmelley]

Did you try the same illumination at lower ISO?

 

Jerry

I need to try this.  I think it will be problematic because the coloured bands are much more sharply defined at lower ISO and therefore impossible to match precisely because small changes in sensor illumination shifts their position.

 

That’s an unexpected success. I’ve had a similar problem with my D7200 on various targets and just stopped using it in favor of a full spectrum D800e which has not had this problem.

What were the exposure times for the flats taken against the top of the roof and were they exposed to approximately 50 percent intensity as recommended?

Still mentally processing the results. Likely to have more questions soon.

The exposure time for the flats was 4 sec (vs 2 min for the lights) because the observatory was lit to a much brighter level than the night sky.  The flats were exposed to exactly the same level as the lights - as far as I could.  It took many iterations of changing the laptop display until I got it just right.

 

Mark

[Bill G.]

Ok, could we dumb this down a bit?

 

I'm getting back into AP (last serious AP was with a Canon 300D I modded) and now have an unmodded D7000 (star eater hack installed). Got a new CGX month or so ago. Z6 (un-modded) to be delivered tomorrow. Although not actually purchased for AP, I would like to give it a shot.

 

So, looking for any recommended starting points for either camera. In particular, the Z6 based on all this testing.

These may not be the best cameras for the job...but they're what I have....still have the 300D if necessary!!

Thanks,

Bill G.

 

BTW, will be using an ED80 and a C8 with these. Have a .63 WO reducer but have not used full frame before so would DX mode on the Z6 be recommended? 

Edited by Bill G., 11 September 2019 - 07:56 PM.

[whwang]

Hi Bill, I think you will find Z6 a wonderful camera for astro.  The "issues" Mark found here will not show up until you really push the camera to its limit.

 

Mark, this result is quite remarkable.  When I encountered this on my D800, I did tried to bring the color of my flat to brownish (instead of bluish, from the twilight), to better match with the color of night sky.  This improved it a little bit.  However, I did not go as far as you did.  I did not expect that exactly matching the color and brightness can completely solve the problem.

 

Based on what you find, at least on Z6, we can say that the problem is not caused by over-subtraction of dark from the optical dark pixels.  It still doesn't provide an answer to what causes the problem, but at least it eliminates one possibility.

 

Cheers,

Wei-Hao