276 replies to this topic

[whwang]

Hi Mark,

 

Which adapter you use for Z6 on Epsilon?  The standard TAK Nikon wide-mount plus a Nikon F to Z adapter?  If so, the illumination may be still limited by the F-mount opening.  What will be interesting is if there can be a TAK native wide-mount for Z-mount.  That will take full advantage of the large Z-mount opening.

[sharkmelley]

Hi Mark,

 

Which adapter you use for Z6 on Epsilon?  The standard TAK Nikon wide-mount plus a Nikon F to Z adapter?  If so, the illumination may be still limited by the F-mount opening.  What will be interesting is if there can be a TAK native wide-mount for Z-mount.  That will take full advantage of the large Z-mount opening.

I bought an inexpensive Canon EOS adapter with a circular opening: https://www.amazon.c...n/dp/B07MT8D18M

It makes the Nikon Z6 look like a Canon camera to all my existing astro equipment.  On the Tak it then connects to my Tak wide-mount Canon adapter.  I would certainly buy a Tak native wide-mount Z-adapter, if and when available.

 

I wanted to keep expenses to an absolute minimum until I was certain to keep the camera.  However, I think that decision is already made.  

I always knew I would ditch Sony when the right camera came along.  My next decision is whether or not to perform an H-alpha mod.  But I'll get some more imaging results first.

 

Mark

Edited by sharkmelley, 15 August 2019 - 03:58 AM.

[sharkmelley]

Master Dark Subtraction

 

I have begun to notice that the subtraction of the master dark is not working very effectively.  Here's an example (a crop at 200% scale)  where the master dark (on the left) is subtracted from a stack of lights (the middle pane) resulting in a dark subtracted image (on the right):

 

[click on image to see it full size]

 

The middle pane seems to have a fair number of apparent warm/hot pixels that do not appear in the master dark and are therefore not subtracted.  This is likely to be a side effect of the spatial filtering.  You may remember that a similar side effect was discovered on the Nikon D5300:  https://www.cloudyni...ring/?p=9082246

 

 

The issue appears to be that some warm/hot pixels are removed from the darks by the spatial filter but the spatial filter does not remove them from the lights.  Unfortunately the spatial filtering algorithm on the Z6 is far more complex than the D5300 so it may never be possible to understand the exact details of what is happening. 

 

My thinking is that there are 3 distinct populations of warm/hot pixels, depending on their relative brightness:

• Those that are removed by filtering in both the darks and lights
• Those that are not removed by filtering in both the darks and lights
• Those that are removed by filtering in the darks but not in the lights

I'll run an experiment very similar to the one I performed on the D5300 in order to understand it a bit better. 

 

In practice I think that dithered acquisition together with sigma stacking is likely to deal with the issue so it's not likely to be a big problem.

 

Mark

Edited by sharkmelley, 18 August 2019 - 06:55 PM.

[SandyHouTex]

Master Dark Subtraction

 

I have begun to notice that the subtraction of the master dark is not working very effectively.  Here's an example (a crop at 200% scale)  where the master dark (on the left) is subtracted from a stack of lights (the middle pane) resulting in a dark subtracted image (on the right):

 

NikonZ6DarkSubtraction.png

[click on image to see it full size]

 

The middle pane seems to have a fair number of apparent warm/hot pixels that do not appear in the master dark and are therefore not subtracted.  This is likely to be a side effect of the spatial filtering.  You may remember that a similar side effect was discovered on the Nikon D5300:  https://www.cloudyni...ring/?p=9082246

 

 

The issue appears to be that some warm/hot pixels are removed from the darks by the spatial filter but the spatial filter does not remove them from the lights.  Unfortunately the spatial filtering algorithm on the Z6 is far more complex than the D5300 so it may never be possible to understand the exact details of what is happening. 

 

My thinking is that there are 3 distinct populations of warm/hot pixels, depending on their relative brightness:

• Those that are removed by filtering in both the darks and lights
• Those that are not removed by filtering in both the darks and lights
• Those that are removed by filtering in the darks but not in the lights

I'll run an experiment very similar to the one I performed on the D5300 in order to understand it a bit better. 

 

In practice I think that dithered acquisition together with sigma stacking is likely to deal with the issue so it's not likely to be a big problem.

 

Mark

I've been following the thread, "Is Sony Really Alpha", and I was wondering, since Nikon gets their sensors from Sony, do you expect the same IR light issue that folks have been discussing over there on your Nikon?  Are you going to have it modded?

Edited by SandyHouTex, 19 August 2019 - 09:53 AM.

[sharkmelley]

I've been following the thread, "Is Sony Really Alpha", and I was wondering, since Nikon gets their sensors from Sony, do you expect the same IR light issue that folks have been discussing over there on your Nikon?  Are you going to have it modded?

At first I will use it unmodded but I may well convert it later on.

 

It's the shutter mechanism in the Sony cameras that causes the problem, not the sensor, so there is no reason for the problem to transfer across to Nikon cameras.  Indeed KolariVision offers full spectrum conversion for the Nikon Z6 and Z7.  However, for the affected Sony cameras they have a warning about the internal IR LED.

 

Also there is an KolariVision article about IR Infrared Photography on the Nikon Z7 here with an example dark frame showing no problem:

https://kolarivision...ography-review/

 

I've seen complaints about PDAF Banding in UV and IR images (fixable with RawTherapee) but not nothing about internal LEDs.

 

Mark

Edited by sharkmelley, 19 August 2019 - 10:33 AM.

[tkottary]

I have a modded z6 . If you want some data for analysis I’d be happy to share .

[sharkmelley]

I have a modded z6 . If you want some data for analysis I’d be happy to share .

That would be great!  A raw file of a single 5min dark at ISO 800 would tell us all we need to know.  If you can take it with the camera at a low ambient temperature then even better.

 

Mark

[sharkmelley]

Raw File Settings

 

I should point out that I am doing most of my testing using PixInsight.  PixInsight does not yet support raw Nikon Z6 NEF files directly because they are not yet supported in a release version of LibRaw.  So I use Adobe DNG Converter to create DNG versions of all my NEF raw files.  This is a surprisingly fast operation and PixInsight supports DNG.

 

I need to correct this earlier statement.  All raw file types except "Uncompressed" are already supported by PixInsight, DeepSkyStacker, IRIS and anything else that uses DCRaw or LibRaw.

 

To be explicit, the following settings under the camera's "NEF (RAW) recording" menu item are widely supported by astro-processing software:

• 14 bit Lossless compressed
• 14 bit Compressed
• 12 bit Lossless compressed
• 12 bit Compressed

Uncompressed is not yet supported because it's not yet in a release version of LibRaw.  But why would anyone want uncompressed in preference to lossless compression?  I have used 14-bit Lossless compression for all my testing.

 

AstroPixelProcessor (which doesn't use DCRaw or LibRaw) currently only supports 12-bit and 14-bit Losslessly compressed files.

 

While we're on this subject, never mix 12-bit files with 14-bit files because the pixel values are completely different.  For instance 14-bit bias frames have a mean value of 1008 but 12-bit bias frames have a value 4x lower i.e. 252.  The same applies to darks, flats and lights.

 

Mark

Edited by sharkmelley, 20 August 2019 - 01:56 PM.

[Jerry Lodriguss]

Hi Mark,

 

What do you think of ISO 800 and 14-bit lossless compressed to work best to minimize PDAF banding? Or do we need to go to 12 bits?

 

I'm saying ISO 800 because that looks like where the second stage gain kicks in to significantly lower the read noise from 5.35e- to 1.70e- and it goes mostly ISO invariant.

 

What do you think?

 

Jerry

[sharkmelley]

Hi Mark,

 

What do you think of ISO 800 and 14-bit lossless compressed to work best to minimize PDAF banding? Or do we need to go to 12 bits?

 

I'm saying ISO 800 because that looks like where the second stage gain kicks in to significantly lower the read noise from 5.35e- to 1.70e- and it goes mostly ISO invariant.

 

What do you think?

 

Jerry

I'm using ISO 800 because that's where the High Gain Conversion (HGC) and low read noise kicks in.

 

In 14 bit mode for Deep Sky Imaging, if your back-of-camera histogram is at least 1/8 from the left then PDAF banding should not be an issue - especially if you use dithering.

 

The amplitude of the banding is approx. 1.5 digital units in 14 bit mode and putting the histogram in the right place will swamp it with noise.  In 12 bit mode the amplitude is 10x less so it will never be visible.  In 14 bit terrestrial daylight photography, the banding is only ever visible in shadow areas, where it is not swamped by noise.

 

Mark

Edited by sharkmelley, 20 August 2019 - 03:26 PM.

[sharkmelley]

Master Dark Subtraction continued

 

I have now carried out a careful experiment on dark subtraction. I shot the following exposures at room temperature:

• 50x 1min ISO 800 darks
• 50x 1min ISO 800 light frames with the back-of camera histogram peak at the 1/4 position
• 50x 1min ISO 800 light frames with long exposure noise reduction (LENR) switched on

Master frames were created by taking the average of each set.  A dark-subtracted version was created by subtracting the master dark from the master light.  For the master dark, here is a plot of pixel values against the maximum of their 24 neighbours in a 5x5 block:

 

 

It contains clear evidence of raw filtering because for instance there are no isolated warm/hot pixels above a level of around 1075.  So the question is whether or not this filtering affects the efficacy of dark subtraction.

 

The answer is here:

 

[click on image to see it full size]

 

It would appear that the dark subtraction has actually worked very well but with just a few bright pixels left behind.  These are bright pixels whose values were capped in the dark exposures but remained unfiltered in the light frames.  The master dark is therefore unable to remove them.

 

It's interesting to note that even LENR has not worked perfectly - the master LENR has a few remnants of the bright pixels seen in the master dark.

 

All in all, the darks on the Nikon Z6 are very clean and dithering is likely to be just as effective as taking darks.  Good news for those too lazy to take darks!

 

I ought to point out that the PDAF banding is not visible in the individual exposures.  It is only visible in the master frames because stacking has reduced the noise levels to the point where the banding becomes visible.

 

Mark

Edited by sharkmelley, 21 August 2019 - 03:49 PM.

[tonyt]

 

All in all, the darks on the Nikon Z6 are very clean and dithering is likely to be just as effective as taking darks.  Good news for those too lazy to take darks!

 

[sharkmelley]

I've now taken my first serious image - over on a new thread:

https://www.cloudyni...z6-first-light/

 

Mark

[RobertM]

Hi Mark, 

 

I’ve not seen results this impressive since the A7S.  Have all those four PI darks been stretched the same amount or did you use the automatic screen stretch ?

 

Robert

[sharkmelley]

Hi Mark, 

 

I’ve not seen results this impressive since the A7S.  Have all those four PI darks been stretched the same amount or did you use the automatic screen stretch ?

 

Robert

Hi Robert,

 

No, I didn't use the screen transfer function.  Each one was scaled by the same amount using the same linear multiplier in PixInsight's PixelMath.  

 

Mark

Edited by sharkmelley, 26 August 2019 - 05:24 PM.

[sharkmelley]

Colour Channel Sensitivities and H-alpha Response

 

The one line summary is that the Z6 has noticeably higher red channel sensitivity than the A7S but there is nothing special happening at the H-alpha wavelength i.e. the increase in H-alpha sensitivity is in line with the overall red channel increase.

 

For those interested, what follows is the detailed description.

 

The Nikon Z6, just like other Nikon cameras, applies white balance pre-conditioning to its raw data.  In other words, the raw digital data of the red and blue channels is scaled, which in the case of the Z6 leads to regularly spaced gaps in the red and blue histograms:

 

[click on image to see it full size]

 

Counting the number of gaps leads to a good estimate of the scaling factor applied.  In this case it is approx 1.18 for both the red and blue channels.  I have confirmed this by calculating the camera gain values from flat frames and bias frames.

 

Now let's take a look at the daylight white balance scalars provided by DXO (the "Color Response" tab at https://www.dxomark....---Measurements) and compare them with the Sony A7S DXO white balance scalars:

• Nikon Z6 RGB scalars are:   1.75  1.00  1.34
• Sony A7S RGB scalars are:  2.54  1.00  1.65

 

This is telling us that the sensitivity of the red and blue channels (relative to the green channel) in the Nikon Z6 is higher than the Sony A7S, even when the factor of 1.18 is taken into account:

• For the red channel we see 2.54/1.75/1.18 = 1.23  i.e. 23% higher
• For the blue channel we see 1.65/1.34/1.18 =  1.04  i.e. 4% higher

 

Earlier in this thread, by taking the 2 cameras and shooting the same target, I estimated that the QE of Nikon Z6 green channel was 14% higher than the Sony A7S.  Unfortunately, further testing showed this result was an outlier and a better estimate is close to 5% i.e. the Nikon Z6 captures 5% more photons in the green channel than the A7S.  Given the margins of error I am seeing, it is possible that the Z6 has anywhere in the range 0%-10% higher green sensitivity than the A7S. In any case, this is not a noticeable difference in performance.

 

Combining the two results, we estimate the absolute sensitivity of red channel of the Z6 has improved by 1.23*1.05 = 1.29 i.e. 29% higher.  This is an improvement worth having.

 

Now for the H-alpha response.  The easiest way to test this was to put a diffuser over my scope pointing up at a clear daylight sky and an H-alpha filter in the camera adapter.  I could then quickly switch cameras between my Nikon Z6 and modified Sony A7S taking the same lengh exposure, both with electronic shutter for best accuracy.  Using my calculated gain factors and allowing for the Nikon's 1.18 scaling of the red channel, this showed that the modified Sony was capturing 2.96x the number of photons (per unit area of sensor) of the Nikon Z6.  So assuming the manufacturer's filter is cutting 75% of the H-alpha transmission then a modified Z6 would capture 4x as many H-alpha photons giving a ratio of 4/2.96 = 1.35 i.e. 35% higher sensitivity.  Given the margins of error, this is pretty similar to the 29% increase in the red channel as a whole.  In other words I see no evidence of anything special happening to the sensitivity near the H-alpha wavelength. 

 

Mark

Edited by sharkmelley, 29 August 2019 - 05:01 PM.

[RobertM]

Thanks for the insight Mark, that's a very interesting comparison and analysis.

 

Robert

[sharkmelley]

Thanks for the insight Mark, that's a very interesting comparison and analysis.

I just need to write a conclusion now

 

Mark

[sharkmelley]

Before concluding I have suddenly hit a problem that requires some investigation.

 

My latest image has some sharply defined weird background colour gradients when stretched:

 

 

It's too early to know the cause.  I used the same ISO and exposure time as the previous images and used the same scope. 

 

Mark

Edited by sharkmelley, 31 August 2019 - 11:56 AM.

[sharkmelley]

I've made a little bit of progress on the "coloured bands" issue.  The breakthrough was finding the same artefacts in my PixInsight integration rejection maps.

 

Here is the stretched image alongside a rejection map which I processed to enhance the artefacts:

 

[click on image to see it full size]

 

 

If you don't know where to find them, I've highlighted the coloured bands in the image below:

 

 

 

There are two big swirls of red/blue marked by the red pen.  These are the bands I first noticed in the image.  In addition there are some yellow concentric rings marked by the blue pen.  Knowing the position of the yellow rings I was able to find them in the image, though I hadn't noticed them previously.

 

These coloured bands do not exist in the calibration frames but only in the lights.   However, knowing where to look I've gone back to my previous images and found the same coloured bands in the same place in the rejection maps and highly stretched integrations of 3 out 4 of the images I've taken so far.  Each image was 60x 2min at ISO 800 giving 2 hours of integration time.  In the 4th image the position of the bands had moved - the swirls and concentric rings were more widely spaced and fainter.  The first 3 images were taken under more or less identical sky conditions (SQM around 20.8) with the back-of-camera histogram peak at the 1/5 to 1/4 position.  The 4th image was taken nearer the horizon and had brighter light pollution background.

 

This 4th image might provide a clue. Moving the histogram peak further to the right in the back-of-camera histogram might overcome the issue.  This was certainly the case when I encountered colour banding in my Sony A7S images.  Before I can be certain, it will be necessary to set up some indoor test conditions where I can reliably reproduce these effects.

 

In general, with very faint artefacts such as these, they only begin to appear when stacking a large number of images because this reduces the noise level down to the point at which they are potentially visible with heavy stretching.  Many users will never encounter them because they are not stacking large numbers of exposures and are not stretching images sufficiently to show, or instance, the colour in faint clouds of background dust.

 

Mark

Edited by sharkmelley, 02 September 2019 - 04:57 AM.

[whwang]

Hi Mark,

 

Interesting finding.  Do you know what's the cause.  I have seen similar things on D810A and my unhacked D800.  It's gone after the hacking.  So on D800 and D810A, I am quite sure it comes from the black clipping or the optical dark pixel subtraction, or both.  I am not sure whether it's the same thing on Z6.

 

On your example image, in additional to the big swirling bands, I also noticed that there is a vertical sharp boundary at the center of the image.  The image is greener to the left of the boundary and more purple to the right.  But that doesn't appear in your rejection map.  Do you know what it is?

 

Cheers,

Wei-Hao

[sharkmelley]

On your example image, in additional to the big swirling bands, I also noticed that there is a vertical sharp boundary at the center of the image.  The image is greener to the left of the boundary and more purple to the right.  But that doesn't appear in your rejection map.  Do you know what it is?

 

I forgot to mention that! 

 

The Sony A7S had something similar.  I am told that Sony makes these full frame sensors with a split down the middle.  The left hand side and right hand side act more or less independently and this can lead to very slight variations in intensity and/or colour at (or near) the boundary between the two halves. 

 

Mark

[whwang]

It's OK if the two halves behave independently.  What surprises me (a little bit) is that it doesn't go away after calibration.

[sharkmelley]

It's OK if the two halves behave independently.  What surprises me (a little bit) is that it doesn't go away after calibration.

I think it's not as simple as both halves acting completely independently.  I'm pretty sure there is also some kind of raw data processing happening across the entire sensor - certainly that seemed to be the case on the Sony A7S.  This data processing might then cause problems for calibration frames.  Maybe this explains what you saw on your D800.  It's possible that hacking the firmware switched off the additional data processing so calibration then worked effectively.

 

Mark

[Alen K]

I've made a little bit of progress on the "coloured bands" issue.  The breakthrough was finding the same artefacts in my PixInsight integration rejection maps.

 

Here is the stretched image alongside a rejection map which I processed to enhance the artefacts:

 

WeirdGradients2.jpg

[click on image to see it full size]

 

 

If you don't know where to find them, I've highlighted the coloured bands in the image below:

 

WeirdGradients3.jpg

 

 

There are two big swirls of red/blue marked by the red pen.  These are the bands I first noticed in the image.  In addition there are some yellow concentric rings marked by the blue pen.  Knowing the position of the yellow rings I was able to find them in the image, though I hadn't noticed them previously.

 

These coloured bands do not exist in the calibration frames but only in the lights.   However, knowing where to look I've gone back to my previous images and found the same coloured bands in the same place in the rejection maps and highly stretched integrations of 3 out 4 of the images I've taken so far.  Each image was 60x 2min at ISO 800 giving 2 hours of integration time.  In the 4th image the position of the bands had moved - the swirls and concentric rings were more widely spaced and fainter.  The first 3 images were taken under more or less identical sky conditions (SQM around 20.8) with the back-of-camera histogram peak at the 1/5 to 1/4 position.  The 4th image was taken nearer the horizon and had brighter light pollution background.

 

This 4th image might provide a clue. Moving the histogram peak further to the right in the back-of-camera histogram might overcome the issue.  This was certainly the case when I encountered colour banding in my Sony A7S images.  Before I can be certain, it will be necessary to set up some indoor test conditions where I can reliably reproduce these effects.

 

In general, with very faint artefacts such as these, they only begin to appear when stacking a large number of images because this reduces the noise level down to the point at which they are potentially visible with heavy stretching.  Many users will never encounter them because they are not stacking large numbers of exposures and are not stretching images sufficiently to show, or instance, the colour in faint clouds of background dust.

Newton's rings?