295 replies to this topic

[Jerry Lodriguss]

Hi Wei-Hao,

 

Could this be related to the backside-illuminated sensor? Something like Mischa Schimer talks about in "Principles of CCD Data Reduction"...

 

3.6 Defringing

That's an exotic topic, and if you don't have a back-illuminated CCD and take images in red or infrared bands with it, then you will be save from fringing. Nevertheless, I want to mention it here since you might be affected by it, and since it completes the picture. Fringing is an interference effect of red/infrared light in the thinned substrate of back-illuminated CCDs. In general, it looks like the wavy pattern in Fig. 9 below. You need a good SUPERFLAT to remove it. If you don't have any, then forget about it.

The example photos he shows don't look exactly like what Mark is showing though.

 

He goes on to describe how to solve the problem.

 

Jerry

Edited by Jerry Lodriguss, 11 September 2019 - 09:25 PM.

[Jerry Lodriguss]

I need to try this.  I think it will be problematic because the coloured bands are much more sharply defined at lower ISO and therefore impossible to match precisely because small changes in sensor illumination shifts their position.

 

The exposure time for the flats was 4 sec (vs 2 min for the lights) because the observatory was lit to a much brighter level than the night sky.  The flats were exposed to exactly the same level as the lights - as far as I could.  It took many iterations of changing the laptop display until I got it just right.

Hi Mark,

 

Check your spam folder, I sent you an email.

 

Jerry

[whwang]

Hi Jerry,

 

The problem doesn't look like fringing, at least not like the fringing I have seen on images taken from meter-class telescopes.

 

As mentioned in the text quoted by you, fringing is caused by interference.  However, different wavelengths have different interference patterns, and broad-band filters allow many different wavelengths to go through.  As a result, these different interference patterns tend to cancel each other out.  The exception is when a narrow-band filter is used, or when there are strong, sharp emission lines in the sky background.  The latter is common in the near-IR, as the sky is full of strong OH emission lines.  These lines can induce fringe patterns.  Since the emission is confined to narrow ranges of wavelengths, their interference patterns do not cancel each other out very well.  

 

On the other hand, because of the lack of strong, sharp sky lines in the optical, it is usually hard to see fringes in visible light under broad-band filters.  I think what Mark found here is something in the camera data processing, not in the sensor.

 

Cheers,

Wei-Hao

[Jerry Lodriguss]

The problem doesn't look like fringing, at least not like the fringing I have seen on images taken from meter-class telescopes.

 

As mentioned in the text quoted by you, fringing is caused by interference.  However, different wavelengths have different interference patterns, and broad-band filters allow many different wavelengths to go through.  As a result, these different interference patterns tend to cancel each other out.  The exception is when a narrow-band filter is used, or when there are strong, sharp emission lines in the sky background.  The latter is common in the near-IR, as the sky is full of strong OH emission lines.  These lines can induce fringe patterns.  Since the emission is confined to narrow ranges of wavelengths, their interference patterns do not cancel each other out very well.  

 

On the other hand, because of the lack of strong, sharp sky lines in the optical, it is usually hard to see fringes in visible light under broad-band filters.  I think what Mark found here is something in the camera data processing, not in the sensor.

Thanks Wei-Hao, just thought I would bring it up cause it sounded vaguely similar.

 

I'm not seeing these problems in the data I've worked with on the Z6 so far, but I'm not pushing it very hard.

 

Jerry

[sharkmelley]

Ok, could we dumb this down a bit?

 

I'm getting back into AP (last serious AP was with a Canon 300D I modded) and now have an unmodded D7000 (star eater hack installed). Got a new CGX month or so ago. Z6 (un-modded) to be delivered tomorrow. Although not actually purchased for AP, I would like to give it a shot.

 

So, looking for any recommended starting points for either camera. In particular, the Z6 based on all this testing.

These may not be the best cameras for the job...but they're what I have....still have the 300D if necessary!!

Thanks,

Bill G.

 

BTW, will be using an ED80 and a C8 with these. Have a .63 WO reducer but have not used full frame before so would DX mode on the Z6 be recommended? 

You will find the Nikon Z6 to be a very competent camera for astrophotography.  The only time anybody might hit problems with these coloured band artifacts is when trying to image very faint interstellar dust and even in that rare case I have described a workaround i.e. use a high ISO and histogram peak at the halfway point.  But that is a shooting strategy for that particular use-case and there is no need to use it for general astro-work.

 

Jerry is using this camera and hasn't hit the issue so far.

 

The ED80 and C8 will not produce aberration-free stars across a full-frame sensor but I would simply crop the final image. 

 

 

Could this be related to the backside-illuminated sensor? Something like Mischa Schimer talks about in "Principles of CCD Data Reduction"...

 

The example photos he shows don't look exactly like what Mark is showing though.

 

 

What I'm seeing does not look like those interference fringes.  Also, my coloured rings change position and spacing depending on the exposure brightness.

 

Regards,

 

Mark

Edited by sharkmelley, 12 September 2019 - 02:49 AM.

[sharkmelley]

Following the success of a colour and intensity matched flat on the ISO3200 lights I've now created one for the earlier ISO800 image that showed the original distinct coloured banding.

 

Here is the result, the original image on the left and the one created with the new master flat on the right:

 

[click image to see full size]

 

 

Clearly, it is not a success.  The intensity of the coloured bands has been reduced but the left-right discontinuity is just as bad or even worse than before.  The discoloration immediately to the left of the discontinuity (the intense green) appears to have also increased.

 

Mark

Edited by sharkmelley, 12 September 2019 - 04:26 PM.

[Bill G.]

Thanks for the responses. I don't expect to be pushing it that hard for awhile. Looking forward to giving it a try. I do appreciate the indepth stuff...even if a fair amount of it is well above me.

[whwang]

Hi Mark,

Is that the image with less exposure in a sub?

[sharkmelley]

Hi Mark,

Is that the image with less exposure in a sub?

Both the ISO 800 image and the ISO 3200 image of the Iris Nebula and nearby dust used 2 min exposures and the same total integration:  75 x 2min.

 

The only difference was the ISO and hence the position of the histogram peak on the back-of-camera display when taking those exposures.  But in the observatory the camera is still sitting on the telescope in exactly the same position, which allows me to experiment with different strategies for taking the flats.

 

Mark

Edited by sharkmelley, 13 September 2019 - 12:54 AM.

[sharkmelley]

We had a clear night tonight, so for the sake of interest I took a series of dusk sky flats at ISO 100 at every shutter speed from severely underexposed (1/1250sec) to slightly overexposed (1/6sec). 

 

Here is a montage where I've divided the blue channel by the green channel and stretched the data by a factor of 25:

 

[click on image for larger version]

 

For the same files here is the red channel divided by the green channel with the same data stretch of 25:

 

[click on image for larger version]

 

I don't think I've got much more to say on the subject and I've still got no idea of the cause of these artifacts.  I think I've disappeared down this particular rabbit hole for too long.  It's time to re-emerge

 

Mark

Edited by sharkmelley, 13 September 2019 - 06:09 PM.

[Traveler]

Can it be caused by somesort of automatic lens correction routine in the camera software?

[sharkmelley]

Can it be caused by somesort of automatic lens correction routine in the camera software?

All the tests in this thread have taken place with the Nikon Z6 mounted on a telescope. So if some kind of correction is taking place (e.g. for vignetting or for channel crosstalk) then it is hardcoded in the firmware.  I'm not convinced it would cause these eccentric (i.e. off centre and non-circular)anomalies in any case.

 

Mark

Edited by sharkmelley, 14 September 2019 - 05:09 PM.

[sharkmelley]

I have an update on the problem of the concentric rings. After much experimentation, the tentative conclusions I have reached are these:

• The rings are caused by a scaling applied to the raw digital data of the red and the blue channels (but not the green)
• The scaling is some kind of fixed functional form that is applied to the red and blue channels even when vignetting corrections are switched off. It occurs with and without a recognised lens attached.  It occurs even when directly illuminating he sensor with diffuse light, with no optics attached
• The main effect in the blue channel is to dim the corners (or brighten the centre) by a factor of around 1.03
• The main effect in the red channel is to brighten the corners (or dim the centre) by a factor of around 1.01
• For the avoidance of any doubt this scaling is in addition to the red and blue white pre-conditioning of approx 1.18 which causes the well-known histogram gaps
• My conjecture is that the purpose of the correction is to fix colour cast issues towards the corners of the sensor.

The problems associated with applying any kind of scaling to digital data is well known.  The data can only be shifted by integer increments and this causes "steps" to form in the resulting data values.  It is these steps in the data that lead to the rings we are seeing.

 

Having understood the cause, it was quite easy to reproduce the effect in a "real world" photo.

 

Here's a photo taken 4 stops underexposed at ISO 100, opened in Photoshop and pushed 4 stops in post-processing:

 

 

 

This is what happens if I stretch the image by crazy amounts in Photoshop:

 

 

 

You can see the rings in the sky.  If I open the raw data file in PixInsight and divide the (bias subtracted) blue channel by the (bias subtracted) green channel then stretch, this is the result:

 

 

 

I don't think you will ever notice this effect in daytime photography without extreme stretching and/or haze removal because the step size causing the rings is exactly one digital unit and this is normally masked by image detail and/or image noise.  The problem for astrophotographers is that we stack together lots of images which reduces the image noise down to a level where the steps can become visible in featureless areas of background after applying flats and subtracting light pollution.

 

For daytime photographers, it is clearly a benefit that the camera will fix corner colour casts.  But this is another example of where the needs of daytime photographers and astrophotographers can conflict.

 

Mark

Edited by sharkmelley, 21 September 2019 - 02:13 AM.

[SandyHouTex]

Nice job Mark figuring this out.

[AgilityGuy]

Mark,

 

I applaud your tenacity in figuring this out.  What is your overall opinion of the camera for astrophotography after this exercise?  Earlier in the thread, another owner of the camera expressed some concern about the artifacts it produces but it was also stated in a subsequent post that the camera should be fine for overall use because you have only been seeing the artifact when you push the camera to the limits.  I honestly can say that I've never set my equipment up expecting middle-of-the-road results.  If I'm going to spend many hours acquiring and processing images I wouldn't want a camera that limits my efforts.  I realize this is quite subjective and opinion but I've battled this problem with another camera and my solution was to leave it in the bag for daytime use.  My full spectrum Nikon D800e has not presented this problem, at least in my hands.  Even imaging from my yard in an orange-red zone I'm reasonably happy with the images it produces. 

 

Joe

[sharkmelley]

Mark,

 

I applaud your tenacity in figuring this out.  What is your overall opinion of the camera for astrophotography after this exercise?  Earlier in the thread, another owner of the camera expressed some concern about the artifacts it produces but it was also stated in a subsequent post that the camera should be fine for overall use because you have only been seeing the artifact when you push the camera to the limits.  I honestly can say that I've never set my equipment up expecting middle-of-the-road results.  If I'm going to spend many hours acquiring and processing images I wouldn't want a camera that limits my efforts.  I realize this is quite subjective and opinion but I've battled this problem with another camera and my solution was to leave it in the bag for daytime use.  My full spectrum Nikon D800e has not presented this problem, at least in my hands.  Even imaging from my yard in an orange-red zone I'm reasonably happy with the images it produces. 

 

Joe

Hi Joe,

 

I don't yet have a final answer because there are still unanswered questions in my mind which I'm unlikely to answer until I start doing some serious astrophotography with the benefit of the lessons I have learnt so far.  Unfortunately I already have a whole load of wasted data (containing coloured banding)  that I will never to able to process to my satisfaction.  However, once I started shooting at ISO 3200 with the back-of-camera histogram peak at the halfway position then the data was much easier to process because it didn't have the coloured bands.  But why did I still have that residual gradient problem in the flats - green in the middle and purple at the edges? 

 

Although all these problems can be worked around, there is certainly a good argument for choosing a camera which doesn't have these problems in the first place.

 

Mark

[erictheastrojunkie]

This discussion on the flats problem has been enlightening for me, I was working on a big mosaic of the Milky Way core region with my Z7 and a Voigtlander 65mm, but got super dismayed when I was utterly unable to correct my framea with a master flat. I never tried matching the exposure with a master flat or overexposing the flat big time, I'll try that. Basically I could not get properly calibrated light frames, corners were always under corrected or over corrected no matter what I tried.

Sent from my Pixel 2 XL using Tapatalk

[sharkmelley]

When applying such a huge stretch to data then tiny imperfections in the flat, especially at the edges, become amplified and can lead to colour bias once the light pollution background is subtracted.  The relative difference between the flat that caused the colour bias and the flat that didn't was less than 0.5% at the edges.  This difference is certainly the cause of the colour bias at the edges but what is the cause of that difference in the flats?

 

One possibility is that there is some flaw in my acquisition of flats.  Another possibility is that my acquisition of flats was fine but there is some slight non-linearity in the sensor's gain curve.  If a slight non-linearity exists then it makes perfect sense to match the background colour and intensity of the flats to the background colour and intensity of the lights.

 

Mark

Edited by sharkmelley, 25 September 2019 - 12:28 AM.

[UKalwayscloudy]

This is an immensely useful discussion. I wonder if I could ask contributors to offer their opinions so far on whether to bother modding for Halpha. I was rather influenced by Roger Clark’s web site in thinking that once the transmission got above about 20% you could reasonably work with boosting in post and not do the mod. I got a few shots of Orion Nebula from Celestron C6 at the beginning of the year and was pleasantly surprised by the amount of red in there with a bit of processing. I’ll be trying this again in a Short while. A specialist UK converter has already told me not to bother as I have a Nikon! Views welcome.

[whwang]

I think only when you have a Fuji, you don't bother with the mod.  For everything else, a filter conversion can significantly boost the H-alpha sensitivity.  Note that I am not against not modifying.  I am just saying that modification can give you much more.

[Jerry Lodriguss]

This is an immensely useful discussion. I wonder if I could ask contributors to offer their opinions so far on whether to bother modding for Halpha. I was rather influenced by Roger Clark’s web site in thinking that once the transmission got above about 20% you could reasonably work with boosting in post and not do the mod. I got a few shots of Orion Nebula from Celestron C6 at the beginning of the year and was pleasantly surprised by the amount of red in there with a bit of processing. I’ll be trying this again in a Short while. A specialist UK converter has already told me not to bother as I have a Nikon! Views welcome.

A modded camera will record 4x more Ha light in the same exposure time.

 

If you are really interested in Ha nebulae, you can get a better picture in less time with a modded camera.

 

The Orion nebulae is the brightest in the sky, so it's easy to take pictures of it, even with an unmodded camera. But when you try the fainter red emission nebulae, then you would need 4 hours with an unmodded camera to get the same picture as 1 hour with an modded camera.

 

It's all about how much time you are willing to invest.  When you are first starting out, try unmodded. Then if you want to shoot red faint stuff, get the mod.

 

Jerry

[Philipp]

Hi

 

I ran into the concentric colored banding issue, shooting at ISO 400, F/2.2 and 30s.

 

I read above that pushing the diagram to the middle and using ISO 3200 will help getting rid of the concentric banding issue. Is ISO 3200 significant? Or would it also help to just expose longer?

 

 

thanks for the very interesting thread!

 

Philipp

[Philipp]

Mark answered my questions on dpreview:

 

• Use 14 bit uncompressed or lossless compressed format (you're probably doing that already)
• Switch off any vignette control
• Choose an ISO and exposure length that puts the back-of-camera histogram peak in the middle, for both the light frames and the flat frames

 

thank you!

[Jerry Lodriguss]

Mark answered my questions on dpreview:

• Use 14 bit uncompressed or lossless compressed format (you're probably doing that already)
• Switch off any vignette control
• Choose an ISO and exposure length that puts the back-of-camera histogram peak in the middle, for both the light frames and the flat frames

Mark,  Vignetting control isn't applied to the Raws I'm pretty sure.

 

Jerry

[sharkmelley]

Mark,  Vignetting control isn't applied to the Raws I'm pretty sure.

 

Jerry

You're probably right but I said it to be extra sure because vignetting control on Sony cameras (shading) definitely affects the raws.

 

Mark

Edited by sharkmelley, 23 December 2019 - 03:46 AM.