25 replies to this topic

[illogicist]

Hello there!

I've a problem shooting with a Fujifilm X-T4 camera. When I process the files using Pixinsight or Siril, the master file is afflicted by some diffraction cirles. The image I attach below is the stacking of 100x30" lights at 1250 ISO with a 50-140 Fuji lens, 30 darks, 20 flats, 50 bias. I've tried to expose flats in the right half of the histogram, as read in the Nikon topic, but I don't see great improvement.

The problem becomes evident whith the background extraction.

 

Anyone have the same problem with this camera?

 

Here the master stacked with Pixinsight:

 

Here some lights and flats from my camera:

https://www.dropbox....JS9PqtOxEa?dl=0

 

Thanks a lot!

 

Luca

 

[sharkmelley]

Thanks for uploaded files.  They allowed me to quickly confirm that your rings are caused by an issue I've only recently identified but is common to many Fujifilm cameras.

 

The problem can be seen in the RawDigger (bias-subtracted) histogram of one of your lights:

 

 

The histogram is not a smooth curve but has definite discontinuities (i.e. steps) at every multiple of 64.

 

This is likely caused by some noise reduction algorithm.  However, the unintended consequence of this algorithm causes rings to appear in underexposed data.

 

The same problem was also diagnosed in this thread on the Fujifilm GFX100:

https://www.cloudyni...ction-circles/ 

 

Note that the problem does not affect your flats because the histogram is much further to the right.  The only solution that I know of is to increase your ISO or exposure length to push the histogram of the lights much further to the right and away from the "dangerous region" where the histogram discontinuities occur.

 

Mark

[vidrazor]

The only solution that I know of is to increase your ISO or exposure length to push the histogram of the lights much further to the right and away from the "dangerous region" where the histogram discontinuities occur.

Doesn't that risk core detail? Would raising the ISO and dropping exposure help?
 

[fmendes]

How did you take the flats?

[sharkmelley]

Doesn't that risk core detail? Would raising the ISO and dropping exposure help?
 

Unfortunately, it won't help.  The histogram needs to shifted away from the low values where the histogram discontinuities occur.

 

 

How did you take the flats?

The flats aren't the problem here.  The problem is in the lights.

[illogicist]

Thanks for uploaded files.  They allowed me to quickly confirm that your rings are caused by an issue I've only recently identified but is common to many Fujifilm cameras.

 

The problem can be seen in the RawDigger (bias-subtracted) histogram of one of your lights:

 

FujiXT4_histogram.jpg

 

The histogram is not a smooth curve but has definite discontinuities (i.e. steps) at every multiple of 64.

 

This is likely caused by some noise reduction algorithm.  However, the unintended consequence of this algorithm causes rings to appear in underexposed data.

 

The same problem was also diagnosed in this thread on the Fujifilm GFX100:

https://www.cloudyni...ction-circles/ 

 

Note that the problem does not affect your flats because the histogram is much further to the right.  The only solution that I know of is to increase your ISO or exposure length to push the histogram of the lights much further to the right and away from the "dangerous region" where the histogram discontinuities occur.

 

Mark

Hello Mark, thanks a lot for your analys. I ‘ve tried to expose the flats on the right half of the histogram as read in the Nikon thread. In the next days I will make a new test exposing the lights further to the right, as you suggest me. I will upload the result here. 

 

In the menu of th X-T4 there is an option about “long exposure noise reduction”: it was off on the lights i’ve uploaded. I will try new light with this option ON. 

I’ve also a doubt about mechanical/electronic shutter: can the mechanical shutter introduce some noise?

 

Thanks for your help.

[illogicist]

How did you take the flats?

I’ve taken them after the lights, outdoor, using a notebook screen in front of the lens. 
Same ISO of the lights (1250), same aperture (f2.8), i’ve adjusted the shutter time to expose on the further

right of the histogram, because i’ve read it could have been better with this problem.

I’ve also taken flats exposing at 45% on the left, but I’ve not seen big difference when processing.

[sharkmelley]

In the menu of th X-T4 there is an option about “long exposure noise reduction”: it was off on the lights i’ve uploaded. I will try new light with this option ON. 

I’ve also a doubt about mechanical/electronic shutter: can the mechanical shutter introduce some noise?

These things are worth a try but I'm not optimistic.  Generally speaking the LENR (long exposure noise reduction) doubles the length of time required to shoot every exposure because it internally takes a dark frame to subtract, to remove the thermal FPN (fixed pattern noise)

[whwang]

So sad to see this. We should complain to Fuji.

[vidrazor]

So sad to see this. We should complain to Fuji.

Although Fuji may be responsive, they, like Canon, Nikon,etc. are in the business of terrestrial imaging, so this isn't really an issue in their primary markets.

[fmendes]

According to photonstophotos, ideal ISO for the X-T4 would be 503:

https://www.photonst...arts/RN_ADU.htm

https://www.photonst.../Charts/PDR.htm

 

Still you should get decent results at ISO 1250. Did you notice any condensation on the lens? Anyway, better use ISO 500 and no in-camera noise reduction.

 

From my experience, flats taken with a screen are problematic, especially in shutter speeds close to 1/60. Better if you use the white shirt against the day sky. Even better using a flat panel, such as a good tracing tablet from Amazon.

 

Try to stack without flats and see if the problem persists.

[illogicist]

According to photonstophotos, ideal ISO for the X-T4 would be 503:
https://www.photonst...arts/RN_ADU.htm
https://www.photonst.../Charts/PDR.htm

I will try if could be better, but if the problem becomes visible on the underexposed parts, to reduce ISO won’t be worse?

Still you should get decent results at ISO 1250. Did you notice any condensation on the lens? Anyway, better use ISO 500 and no in-camera noise reduction.

No condensation on the lens; I’ve tried different nights with different weather conditions and the problem is still present, unluckly.

From my experience, flats taken with a screen are problematic, especially in shutter speeds close to 1/60. Better if you use the white shirt against the day sky. Even better using a flat panel, such as a good tracing tablet from Amazon.

Try to stack without flats and see if the problem persists.

I’ve taken the flats also using a t-shirt but I think th problem is not visible on flats. I’ve tried to stack the lights only and rings are present. They are present also in a single light, if I stretch the histogram and extract the background.

[sharkmelley]

I'll present here some further analysis of one of the lights.  Remember I said earlier that each histogram step (at multiples of 64 in the bias-subtracted data) correlated with a coloured ring in the image.  But I stated that without any evidence.

 

Here I've calibrated one of the lights with a flat and then increased the contrast of the rings to make them more obvious:

 

 

 

For the top row of the montage below I've separated out the red/green/blue channels so you can see their rings:

 

 

Immediately after debayering the image (it's an XTrans pattern) I ran a Pixelmath function to indicate the pixels whose values are multiples of 64.  This forms the bottom row of the plot above, again separated out into the red/green/blue channels.

 

Note the very strong correlation between the rings in the top and bottom rows, including the displacement to the left caused by the background gradient.  For me this is strong evidence that the histogram steps at multiples of 64 are related to the coloured rings.  They are the result of some internal processing by the camera firmware.

 

Mark

Edited by sharkmelley, 21 February 2023 - 12:55 PM.

[fmendes]

I'll present here some further analysis of one of the lights. 

That is pretty solid analysis. I used to use a camera with X-Trans but never saw that ringing, that's why I was looking into other possible explanations. However my camera was a X-E2. I also have a X-H1 with a newer sensor, but used to prefer the image of the older sensor. Maybe they messed up something with the newer cameras.

[bclaff]

I rarely disagree with Mark but I think this is lens corrections and definitely not any form of noise reduction.

My raw analysis of the flats doesn't show any "64 pattern".

When converted to DNG I do see

 

OpcodeList2                  
Opcode FixVignetteRadial
Coefficients 0.503645,2.882931,-11.019517,15.257146,-6.644967
Center: 0.5,0.5
OpcodeList3                  
Opcode WarpRectilinear
Planes: 3
Plane 0:
Radial params 0.976938,-0.000977,0.055836,-0.031878:
Tangential params 0.0,0.0:
Plane 1:
Radial params 0.977174,-0.000818,0.055431,-0.031731:
Tangential params 0.0,0.0:
Plane 2:
Radial params 0.976961,-0.000412,0.054488,-0.03133:
Tangential params 0.0,0.0:
Optical center: 0.5,0.5

 

These radial operations could definitely set up a radial beat pattern.

 

I don't have an X-T4 handy. I would check  to see if Lens Modulation Optimizer or some related setting changes the result.

 

If you use a DNG workflow you could use Exiftool to remove the lens correction opcodes.

[sharkmelley]

Hi Bill,

 

The reason I think this is noise reduction is that the characteristics of the histogram (discontinuities/steps at regular intervals - in this case 64) are identical to non-XTrans FujiFilm cameras that definitely do apply heavy noise reduction.

 

For instance, here's one of my usual plots for a dark frame on the GFX50R:

 

Regarding lens corrections, for astro image processing we generally use software based on the same Libraw as RawDigger, which totally ignores any lens correction opcodes in the EXIF.  I can guarantee that those rings are embedded in the raw data and are not applied during raw conversion.

 

Mark

Edited by sharkmelley, 22 February 2023 - 12:31 PM.

[illogicist]

I don't have an X-T4 handy. I would check  to see if Lens Modulation Optimizer or some related setting changes the result.

I've checked the menu and Lens Modulation Optimizer is set to ON

[illogicist]

Hi!

 

I'm trying to make new tests and in an external hd I've found files from january 2022 while I was shooting at horse head.

It seems the noise different, but I used the same configuration.

 

Here some lights and darks: https://www.dropbox....MSnZ-PdLXa?dl=0

 

Luca

[sharkmelley]

I've checked the menu and Lens Modulation Optimizer is set to ON

Yes the Lens Modulation Optimizer is set to ON in your files that I tested and also in the Horsehead files that you have just unloaded.

 

However, here is a link to the post in the other FujiFilm thread where a raw GFX100S file is available that has similar histogram steps and similar rings resulting from whatever induced those histogram steps:

https://www.cloudyni...les/?p=10394501

 

That file has Lens Modulation Optimizer set to OFF

 

I also have some raws from another GFX100S camera, again showing histogram steps and associated rings where again the Lens Modulation Optimizer set to OFF.

 

Mark

Edited by sharkmelley, 22 February 2023 - 05:33 PM.

[whwang]

I will celebrate by buying everyone here a beer (or two) if it's really just a lens modulation optimizer problem.  But it doesn't seem to be.  I had such rings even when my camera is attached to a telescope and no lenses were detected.  The ring behavior changes with telescope.  So even if you say it's some lens correction that cannot be turned off in the menu, I am not convinced.

[sharkmelley]

I'm trying to make new tests and in an external hd I've found files from january 2022 while I was shooting at horse head.

It seems the noise different, but I used the same configuration.

 

Here some lights and darks: https://www.dropbox....MSnZ-PdLXa?dl=0

Very interesting.  Here is the (bias-subtracted) histogram from one of your Horsehead lights:

 

 

Notice there are no steps in the histogram at multiples of 64, apart from the weird spike in the blue channel at value 32.  There are also no rings in your lights.

 

The image is incredibly underexposed.  But this underexposure might be an alternative workaround to your rings problem.

 

Mark

Edited by sharkmelley, 23 February 2023 - 01:52 AM.

[whwang]

Hi Mark,

 

In this example, the histograms did go across 64.  The green one actually went across both 64 and 128.  And no steps are seen.  So the steps would only appear in moderately low exposure area, not extremely low nor high exposure areas?

[sharkmelley]

In this example, the histograms did go across 64.  The green one actually went across both 64 and 128.  And no steps are seen.  So the steps would only appear in moderately low exposure area, not extremely low nor high exposure areas?

It's certainly an interesting finding.  To gain a better understanding of what's going on, it would be useful to look at a sequence of shots (e.g. flats) with varying levels of recorded exposure.

 

On your GFX100S camera, the (bias-subtracted) histogram steps occurred at multiples of 256 instead of 64, maybe because the GFX100S is a 16bit camera.

 

Mark

Edited by sharkmelley, 23 February 2023 - 03:37 AM.

[whwang]

That's an interesting point.  Seems like 64 is the magic number in those people's head.

 

I plan to find time in the coming months to conduct more tests on my 100S, to see exactly what are the triggering criteria for the rings.  I will do some indoor flat tests and bring the camera/lens under real sky.  I will test if exposure dithering can reduce the problem.

[ader0101]

Luca, I think "Lens Modulation Optimizer" is for jpgs only, I am sure you are shooting RAW. I have an Fujifilm X-T4, X-Pro2, X-M1 and have not had these rings. Could an interference pattern be introduced somewhere in your workflow? I have used multiple lenses (Nikon, Fuji, Samyang), various ISO, exposure lengths, DSS, Siril, Starry Star Stacker, Lightroom and Photoshop and have not seen this circular pattern or any pattern. Try changing things up perhaps? Try another RAW converter before stacking? Some of the discussion here is beyond my knowledge and I can't comment to some of it, I have only been doing DSO for a year so my knowledge is limited to that.

 

Allan