582 replies to this topic

[Borodog]

Is the need for this color correction matrix solely a function of the typical channel sensitivity curves of DSLR/mirrorless cameras? And hence not necessary for OSC astro cameras with their expanded sensitivity?

[BlueMoon]

Is the need for this color correction matrix solely a function of the typical channel sensitivity curves of DSLR/mirrorless cameras? And hence not necessary for OSC astro cameras with their expanded sensitivity?

Right now, the question seems to be with images from unmodded DSLRs. I haven't heard anyone mention OSC cams or shown any processed imaging from one. Would be interesting to know however.

[vidrazor]

Not at all—thanks for giving the data a go! This last one is what I would expect—you even got some strong Ha from the whole Statue of Liberty Nebula region.

BTW, I'll add that I had to do manual color correction on these in Siril, I could not get photometric color calibration to work. At first I did not know what to even punch in, as an astrometry.net analysis did not identify any targets, only constellation outlines, so I used them and searched in Stellarium and found NGC 3372. I punched that in, but photometric color calibration still failed. So I performed manual color correction instead.

[BQ Octantis]

BTW, I'll add that I had to do manual color correction on these in Siril, I could not get photometric color calibration to work. At first I did not know what to even punch in, as an astrometry.net analysis did not identify any targets, only constellation outlines, so I used them and searched in Stellarium and found NGC 3372. I punched that in, but photometric color calibration still failed. So I performed manual color correction instead.

Use HIP 54301. Works like a champ.

[BQ Octantis]

That would be nice to see. I'm in the "one program to rule them all" mode these days for image stacking/processing. I could learn RT but as it's already part of Siril, as you say perhaps working the CCM option in would be doable.

I can actually simulate the green-heavy output in Siril if I do two things in the RawTherapee Color pane:

 

1. Turn off White balance. This allows the green to dominate by 200%.
2. Set Color management to No profile. This puts the red and blue curves exactly on top of each other.

 

If I then do a photometric color correction in Siril, I end up with the same result as the all-Siril workflow: a Carina Nebula devoid of any red:

 

 

I'm now wondering what input ICC profile RawTherapee is selecting to apply when I set it to automatic. If I look in the input ICC profile folder (Finder: Select RawTherapee → Show Package Contents → ./Contents/Resources/iccprofiles/input/ ), it's only got Canon profiles up to the 550D. Regardless of how, it seems to boost the reds to a perceptible level.

 

BQ

Edited by BQ Octantis, 18 September 2022 - 12:06 PM.

[Alen K]

I can easily answer that because I investigated it a while back.  Sequator is applying the proper colour space conversions, so essentially it behaves more like RawTherapee and LightRoom.  Under the hood, Sequator is using LibRaw to perform the demosaicing and also LibRaw's CCM (colour correction matrix) for the colour space transformations. I believe the CCM ultimately comes from Adobe, just like RawTherapee's CCM.

Great. I have always liked the output from Sequator. It's fast at stacking too, much faster than DSS, whose output I always felt was lacking (particularly color saturation). I don't know how  Sequator compares with stacking in SiriL, either for speed or quality, since I have never tried stacking in the latter. I also don't know how the debayering in Sequator compares to that in RawTherapee or SiriL or other programs. Unlike RawTherapee, Sequator doesn't allow a choice of debayering algorithm. 

[sharkmelley]

 

I'm now wondering what input ICC profile RawTherapee is selecting to apply when I set it to automatic. If I look in the input ICC profile folder (Finder: Select RawTherapee → Show Package Contents → ./Contents/Resources/iccprofiles/input/ ), it's only got Canon profiles up to the 550D. Regardless of how, it seems to boost the reds to a perceptible level.

The RawTherapee documentation details where it picks up the camera's white balance and CCM (colour correction matrix).  See the section "Storing and Reading This Information" where it says that the information is picked up from three possible places:

• In the dcraw code which is embedded inside RawTherapee,
• In the raw file itself,
• In a text file on your system which is installed with RawTherapee called camconst.json

Mark

[BQ Octantis]

The RawTherapee documentation details where it picks up the camera's white balance and CCM (colour correction matrix).  See the section "Storing and Reading This Information" where it says that the information is picked up from three possible places:

• In the dcraw code which is embedded inside RawTherapee,
• In the raw file itself,
• In a text file on your system which is installed with RawTherapee called camconst.json

Mark

There's also a page I'm working through now on how and where to get custom profiles and what settings to use:

 

https://rawpedia.raw....2C_if_possible

 

There's also a page that describes how to get ACR DCPs by installing Adobe DNG Converter:

 

https://rawpedia.raw...nd_DCP_profiles

 

This last page omits Mac OS; once installed, the DCPs are in /Library/Application Support/Adobe/Camera Raw/Camera Profiles/Adobe Standard/

 

BQ

[BQ Octantis]

Ok, got the 600D DCP installed and tested.

 

Per the Rawpedia entry on DCPs, there are a few options to consider:

 

• Illuminant: Can include D50 only or also StdA. The 600D ACP had both, so I left it at Interpolated.
• Lookup Table: Should be on
• Tone Curve: Can be off or on; on will boost saturation while desaturating highlights

 

I tested with the Lookup Table and Tone Curve either off or on and compared to my automatched workflow. Here's how they compare in Siril Autostretch (no other manipulation):

 

 

In all the DCP cases, the OIII around the Keyhole was a lot more bluish than the automatched. There was a clear reduction in red in the background with the Lookup Table and Tone Curve both on. With both on, the result was the most similar to the automatched.

 

Now for a stack…

 

[galacticinsomnia]

I think the point being missed may be that Siril is doing what it is designed to do.  Just like shooting raw images in the first place a flat color space in order to be color graded as needed.

Siril is a Stacking Software, and Pre-Processor.  While you can make final outputs straight from Siril, it is doing so from a flat or raw color space.  Just like Mark is talking about.  It is apples and oranges.
It's like taking images of roses and editing them with Pixinsight, and not undrestanding why it looks horrible. 

A 40 minute stack from an unmodified camera that can get more color from one software package verses Siril, seems to be a don't care.  I think being critical of software being used is completely fair, and have noted some "automatic" processes in Siril that I haven't full explored can show some data loss, but that is user error, like overstretching and losing your color values.

I posted two stacks as requested and have not heard anything about the comparison of the two stacks.
I don't see any issue with Siril, losing or eating color, not saying it doesn't happen, but I don't see it.

Clear Skies !!

 

[BQ Octantis]

I posted two stacks as requested and have not heard anything about the comparison of the two stacks.

Sorry, mate, lost your post to the scroll.

 

I can get a lot more color out of the DSS stack.

 

 

But the histogram is way on the upper end (the black point is at ~40%).

 

So yet another data point…

 

BQ

 

[cissou8]

Hi again,

 

Thanks for the RT calibrated images, had a look and they did shed some light, pun intended, on the matter.

Also had a discussion with Cyril (lock042) as he has been involved a lot more into all of this. A lot of it goes down to the response from Mark. I'll try to summarize below.

When software like RT or Sequator open a raw image, they do a lot of stuff to the data to apply the color profile as set by the camera manufacturer, or any other of your liking, for pictures that would be taken under well-lit conditions. There is WB, CCM but even more importantly (to our discussion) there is this gamma correction, what I was referring to as "stretching" in my previous post.

After this gamma step, the data is complelety nonlinear. So basically the stars do not have their "gaussian" shape anymore. It does not interfere too much with detection, although  it is making it more difficult as some of the criteria to identify that stars are stars are linked to their shape not departing too much from a gaussian. But this non-linear stretch makes color calibration by photometry nothing more than wishful thinking. There is also some clipping going on, both black and white points, but that's probably because the profiles where not made for astrophotography kind of histograms in the first place. So just to be clear, adding gamma correction at any point before stacking is not a route we wish to take.

 

So basically there are two ways to handle that:

- either you like it more when the pics are calibrated with RT, giving up on the "one-software" approach and you keep your workflow as is.

- or you adapt a bit the workflow in Siril when you want to pull out more red (that's always at the expense of noise). For instance, for this particular kind of nebs where you want to see more "Ha", you could for instance uncheck the equalize cfa box in preprocessing. Provided your flat source is "white" enough, this would somehow apply some sort of white balance to the image. They come out redish by the way because now this layer is over-but-linearly-stretched. You can go ahead and register/stack. And then on your stack, you would just neutralize the background and nothing more. Do not try to calibrate with photometry, it would get back the now-exxagerated spread of the red channel where it belongs, which is nowhere because most of it was filtered out by the unmodded DSLR. You could also achieve the same by splitting the R G B layers and recompose with Pixel Math with larger coefficients on the red (and always some kind of neutralization).

Don't get me wrong, this is no different than the tweaking I was doing with the starless layer in my initial post. Except it is a bit more automated and at a linear step so you wouldn't have to pull that much on the histo during the de-linearized phase.

Just to illustrate, here-below a single frame calibrated as mentionned with bkg neutralization in Autostretch view (you can get even more red with saturation).

 

Just a quick word about color calib with photometry (and plate solving in general). On particularly wide fields, you can select an area centered around an object you know (NGC 3372) and it would restrain the solve to that area. This is how I got it to work on that one.

 

Clear skies,

 

Cecile (which is a girl's name in France btw )

Edited by cissou8, 18 September 2022 - 04:11 PM.

[BlueMoon]

Ah, okay. Thank you all for the clarifications and it answers a number of questions for me. Admittedly, I don't have the level of experience some of you have but the "why and how" of the processing chain in Siril makes more sense to me now. For the time being, I'll stick with my unmodded T3i and SL3 and see just how far I can take Siril in my work. If need be I'll look into RT to augment the processing.

 

Clear skies.

Edited by BlueMoon, 18 September 2022 - 04:21 PM.

[xploeris]

It's fine that Siril produces a a "raw" stacked image, with linear gamma and no color correction of any kind. This is obviously a good starting point for any kind of processing, with any tool. But I don't understand why Siril's set of tools for processing the stacked image doesn't include a simple way to apply color correction similar to RT and other photo processing apps. Producing a "natural" photo (i.e. one with stretched luminance and exaggerated saturation, but hue similar to what might be seen if the subject could be clearly viewed with the naked eye) is a very common use case. If there are tools like photometric calibration, gradient reduction, etc. that should only be applied to uncorrected images, it shouldn't be too hard to expose that fact in the interface and/or the manual.

Possibly there's a way to do this with pixelmath, I don't know. I still don't understand how to use the pixelmath tool.

There could also be a "color correction" checkbox which will apply a color correction transformation to the displayed image but not to the underlying data, so that you can view how your photo will look after correction. This would be similar to the existing control for displaying your data with autostretch, histogram equalization, etc.

This forum isn't really the right place for feature requests, but it seems like the problem here is a missing feature.

Edited by xploeris, 18 September 2022 - 04:20 PM.

[lock042]

 

But I don't understand why Siril's set of tools for processing the stacked image doesn't include a simple way to apply color correction similar to RT and other photo processing apps.

Because this is absolutely not the goal of Siril.

 

 

This forum isn't really the right place for feature requests, but it seems like the problem here is a missing feature.

A lot of features are missing yes, but probably not that. You want something that apply the gamma curve and the DSLR color factor. But this is really not the goal of Siril to do that.

Also, true color, natural color mean nothing in astrophotography.

 

In fact, the real problem here is trying to capture Ha photons with a filter that prevents it. It's like trying to catch fish through glass.

[FrankieT]

When software like RT or Sequator open a raw image, they do a lot of stuff to the data to apply the color profile as set by the camera manufacturer, or any other of your liking, for pictures that would be taken under well-lit conditions. There is WB, CCM but even more importantly (to our discussion) there is this gamma correction, what I was referring to as "stretching" in my previous post.

After this gamma step, the data is complelety nonlinear. So basically the stars do not have their "gaussian" shape anymore.

 

 

RT at least (not familiar with Sequator) does not apply a "gamma" correction on file import unless configured to do so by the user. Rawpedia says that some of the auto-matched camera profiles may contain a (non-linear) tone curve so it's safer to select "camera standard" which only has a CCM. RT also uses a linear RGB colour space as the default working profile so it's possible to apply both WB and a CCM and still maintain linear data for stacking later (in Siril?) provided the selected output colour profile also has a linear tone response curve.

 

Clear skies!

[BQ Octantis]

Thank you Cecile and Cyril! Fascinating inputs! And a lot of rabbit holes to go down. Frankie, I'll do a run with the Camera standard CCM and no DCP to see what that does. Not sure how to prevent the slight highlights clipping, though. I've used highlights propagation to improve star color, but that's a whole other workflow.

 

The application of non-linear coefficients does indeed result in different star shapes and—more significantly IMO—some purple haloing quite likely from CA. But plucking the stars from Siril and plugging them into the otherwise colorful image is a fairly trivial step with a star mask—giving the best of both worlds:

 

Preview (click for 100% scale):

Click for before/after toggle @ 100%

 

More to ponder and test…

 

BQ

Edited by BQ Octantis, 18 September 2022 - 06:43 PM.

[FrankieT]

There could also be a "color correction" checkbox which will apply a color correction transformation to the displayed image but not to the underlying data, so that you can view how your photo will look after correction. This would be similar to the existing control for displaying your data with autostretch, histogram equalization, etc.

I'm not sure this would help - AFAIK, Siril does not work with color profiles and so it does not apply the color profile of the monitor to the displayed image. In other words, the image in Siril's preview window is probably not rendered properly anyway. This may have changed in recent versions so perhaps lock042 or cissou8 will correct me.

 

Clear skies.

[xploeris]

Because this is absolutely not the goal of Siril.

 

A lot of features are missing yes, but probably not that. You want something that apply the gamma curve and the DSLR color factor. But this is really not the goal of Siril to do that.

Also, true color, natural color mean nothing in astrophotography.

 

In fact, the real problem here is trying to capture Ha photons with a filter that prevents it. It's like trying to catch fish through glass.

As it stands Siril combines preprocessing and stacking with a substantial set of postprocessing tools. It appears to be a more or less comprehensive astrophotography solution, or to be heading in that direction. The idea that you can use it to calibrate color, arbitrarily alter saturation or color balance, remove LP and other gradients, apply all sorts of nonlinear stretches, but saying that applying camera-specific corrections to change the colors to something resembling a normal visible light photograph is "absolutely not the goal" seems incoherent.

Natural color does mean something in astrophotography. Most targets of interest to astrophotographers give off visible light. Visible light has color - real, actual color, not simulated color. I'm sure there are folks here shooting outside the visible spectrum but not very many; for instance, most folks recommend against full spectrum mods as Ha mods that retain IR filtering are considered superior, and UV is just a pain to shoot generally. And while there's some room for artistic interpretation or personal preference in processing, visible AP targets are expected to generally look a certain way due to the visible light they give off. If I were to post up, for instance, a green M42, or a purple M31, I'm sure the discrepancy would get noticed!

It sounds to me like you're making nonsense arguments for the sake of being contrary. But maybe I just don't understand your perspective.

Edited by xploeris, 18 September 2022 - 07:30 PM.

[BlueMoon]

 

"absolutely not the goal"

You are aware, or perhaps not, that both lock042 and cissou8 are Siril devs? I would think if anyone knows "what the goal of Siril" is they would.

 

 

But maybe I just don't understand your perspective

Apparently.

[xploeris]

You are aware, or perhaps not, that both lock042 and cissou8 are Siril devs?

I am aware, yes. Did they ask you to speak for them?

[Alen K]

Disappointing to hear a Siril developer dismiss the importance of the CCM for getting correct color out of an unmodified DSLR. I guess we'll never get that option in Siril since that is not one of the goals of Siril. Too bad, because it wouldn't be hard for them to add it and, as an option, would only be used by people who want to use it. 

 

Edited by Alen K, 18 September 2022 - 08:12 PM.

[BlueMoon]

 

Did they ask you to speak for them?

No, they did not. However, I invited them to participate in this thread to help clear up some questions. So, what is your point?

[xploeris]

No, they did not.

 

Then don't. Thanks.

[BQ Octantis]

See? Now this is why we can't have nice things.

 

For my part, I'm excited to be bridging the gap between terrestrial and astro photography. I wouldn't have thought that in late A.D. 2022 one of my posts would be pressing the envelope. But here we are.

 

If we're making demands to developers, my needs are far more difficult to meet. I want the best of all worlds. RawTherapee's adjustable luminosity masking is the tip of the envelope of what I need. While I can happily tweak its masks on luminosity noise reduction, color noise reduction, and saturation, I would like the ability to scale demosacing algorithm and parameters with scene complexity and brightness. Comparing the RawTherapee and Siril demosaicing results, I need star masks that scale diameter with star brightness to maintain color and control CA while bringing up the color in the nebula. Apply Siril to the stars and RawTherapee CCM to the nebula. All with the AI accuracy of Starnet++—but I want quality far better than Starnett++. My goal is always quality that allows me to scale to 150-200% sensor pitch.

 

Other than that, I've got what I need.

 

Cheers,

 

BQ

Edited by BQ Octantis, 18 September 2022 - 08:51 PM.