436 replies to this topic

[timaras]

No, Siril appends the sRGB ICC profile to the metadata of the linear image but doesn't actually apply the sRGB TRC, so the data remains linear. Lumariver interprets the (linear) data as non-linear and reverts the imaginary sRGB TRC, which finally results in non-linear data. 

That makes sense.

 

Can you point me to a reference? If you are referring to the Tone Curve tab, these settings are not meant to set the TRC of the target file.

See the section of Lumariver manual about "Input curve selection":

 

The target transfer function is derived from information embedded in the target image itself and describes how to convert the pixel values to linear.

 

I also had an email exchange with the developers a few weeks ago asking whether Lumariver expects linear data, and confirmed that it's not necessary.

Edited by timaras, 08 June 2025 - 10:33 AM.

[FrankieT]

That compares favorably to your result.  I think this addresses the issue, and after all, Lumariver can be used for our intended purposes with those settings.

Martz, timaras

 

Just wanted to clarify that the defect we discovered in the ICC profiles exported by Lumariver when using the "Simple" tone reproduction operator is still present, despite the result of profile report. This bug will need to be fixed before using those ICC profiles with your images.
 

Edited by FrankieT, 08 June 2025 - 10:42 AM.

[timaras]

I think your findings are consistent with the following lines from the Lumariver manual: "As cameras have similar response in similar lights it’s not important that the illuminant specified exactly matches what’s used in the target image. . . .  Note that it’s more important to be close in temperature for lights below 4000K than above as the change in behavior is non-linear. For example, a 100 degree change represents a much larger difference below 3000K than above 7000K."

Indeed this explains the high error for StdA. But I do not understand why the error *improves* as I deviate from my custom calibration illuminant (which I measure exactly in this case!).

Edited by timaras, 08 June 2025 - 10:48 AM.

[FrankieT]

The target transfer function is derived from information embedded in the target image itself and describes how to convert the pixel values to linear.

 

I also had an email exchange with the developers a few weeks ago asking whether Lumariver expects linear data, and confirmed that it's not necessary.

That's right. Lumariver does not require a linear target image when generating ICC profiles, but it does expect the "target transfer function" embedded in the exif data to match the transfer function actually applied to the data. At least, I have not found a way to change the target transfer function in the settings.

[timaras]

Regarding white balancing in an astro workflow, am I correct that, since icc profiles include white balance values (so does BQ's), then one would not want to perform a white balancing step (e.g., SPCC) prior to applying the profile, or if one does, then one would change the profile's white balance values to 1,1,1? 

It's worth reminding ourselves that there are two distinct transformations we perform. The first is from camera raw color space to a standard color space (XYZ or sRGB). This, in my opinion, is not optional for correct color. The second is the white balancing that aims to mimic the brain's chromatic adaptation, aka remove any effect of illuminant variation.

 

The forward matrices embedded within the icc profiles discussed in the last few posts do all of the color transformation and a little bit of the white balancing, with the WB heavy lifting being done from the WB gains applied just before.

 

The white balancing choice is an interesting one. A D65 choice, a-la-BQ, will most likely make many stars and galaxies appear warm. An SPCC choice will make most of them appear neutral white. How do we want them to appear?

If our goal is "how would the astro objects look to our brain if the eyes were much more sensitive", the question to answer really is: How much would our brain chromatically adapt in that (very hypothetical) setting?

Edited by timaras, 08 June 2025 - 11:43 AM.

[martz]

Martz, timaras

 

Just wanted to clarify that the defect we discovered in the ICC profiles exported by Lumariver when using the "Simple" tone reproduction operator is still present, despite the result of profile report. This bug will need to be fixed before using those ICC profiles with your images.
 

Here are the values of the icc profile prepared with DCamProf you previously shared (post #383):

 

 

And here the values of the icc profile prepared with Lumariver after I fed it the sRGB (linear TRC) image (post #392):

 

 

Does it still look off - it doesn't have zero values like the other one?

 

Thanks.  

[timaras]

Here are the values of the icc profile prepared with DCamProf you previously shared (post #383):

 

 

 

And here the values of the icc profile prepared with Lumariver after I fed it the sRGB (linear TRC) image (post #392):

 

 

 

Does it still look off - it doesn't have zero values like the other one?

 

Thanks.  

martz, can you share again the raw fits file and the file you feed at Lumariver, to have another go (you may have done so earlier but I want to avoid any mistakes)? I wonder if the bug is platform specific (I am at macOS).

[martz]

martz, can you share again the raw fits file and the file you feed at Lumariver, to have another go (you may have done so earlier but I want to avoid any mistakes)? I wonder if the bug is platform specific (I am at macOS).

Here are both.  Thanks.

[timaras]

martz, can you share again the raw fits file and the file you feed at Lumariver, to have another go (you may have done so earlier but I want to avoid any mistakes)? I wonder if the bug is platform specific (I am at macOS).

Indeed, the export works correctly in Windows (I tested Lumariver in a backup windows machine I have). The Mac version has the bug with the 0 elements (tested in two different machines). So you should be good to go! (I think you never got that 0 problem yourself, right?)

Edited by timaras, 08 June 2025 - 12:56 PM.

[martz]

Indeed, the export works correctly in Windows (I tested Lumariver in a backup windows machine I have). The Mac version has the bug with the 0 elements. So you should be good to go! (I think you never got that 0 problem yourself, right?)

I didn't see zeros.  My issues were due to user error--incorrect settings and the sRGB tag.  Thank you.

[timaras]

I had a very exciting day (astrophotographically speaking), as I managed today to color-calibrate my Canon R6 when shooting with a light pollution filter (Optolong L-Pro). I thought I would get a very poor result as the filter throws away almost half of the spectrum. I was reading a lot that these filters distort the colors too much to be useful for true broadband photography.
 
And, yet, when following the practices discussed in this thread to create a custom color correction matrix, I was able to achieve incredible color. 
 
I present the results, followed by detailed steps, in case others want to use these methods.
 
I used Canon R6 with EF 24-70mm lens, Optolong L-Pro, and captured the X-Rite Colorchecker Passport with a "full spectrum" bulb (4400K CCT).

 
Original colorchecker image, rendered using the "regular" R6 matrices and D50 white balance (error ΔΕ=11.82) :
 


 
Same image after correcting the white balance per channel gains [2.30 1 1.30], but keeping the regular R6 CCM (ΔΕ=6.19):
 


 
And after correcting with both WB gains and the new, optimized CCM (ΔΕ=2.49):
 


 

The last result, even though it has the filter, looks visually indistinguishable from the one without using a filter at all!
 
 
The steps I followed (high level) were:

• Capture the colorchecker image
• Create reference image for profiling (I used Rawtherapee)
• Generate color correction matrix and WB gains (I used dcamprof)
• Apply CCM/WB to original image (I used Pixinsight)

In an astro photo, I would apply the WB gains at the very start (before debayering) and the CCM at the last linear processing step.
 
In more detail:

 

1. Install Rawtherapee, DCamProf, and Argyll.

 

2. Capture the colorchecker image using this guidance (I also captured flats using the white back of the colorchecker passport but they did not meaningfully improved the results)

 

3. Open the RAW file in Rawtherapee, wengot to color management tab, scroll down and hit "Save reference image" without any WB. This saved a linear image as 16bit tif, R6_REF.tif .

 

I found that this type of image did not work when opening it for profiling in Lumariver in macOS. In seems the OS applies a default sRGB tone response curve, which throws away Lumariver's expected linear data. This was not an issue if the image was assigned a linear profile first, or processed in command line).

 

4. Generate intermediate ti3 file (this holds the RGB values of the captured colorchecker patches and the reference ones for the particular target shot):

 

scanin -p -v -dipn \
  R6_REF.tif \

    /Applications/Argyll_V3.3.0/ref/ColorCheckerHalfPassport.cht \
    /Applications/Argyll_V3.3.0/ref/ColorCheckerHalfPassport.cie

 

I am running this from the directory that the reference file is stored, but I need to point to the target chart files that come with Argyll.

 

5. Use DCamProf to solve for the optimal white balance and forward matrix (that's the core of the operation):

 

dcamprof make-profile \
  -g /Applications/dcamprof/data-examples/cc24-layout.json \
  -i D50 \
  R6_REF.ti3 \
  R6_profile.json

 

-g points to the location of the layout used, -i specifies the illuminant used (mine was close to D50), and the json file stores the results. I note the WB gains and forward matrix values displayed on the screen.

 

6. Optionally, create an ICC profile for the camera system:

dcamprof make-icc \
  -n R6 \
  -L \
  R6_profile.json \
  R6.icc

 

-L forces a linear, matrix only profile. I consider this step optional as I sometimes prefer to multiply with the matrix values manually.

 

7. Open the RAW image in the astro software (I used Pixinsight). White balance, Debayer, then either i) assign linear sRGB profile (I use this one), multiply with the ForwardMatrix (to take the data to XYZ D50), then multiply with the standard XYZD50 to XYZD65 CAT matrix and XYZ to sRGB matrices, or ii) assign the icc profile created earlier.

 

In the above, I also use the linear sRGB profile as my working profile.

 

8. Finally, convert the profile to the standard gamma sRGB, then save the image with that profile embedded.

 

Phew!

 

PS: In Pixinsight, the following code in Pixelmath (RGB/k single expression) will apply WB per channel gains to a RGGB CFA:

 

iif( x()%2==0 && y()%2==0,
     $T * Rgain,
   iif( x()%2==1 && y()%2==0,
        $T * Ggain,
     iif( x()%2==0 && y()%2==1,
          $T * Ggain,
          $T * Bgain)))

Where Rgain, Ggain, Bgain are the WB gains.

Post-debayer WB is instead applied per channel using the equations:

$T[0]*Rgain
$T[1]*Ggain
$T[2]*Bgain

Edited by timaras, 08 June 2025 - 04:36 PM.

[FrankieT]

(...)

 

The white balancing choice is an interesting one. A D65 choice, a-la-BQ, will most likely make many stars and galaxies appear warm. An SPCC choice will make most of them appear neutral white. How do we want them to appear?

If our goal is "how would the astro objects look to our brain if the eyes were much more sensitive", the question to answer really is: How much would our brain chromatically adapt in that (very hypothetical) setting?

 

 

(...)
 
I used Canon R6 with EF 24-70mm lens, Optolong L-Pro, and captured the X-Rite Colorchecker Passport with a "full spectrum" bulb (4400K CCT).

 

 

In normal lighting conditions, our visual system adapts to the ambient illuminant to achieve colour constancy, perceiving neutral whites as white. Under dark skies, with minimal ambient light, our eyes undergo dark adaptation, transitioning from photopic vision (cone-based, color-sensitive) to scotopic vision (rod-based, colour-insensitive). Since rod cells are not sensitive to colour, stars often appear whiteish rather than showing their true spectral colours, unless they are bright enough to engage the cone cells (e.g., Betelgeuse, reddish; Arcturus, yellowish).

 

In astrophotography, where a dominant illuminant usually does not exist, calibrating a camera to a D65 illuminant (~6500K, daylight-like) may, as you suggest, introduce a warm (yellowish) bias, misrepresenting stars’ true spectral colours. A neutral profile, calibrated under a flat-spectrum light source, better preserves intrinsic colours, though a perfectly flat-spectrum source is hard to achieve. In practice, a D55 illuminant (~5500K) is a reasonable compromise for a near-neutral white point. Alternatively, D50 is also a viable option for a general purpose profile, with a slightly cooler colour bias compared to a neutral white point. However, calibrating to an illuminant with a CCT ~4000K is, in my opinion, sub-optimal as it may make stars appear unnaturally cool and diminish red hues in objects like emission nebulae. That said, you can always consider a custom white balance based on a reference star (e.g., a G2V star like the Sun) to achieve a more "natural" look.

 

Anyhow, you now have the tools to create your own profiles, experiment and find what works best for your application.  

Edited by FrankieT, 09 June 2025 - 02:39 AM.

[FrankieT]

Bellow are the ΔΕ errors measured in darktable, the input target file is always the same, but the indicated illuminant is changing:

 

Indicated Illuminant     ΔΕ

Biolight (4400K)        1.84

Biolight /w flatfield   2.24

D65                     1.76

D50                     1.40

StdA (2850K)            12.29

 

I know this does not matter at the end of the day for this forum but it bothers me that except for StdA the error seems to be going in the opposite direction from what I would expect. If I had to guess I would think darktable calculates the errors using wrong/different illuminants.

What do the actual profile reports indicate?
 

Edited by FrankieT, 08 June 2025 - 07:39 PM.

[Domdron]

At least in Siril 1.2.x, PCC applies a BE as well, which may or may not be what you're looking for.

What makes you think that? The Siril docs for PCC (both for 1.2 and 1.4) don't mention BE in PCC, and the Siril docs are generally very thorough, so I will be surprised if that's the case.

[timaras]

calibrating to an illuminant with a CCT ~4000K is, in my opinion, sub-optimal as it may make stars appear unnaturally cool and diminish red hues in objects like emission nebulae. That said, you can always consider a custom white balance based on a reference star (e.g., a G2V star like the Sun) to achieve a more "natural" look.

It seems (as it was mentioned in earlier posts) that using a color rotation matrix optimized for ~4000K CCT illuminant is good enough for a wide range of actual CCTs, so long the white balance gains are chosen for carefully (e.g. based on star color as you mention). This is why I performed the tests in my earlier post: I obtained excellent ΔΕ across the 4400K-6500K CCT range by using the same 4400K-optimized matrix (but with optimized WB gains). 

 

I wish I had access to a real D50-D65 illuminant for these calibrations, but even though I offered to pay for it, it doesn’t seem it’s something that people offer as a service. And having to choose between this full spectrum bulb and an outdoor unknown illumination, I prefer the consistency of the former.

Edited by timaras, 09 June 2025 - 02:52 AM.

[BQ Octantis]

What makes you think that? The Siril docs for PCC (both for 1.2 and 1.4) don't mention BE in PCC, and the Siril docs are generally very thorough, so I will be surprised if that's the case.

Sorry, I should have written "BN" for background neutralization. Siril literally pushes the gain constants K0, K1, K2 and background neutralization constants B0, B1, B2 in the console when you run PCC:

 

 

To apply those in pixel math, it's Cn1 = Kn*Cn0 - Bn.

 

Once I figured that out, I quit doing BN manually (for images where PCC produces a usable result).

 

BQ

Edited by BQ Octantis, 09 June 2025 - 05:40 AM.

[Domdron]

Sorry, I should have written "BN" for background neutralization. Siril literally pushes the gain constants K0, K1, K2 and background neutralization constants B0, B1, B2 in the console when you run PCC:

 

 

To apply those in pixel math, it's Cn1 = Kn*Cn0 - Bn.

 

Once I figured that out, I quit doing BN manually (for images where PCC produces a usable result).

 

BQ

Ah, thanks for clarifying.

[BQ Octantis]

BTW, here are the colors of the Plankian locus with a linear CCT scale. Colorimetric chromaticity measurements map CCT to each star's effective temperature. If you choose to white balance or adapt to some arbitrary temperature, make a copy of the scale and slide the desired white temperature to the 6500K position. The entire locus will have that bias.

 

 

The consequence of this bias is a warping of the entire visual gamut to shift the perceptual white to the desired white point on the Planckian or Daylight locus. This will shift the ratios at which emission mixtures change from blue to purple and from pink to red.

 

Toggle

 

In my experience (in my first couple of years of AP, I followed the guidance of processing in Canon "Daylight" 5200K), this results in much more purple nebulae…which may or may not be your goal.

 

Cheers,

 

BQ

Edited by BQ Octantis, 10 June 2025 - 09:31 AM.

[timaras]

Indeed this explains the high error for StdA. But I do not understand why the error *improves* as I deviate from my custom calibration illuminant (which I measure exactly in this case!).

 

What do the actual profile reports indicate?
 

 

I had another go at this that resolved most of the issues. I now checked the error upon creating the profile, not after jpg export and assessing color error  in Darktable. The numbers make more sense now:

 

Indicated Illuminant    NEW ΔΕ (av/max)
Biolight w/flatfield    1.80 / 5.02  
Biolight                1.76 / 4.83
D65                     2.09 / 5.83
D50                     1.94 / 4.49
StdA                    2.40 / 4.76

 

This is expected behaviour, where the best result is when I feed the measured spectrum of the light source (~4400K), then it gets worse as the CCT used to optimize deviates. 

 

It's more evidence that the illuminant choice is not as critical, so long the correct WB gains are used. All the images produced with these profiles are visually very close (except StdA).

Edited by timaras, 10 June 2025 - 06:25 AM.

[timaras]

With the upgraded understanding of white balance and illuminants, I had another look at what SPCC does (e.g. as explained in Siril and PixInsight), and how does it differ from the processes discussed in this thread.

 

I am not sure it's needed at all?

 

To keep things simple we can assume that the spectral sensitivity functions of the dslr/mirrorless camera bayer filters are known, and that the color rotation matrices (from WB camera space data to e.g. sRGB D65) do not vary much between illuminants.

 

What I gather is this:

 

The ultimate goal is to decide the WB gains, i.e. which R/G/B point the camera records is neutral white and is mapped at the monitor's D65 white point.

 

The WB gains obtained via the conventional terrestrial methods are effectively: I am assuming some "white" illuminant spectrum (e.g. D50/D65), calculate what R/G/B values my camera records via its bayer filters, which is then mapped to D65 via the chromatic adaptation transform. The WB gains are the multipliers required to land the recorded R/G/B to D65 at sRGB space.

 

SPCC similarly assumes a known "white" spectrum to be used as reference. Let's further assume it's that of a G2V star, aka close to 5770K blackbody. SPCC then searches through WB gains such that the captured star colors best match the reference star colors. 

 

What I wonder is: since we have a terrestrially well-calibrated camera system, how much better would SPCC do compared to just using calculated WB gains of the desired white reference spectrum (WB gains for 5770K in this case)? 

 

In effect it seems to me that SPCC is the space equivalent of using a colorchecker to calculate WB gains. However in terrestrial photography would rely on the pre-calculated WB gains to white balance the captured images, in most non-professional scenarios.

 

I presume in astrophotography there can be some variations, particularly the absorption from the atmosphere and other day-to-day sky variations. It also may make sense to use SPCC for a new camera or when a filter is used, where the WB gains are not known. But other than these, SPCC doesn't seem to add that much value to the processing chain?

[BQ Octantis]

SPCC is just primitive XYZ-scaling AWB with a gray card. It does not do color matching at all.

 

A calibrated transfer function is an open-loop measurement. Applying SPCC to your calibrated measurement is like the camera running AWB on top of the Daylight setting. Pick one.

 

In colorimetry, there's no such thing as white balance. There is just a spectral distribution and an accurate transfer function to RGB proportions in the final color space.

 

Edited by BQ Octantis, 14 June 2025 - 03:39 AM.

[timaras]

Applying SPCC to your calibrated measurement is like the camera running AWB on top of the Daylight setting. Pick one.

Thanks for confirming, indeed SPCC seems redundant if the system has already WB applied to it.

I also wonder why is SPCC not used to full camera calibration (including the forward matrix). At the core of it, it's the same process as a colorchecker calibration: find a matrix so that the captured camera RGB values match reference RGB values obtained from known spectra (stars in the SPCC case, patches in the colorchecker case). So why is SPCC content with finding channel gains (basically, a diagonal matrix) instead of a 3x3 matrix?

My guess is that there just isn't enough varying spectral information in the deep sky objects to estimate a well-behaved transform matrix. Stars are mostly white, and everything else is narrowband emissions. But in principle one could capture several locations of the deep sky with enough spectral data, and match these against Gaia-derived references to calibrate the camera.

Edited by timaras, 14 June 2025 - 03:46 AM.

[FrankieT]

I presume in astrophotography there can be some variations, particularly the absorption from the atmosphere and other day-to-day sky variations. It also may make sense to use SPCC for a new camera or when a filter is used, where the WB gains are not known. But other than these, SPCC doesn't seem to add that much value to the processing chain?

I think you answered your own question in a way.
 

Typically, camera calibration is performed in a controlled environment. The profiles provide a very good starting point for accurate colour reproduction, but are not perfect and usually don’t account for all scene-specific imaging effects affecting colour and white balance.

 

In daylight photography for example, the scene illuminant rarely matches the calibration illuminant, so photographers use gray cards or colour checkers on-site to fine-tune white balance. In astrophotography, the “scene illuminant” (e.g., starlight or nebulae) has relatively consistent spectral properties, but several external factors, such as atmospheric effects, light pollution, or optics, may cause the observed white point to shift from the calibrated profile.

 

General purpose ICC profiles, with only a forward matrix, don’t correct white balance automatically. Then, SPCC becomes a useful tool for setting white balance, analogous to the white balance module in raw converters. A calibration profile that includes white balance correction should provide a consistent starting point, which might be sufficient in many cases. Nevertheless, SPCC or PCC may be still be useful to fine-tune white balance based on spectral or photometric data, to account for external effects or to apply a subjective look to the image.

Edited by FrankieT, 14 June 2025 - 09:54 AM.

[BQ Octantis]

Thanks for confirming, indeed SPCC seems redundant if the system has already WB applied to it.

 

I also wonder why is SPCC not used to full camera calibration (including the forward matrix). At the core of it, it's the same process as a colorchecker calibration: find a matrix so that the captured camera RGB values match reference RGB values obtained from known spectra (stars in the SPCC case, patches in the colorchecker case). So why is SPCC content with finding channel gains (basically, a diagonal matrix) instead of a 3x3 matrix?

 

My guess is that there just isn't enough varying spectral information in the deep sky objects to estimate a well-behaved transform matrix. Stars are mostly white, and everything else is narrowband emissions. But in principle one could capture several locations of the deep sky with enough spectral data, and match these against Gain-derived references to calibrate the camera.

 

Themos,

 

Stars are not white at all—every one of them lies along the Planckian locus, each with a measurable (CCT,Duv). But even if you can accommodate lens aberrations, the Planckian locus does not have enough orthogonality to perform a reliable mapping of all spectra to the sRGB color space. Let me post it again so you can see it visually—there are no greens or purples:

 

 

So what I discovered with my Planckian CCM endeavor was that using stars alone to generate a CCM results in a collapse of the chromaticity measurements onto a straight line tangent to the Planckian locus. Indeed, in later attempts, I simply collapsed the sRGB color space around the Planckian locus, which produced a better curve. For a more accurate overall transfer function, it would be better to use a shot of a display with an image of just the sRGB primaries—a project that's been in the back of my mind for some time now.

 

Lastly, no, the deep sky is not just narrowband emissions. The stars that provide the energy to excite the emissions from elements are all full-spectrum sources that also produce reflection blobs comprised of a wide diversity of chromaticites—to include the dreaded greens. These chromaticities are further morphed by absorption along the path to the camera. It is Pointer's Gamut that best defines the bounds of expectation:

 

 

Even emission nebula contain numerous primaries that form the gamut of expectation—and the chromaticities produced are quite diverse:

 

 

I would suggest you make some measurements of your own to refine your understanding and cosmological worldview. Report back once you've measured everything…

 

 

Cheers,

 

BQ

Edited by BQ Octantis, 14 June 2025 - 07:42 AM.

[timaras]

So what I discovered with my Planckian CCM endeavor was that using stars alone to generate a CCM results in a collapse of the chromaticity measurements onto a straight line tangent to the Planckian locus. Indeed, in later attempts, I simply collapsed the sRGB color space around the Planckian locus, which produced a better curve. For a more accurate overall transfer function, it would be better to use a shot of a display with an image of just the sRGB primaries—a project that's been in the back of my mind for some time now.

 

You know, I had read that post a few times in the past few months but I just now realized the value - you already implemented what I was suggesting earlier! And is seems to work as expected, i.e. it's good at making star/planckian colors but not a broad range.

 

 

I would suggest you make some measurements of your own to refine your understanding and cosmological worldview. Report back once you've measured everything…

 

 

 

Where are the colorchecker patches in that plot??