582 replies to this topic

[BlueMoon]

Uh-oh!—wrong DCP. The DCP we're using is the Digital Camera Profile. It's unrelated to the Digital Cinema Package.

Okay. Thanks. It seemed a bit odd to me. I should have recalled from it earlier in the thread ...   Edited my error and removed the link.

Edited by BlueMoon, 20 September 2022 - 10:38 PM.

[900SL]

This. Time to learn a bit of Pixel math: https://siril.org/tutorials/pixelmath/ along with better manual pre-processing skills. https://siril.org/tu...ls/tuto-manual/

 

I'm still in the "do it all in Siril" mode of thinking.

I was in a similar boat but I forced myself to watch a couple of the youtube videos on manual processing in Siril (I think Borealist.lite and a couple of others). I recommend the BL video although I have trouble staying awake . Sirils own tutorials are a good start too. 

 

Learning about how the automatic scripts work is very useful, there's a video out there on how to edit scripts so that you can do frame by frame background extraction and stopping before stacking, which means you can reject the poorer quality subs etc.

 

I'm still at novice-intermediate stage, it's a heck of a learning curve in general  

[Challenger75]

I've had some bad looking gradients, but I have to say, with very careful placement of points in the Background Extraction in Siril, Siril has done a great job in removing gradients.  After an auto generate of the points and zooming in and repositioning/deleting/adding some points is the most important process one can do in PP, IMO. Many times, I don't even need to use Gradient Xterminator in PS.

[BlueMoon]

This thread has really encouraged me to dig deeper into Siril, past my more casual use of the "pre-canned" scripts which for the most part have worked pretty well for me. Most of my targets are star clusters as I haven't had the time to do long integration for nebulae and the like. That, and I use an AZMP mount without a field rotator so my subs are many, short exposure shots.

 

One thing I'm interested in is trying to work out exposure times that prevent over-saturation of the image. I've heard that programs like Pixinsight have image analyzing for saturated pixel tools, but I'm a die-hard Linux and Open Source user and won't switch back to MS. I limit myself in that regard but it encourages me to find other solutions and keeps my "gray matter" tuned up. ImageMagick on Linux for instance is a powerful image manipulation software that has a CLI which I can run from a Bash script. I'm curious about it's capabilities as a pre-processor. Or perhaps RT can do it but I haven't investigated that yet.

 

At this point, I'm now studying the processing scripts, learning to do manual processing and looking into BQ's discussion concerning DCP. I have a whole Winter ahead of me ...

 

Clear(er) skies.

Edited by BlueMoon, 21 September 2022 - 09:02 AM.

[BQ Octantis]

This thread has really encouraged me to dig deeper into Siril, past my more casual use of the "pre-canned" scripts which for the most part have worked pretty well for me. Most of my targets are star clusters as I haven't had the time to do long integration for nebulae and the like. That, and I use an AZMP mount without a field rotator so my subs are many, short exposure shots.

 

One thing I'm interested in is trying to work out exposure times that prevent over-saturation of the image. I've heard that programs like Pixinsight have image analyzing for saturated pixel tools, but I'm a die-hard Linux and Open Source user and won't switch back to MS. I limit myself in that regard but it encourages me to find other solutions and keeps my "gray matter" tuned up. ImageMagick on Linux for instance is a powerful image manipulation software that has a CLI which I can run from a Bash script. I'm curious about it's capabilities as a pre-processor. Or perhaps RT can do it but I haven't investigated that yet.

I can't speak to the scripts, but there are heaps of variations to test on your data—and Siril's speed makes it easy to try different things! For instance, I'm finding I can get reasonable Ha from my 200mm f/2.8 data if I follow the all-Siril workflow

 

1. Calibrate
2. Register
3. Stack
4. Photometric color balance
5. Stretch
6. Remove green noise without preserving luminosity

 

This actually produces almost as much Ha reds as RT DCP processing, though with a bit more noise in the background—but sharper highlights, too. So it's certainly worth experimenting to find what works with your setup.

 

BQ

[BlueMoon]

 

I can't speak to the scripts, but there are heaps of variations to test on your data. So it's certainly worth experimenting to find what works with your setup.

Bingo. Lots of variations. Right now, I'm working on some star cluster data I've been shooting, M2, M15, a few others. I had a nice set of wide-field subs of the Cygnus Wall but I tossed it thinking I would re-shoot it again later. Er, nope. I just started shooting with the SL3 as well so, more experimenting, more testing, more trying. If I come up with a good processing sequence, I'll be sure to post it as well. Perhaps we should start a thread just for sharing Siril techniques.

Edited by BlueMoon, 21 September 2022 - 07:24 PM.

[BQ Octantis]

You are correct.. Frack, I got so confused.  Even in my previous post I was thinking GBRG and I'm an flippin idiot.
Thank you for the correction.    Now I'm going to go through the images again and recheck.  I get the green images, but Sometimes I get them and sometimes not, so it that was throwing me.  On a restack with Background neutralization the skies are black as expected.   Thank you Mark !!

Clear Skies !!

I got tired of seeing the Bayer pattern error message, so I updated it to GBRG while Siril was doing a calibration run. It immediately quit complaining mid-run!

 

As to the green dominance, I discovered in RT that without applying any white balance, the green is always overdriven. As per developer Cecile's comment on the matter (Post #41), the sensor is just more sensitive in the green channel. I worry about tackling this with background neutralization on the subs vice photometric color calibration on the stack. For lenthy, wide-field gradients, it's a tough trade though…

Edited by BQ Octantis, 21 September 2022 - 07:30 PM.

[BQ Octantis]

Hey, I was able to get some very nice color with Siril photometric color calibration and without the DCP. I still used RawTherapee, but I disabled white balance and selected no camera profile. I also turned off fringe and CA control. I crushed the output with a linear tone curve set to 35% to prevent RT from clipping the highlights. RT always normalizes the output, but 35% put it in the ball park of the Siril stack histogram range.

 

The flow after stacking in Siril was

 

1. Photometric calibration (Siril)
2. Background extraction (Siril)
3. Asinh transformation (Siril)
4. Histogram transformation (Siril)
5. Star removal (Starnet++)
6. Luminosity-scaled noise control (RawTherapee)
7. Color saturation by color group (Siril)
8. Star layer recombination (Photoshop)

 

The green removal tool turns my greenish OIII a deep blue, so I skipped it.

 

Preview (click for full size @ 67% scale)

 

I tried to keep the saturation down—the asinh really increases color, so it's easy to go overboard…

 

BQ

[vidrazor]

Hey, I was able to get some very nice color with Siril photometric color calibration and without the DCP. I still used RawTherapee, but I disabled white balance and selected no camera profile. I also turned off fringe and CA control. I crushed the output with a linear tone curve set to 35% to prevent RT from clipping the highlights. RT always normalizes the output, but 35% put it in the ball park of the Siril stack histogram range.

I tried to keep the saturation down—the asinh really increases color, so it's easy to go overboard…

BQ

That's looking good. Surprisingly not a lot of color noise, which tends to really pop with stretches, especially when Asinh is involved. RT noise reduction tools work quite well for both luminosity and color noise. The deconvolver sharpening tool is useful too, used judiciously. I have not experimented with the wavelet tools in RT, supposedly they are quite useful as well.

[BQ Octantis]

That's looking good. Surprisingly not a lot of color noise, which tends to really pop with stretches, especially when Asinh is involved. RT noise reduction tools work quite well for both luminosity and color noise. The deconvolver sharpening tool is useful too, used judiciously. I have not experimented with the wavelet tools in RT, supposedly they are quite useful as well.

 

I'm finding the DCP (as with the ICC) results in a lot of green noise in the background. But this most recent method results in a reasonably clean background—even without the green removal function in Siril.

 

Step 6 was actually a combined set of actions. RawTherapee doesn't commit the changes until you save off a new version. The combined action on the starless image was:

 

• Global saturation boost, with a Luminosity mask shaped to cut off below where the background detail falls off.
• Luminosity noise reduction, with a Luminosity mask shaped shaped to to peak in the background and cut off where the cloud detail stars.
• Chromatic noise reduction, with a Luminosity mask shaped to cut off just above the background.

 

This resulted in a shockingly clean background with some nicely boosted colors in the nebula. The color boost by color group is then really effective on the color contrasts in the nebula without having to worry about rubbish in the background.

 

I'll probably post a little how-to on the RT noise reduction here in the forum in short order…

[BQ Octantis]

Here's what I was able to muster with the same workflow on the original data I posted:

 

Preview (click for full size @ 67% scale)

 

The Ha is a little more subdued and more magenta than with the DCP (Post #78). But there is most definitely plenty of color. And the OIII from NGC 3199 is a lot more prominent in this one.

 

I'm now wondering about using the crushing method with the DCP…

 

BQ

Edited by BQ Octantis, 23 September 2022 - 07:23 PM.

[galacticinsomnia]

Hey, I was able to get some very nice color with Siril photometric color calibration and without the DCP. I still used RawTherapee, but I disabled white balance and selected no camera profile. I also turned off fringe and CA control. I crushed the output with a linear tone curve set to 35% to prevent RT from clipping the highlights. RT always normalizes the output, but 35% put it in the ball park of the Siril stack histogram range.

 

The flow after stacking in Siril was

 

1. Photometric calibration (Siril)
2. Background extraction (Siril)
3. Asinh transformation (Siril)
4. Histogram transformation (Siril)
5. Star removal (Starnet++)
6. Luminosity-scaled noise control (RawTherapee)
7. Color saturation by color group (Siril)
8. Star layer recombination (Photoshop)

 

The green removal tool turns my greenish OIII a deep blue, so I skipped it.

 

Preview (click for full size @ 67% scale)

 

I tried to keep the saturation down—the asinh really increases color, so it's easy to go overboard…

 

BQ

This cannot come from the 40 min of lights you posted earlier.  New data, more data ?  It looks really good whatever you did, but I can't image you created this from the original 40 min.

Congrats on the flow..

Clear Skies !!

[BQ Octantis]

This cannot come from the 40 min of lights you posted earlier.  New data, more data ?  It looks really good whatever you did, but I can't image you created this from the original 40 min.

Congrats on the flow..

 

Ha! Thanks! And no, this wasn't from raw data set I posted—that was from my Canon 50mm f/1.8 Mark 1 @ f/3.5. This second one was from my Canon 200mm f/2.8L II @ ∅52mm. I shot Carina with every gun I've got: 8mm, 15mm, 50mm, 55mm, 200mm, 250mm, 750mm, and 2700mm. So I've got a fair bit of data to play with. If anything, this exercise has validated my decision to keeps raws instead of stacks!

 

This Siril processing is really enabling a lot of exciting workflow experimentation. Stacking with Lynkeos is quite processor intensive—particularly with statistical rejection—so any variation on the demosaicing would take at least an hour to evaluate at the stack level. Siril is way less processor intensive and way faster even with statistical rejection—single-digit minutes at most. And the Autostretch feature allows me to immediately simulate what would otherwise be many additional steps, even at a sub level. I should be able to produce a Warhol-esque set of results to choose from in short order…

 

 

BQ

[BlueMoon]

 

This Siril processing is really enabling a lot of exciting workflow experimentation.

+1 To be sure. I've been working with manually processing subs instead of using the scripts (which are fine in their own right) and I'm discovering a lot of tweaks. Using the registration and plot tool has helped me remove outlier subs and improved my final stack already. The Siril folks have a good tutorial for starting: https://siril.org/tu...ls/tuto-manual/

 

My current project is to extract whatever color data I can get from stars in various clusters to show how they look naturally.

 

Clear(er) skies

Edited by BlueMoon, 24 September 2022 - 10:29 AM.

[BQ Octantis]

Today's color project was a revisit of the Great Southern Chook (a.k.a., the Running Chicken Nebula). It was my first good result with boosting Ha after Starnet++ and RawTherapee back during the Great Pandemic. The three ICs that comprise the chicken are buried deep in the stars of the Carina Arm of the Milky Way, making bringing up the nebula quite tricky. The objective here was to find the parameters for the DCP that produced the best results.

 

I ran a set of 16 raws through 6 options to compare outcomes:

 

1. DCP, AWB, no LT, 80% tone curve


2. DCP, AWB, LT, 80% tone curve


3. DCP, Daylight WB, no LT, 80% tone curve


4. DCP, Daylight WB, LT, 80% tone curve


5. No DCP, No WB, 30% tone curve


6. All Siril (control)

 

First, the LT cleans up the purple/magenta in the highlights a fair bit. It looks like the ICC/DCP is the source of purple CA around the bright stars in most beginner images; I had to leave CA and fringe control on to reduce it. The LT helps reduce the purple around the bright tones, albeit not completely—dense star clusters suffer the worst. Within the nebula, AWB was was too blue while daylight was too red—so the color temperature is clearly necessarily important. A 50% blend of the two layers seemed to neutralize the background, so I the color temperature between the daylight and AWB, I was surprised by the outcome: 4750K. One of the reasons I first used raws was so I could adjust the color temperature to get a neutral sky background in my nascent dark sky shots. Through tweaking back then, I found the best color temperature for ACR was 4750K! Coming to the same result through an entirely different flow felt somehow validating.

 

So that converged my workflows to just three for full stacks of the 146 ∅52mm subs. Starnetting the results, there was a clear delineation of what was emphasized by each color process:

 

Click for toggle @ 50% scale

 

I think which process to chose depends on the color outcome I want—accuracy be ****! I felt the Siril control was a bit too green-dominated for my taste, so the end of the day (literally), a merge between the DCP result and the no-profile RawTherapee result yielded a peak in color contrasts (particularly in the background) that to me were the most intriguing to the eye. This included star colors—the Siril control yielded too much green in the star wash of the Carina Arm, so the RawTherapee no-DCP result was more eye-pleasing. The final result was worth a push to the 'Bin:

 

 

BQ

Edited by BQ Octantis, 25 September 2022 - 05:16 AM.

[BlueMoon]

 

so the color temperature is necessarily important. Averaging the color temperature between the daylight and AWB, I was struck by the average: 4750K.

I'm following your posts with great interest BQ and learning quite a bit. I agree with your assessment on temperature. Interesting as I've been messing around in Darktable and hit almost the same temp as you, 4700K to my eye on some clusters I've imaged. Haven't tweaked AWB much yet ...

[BQ Octantis]

I'm following your posts with great interest BQ and learning quite a bit. I agree with your assessment on temperature. Interesting as I've been messing around in Darktable and hit almost the same temp as you, 4700K to my eye on some clusters I've imaged. Haven't tweaked AWB much yet ...

The AWB puts the bulk of the low end of the RGB tone curves roughly on top of each other on the histogram. Even though I used photometric calibration on the result to set the final white point, the star colors were way off. I read that the DCP constants are applied in HSV color space—after obtaining the XYZ tristimulus values from the RGB data. So whatever AWB does to the white point isn't easily recoverable. Between that and the too much blue in the nebula, I did not find a benefit to it.

 

Honestly, the no WB/no DCP option produced the most "natural" looking result across the image as a whole. That is the luminosity I used for the nebula layer. The prominent green across the Siril control drove the luminosity contrast way down, and the DCP luminosity had blowouts around the star cluster above the chicken (NGC 3766).

 

Here's a fun toggle through the interim steps—you can see the comparison between the no DCP/WB and the DCP+LT+4750K outcomes:

 

https://www.cloudyni...412_8094873.png

 

I think from here, I'll try both options in processing. I really enoyed the non-DCP result on the Carina Nebula (Post #108)…but I might still have to give the DCP/LT/4750K a go…

 

Cheers,

 

BQ

Edited by BQ Octantis, 25 September 2022 - 07:06 AM.

[sharkmelley]

Even though I used photometric calibration on the result to set the final white point, the star colors were way off.

Photometric calibration needs to be performed on linear data.  The image file coming from RawTherapee will have the (non-linear) tone response curve applied for the colour space you are using (which is probably sRGB).  It's not clear to me what photometric calibration will do to such data.

 

Mark

[Alen K]

Photometric calibration needs to be performed on linear data.  The image file coming from RawTherapee will have the (non-linear) tone response curve applied for the colour space you are using (which is probably sRGB).  It's not clear to me what photometric calibration will do to such data.

 

Mark

Linear (32 bit?) output is an option from RT. 

[sharkmelley]

Linear (32 bit?) output is an option from RT. 

That's true.  Since RT is properly colour managed, a linear (i.e. gamma=1.0) output profile can certainly be used.  But it's quite a specialized workflow and is not something that most people will be doing by default.

 

[Edit:  Also, since Siril is not a colour-managed application, it won't correctly display data imported with a linear profile]

Edited by sharkmelley, 25 September 2022 - 09:21 AM.

[BQ Octantis]

Photometric calibration needs to be performed on linear data.  The image file coming from RawTherapee will have the (non-linear) tone response curve applied for the colour space you are using (which is probably sRGB).  It's not clear to me what photometric calibration will do to such data.

The sRGB is non-linear compression from display to file, but I apply a gamma of 1.0 in RT with a rendering intent of Absolute Colorimetric. Siril recognizes the sRGB decompressed data as linear, and asinh and histogram manipulation behave as expected. Siril has no qualms with trying to apply photometric calibration to anything you feed it, to include sRGB-compressed TIFs saved by Siril itself.

 

What vexes me is that I get a more aesthetically acceptable result if I apply the photometric color calibration to linear RGBs (with darks & flat applied) out of RT LMMSE TIFs (no white balance, no camera profile—so it's green to start with) than I do on the FITS stack from Siril-imported raws (with the same darks and flat applied)—demosaiced with literally the same LMMSE algorithm.

 

For instance,

 

 

Here are three samples from this last stack at 100% scale:

 

• Sample 1 (preview above)
• Sample 2
• Sample 3

 

If I do a Remove Green Noise… with preserve lightness checked, it blows away both the green and most of the color—which was the original problem that made me start this thread!

 

BQ

Edited by BQ Octantis, 25 September 2022 - 12:29 PM.

[BlueMoon]

M2 with 30x20 subs, Canon SL3 (stock). Siril scripted not manually processing. I tried an experiment by converting my .CR3 files to .tif in Darktable, then importing the .tif files into Siril for conversion to .fits. In this test, I didn't apply any parameter changes to AWB, temp or camera profile in Darktable, just default values. The image does not have any photometric color info applied (unable to solve) and is only lights, no dark, biases or flats (next tests). The colors you see are captured in the images which is a primary goal for me. The first thing I noticed was a huge decrease in green, no need to apply remove green color in Siril. The second is that I was able to tease the colors out with less stretching and background noise. I don't know if this is going in a right direction but it seems promising. The image is a crop from a crop so it's sharpness is not the best and I need to continue working on it.

Attached Thumbnails

• 

Edited by BlueMoon, 25 September 2022 - 10:56 AM.

[BQ Octantis]

M2 with 30x20 subs, Canon SL3 (stock). Siril scripted not manually processing. I tried an experiment by converting my .CR3 files to .tif in Darktable, then importing the .tif files into Siril for conversion to .fits. In this test, I didn't apply any parameter changes to AWB, temp or camera profile in Darktable, just default values. The image does not have any photometric color info applied (unable to solve) and is only lights, no dark, biases or flats (next tests). The colors you see are captured in the images which is a primary goal for me. The first thing I noticed was a huge decrease in green, no need to apply remove green color in Siril. The second is that I was able to tease the colors out with less stretching and background noise. I don't know if this is going in a right direction but it seems promising. The image is a crop from a crop so it's sharpness is not the best and I need to continue working on it.

Very nice colors!

[BQ Octantis]

… since Siril is not a colour-managed application, it won't correctly display data imported with a linear profile

Is this true? Siril gives the option to embed an ICC in its TIFs…

[sharkmelley]

Is this true? Siril gives the option to embed an ICC in its TIFs…

I saved a file from RawTherapee with a linear ICC profile.  When I opened it in Siril it was displayed incorrectly.  Siril was clearly assuming the file was standard sRGB, ignoring the embedded ICC.