582 replies to this topic

[BQ Octantis]

G'day all,

 

I was hoping to develop an all-Siril workflow this year. I had my first crack at Siril calibration earlier in the year on my Lobster & Cats Paw Nebulas, but the results seemed quite subdued…so I abandoned the effort. I tried again with some Milky Way shots recently and found my Carina and Running Chicken Nebulas disturbingly without color. I found another user with a similar result who claimed it was the darks that were over correcting, but I got the same result with using =2048 as with using a stack of temperature-matched darks. Boosting saturation in post only brings up the star color.

 

Here's a stack comparison of my baseline RawTherapee calibration to a basic Siril calibration with =2048 as the dark. I used LMMSE demosaicing in both apps. Stacking was in Siril with parameters exactly the same for both (linear background extraction + global star registration + 90% FWHM stacking); processing was also in Siril with the same function flow for both: colorimetric color + arcsinh + histogram + green reduction + saturation boost:

 

 

I would expect a bleeding-edge astrophotography app to blow even the most sophisticated photography app out of the water. But the results aren't even close! Is there something I'm missing? Or does Siril require a modded DSLR at a minimum?

 

BQ

Edited by BQ Octantis, 16 September 2022 - 04:17 PM.

[BlueMoon]

 

Or does Siril require a modded DSLR at a minimum?

I've wondered the same thing. I imaged in Cygnus quite a bit this summer with an unmodded Canon T3i and noticed subdued coloring as well.  At the time I was learning Siril so I'm not sure I was using it to it's best and may not have processed it too well. I wasn't doing a lot of subs either, mostly 30x30 which I suspect wasn't enough color data to start with.

 

One thing I read from another CN poster was that he pushed the global saturation up BEFORE any other processing. He mentioned loss of color and being very difficult to get it back after stretching, etc. I adopted that idea and I push my global saturation to 0.25 beforehand. It does seem to provide better color in some images.

 

Clear skies.

Edited by BlueMoon, 16 September 2022 - 04:41 PM.

[galacticinsomnia]

Siril has given me near perfect color regardless of full spectrum or un-modded canon EOS-m and Eos-r.

I have noticed, a very very flat profile, but I have no idea what and how you are stretching, but the examples BQ posted seem to be out of focus.  Average FWHM for the stars in the image ? 

I stretch in Siril also most of the time and don't have any issue with my colors, and the only time I use saturation is when I am seperating and combining channels. 

Going to say that the color data is there, just need to work on the data. 

Feel to post your stack, or a folder of subs and calibration frames.
Note the FL and Sensor in case someone wants to use photometric calibration on the colors.


Edit:: Further thought, that you may want to try calibrated and uncalibrated.  Don't use any bias settings at all and see what happens.  You know how I am about calibration frames, and maybe this is the secret sauce ?  I have no idea really, but not having your issues so I am throwing stuff at the wall and maybe something will stick..
Clear Skies !!

Edited by galacticinsomnia, 16 September 2022 - 05:43 PM.

[vidrazor]

G'day all,

 

I was hoping to develop an all-Siril workflow this year. I had my first crack at Siril calibration earlier in the year on my Lobster & Cats Paw Nebulas, but the results seemed quite subdued…so I abandoned the effort. I tried again with some Milky Way shots recently and found my Carina and Running Chicken Nebulas disturbingly without color. I found another user with a similar result who claimed it was the darks that were over correcting, but I got the same result with using =2048 as with using a stack of temperature-matched darks. Boosting saturation in post only brings up the star color.

 

Here's a stack comparison of my baseline RawTherapee calibration to a basic Siril calibration with =2048 as the dark. I used LMMSE demosaicing in both apps. Stacking was in Siril with parameters exactly the same for both (linear background extraction + global star registration + 90% FWHM stacking); processing was also in Siril with the same function flow for both: colorimetric color + arcsinh + histogram + green reduction + saturation boost:

 

I would expect a bleeding-edge astrophotography app to blow even the most sophisticated photography app out of the water. But the results aren't even close! Is there something I'm missing? Or does Siril require a modded DSLR at a minimum?

 

BQ

Are you using generalized hyperbolic transformation? Although in this image below I only got 1 hour's worth of time with a blaring full moon on this target (and I was out of focus too, bad night LOL), I got the majority of this color making adjustments in the generalized hyperbolic transformation window. I have to say I am not familiar with GHT, but I pulled the two top sliders out around 2/3 out from the left, and a lot of color popped out. Unmodded Olympus E-M5 Mk II on an SVBONY 102mm f/7 with a Hotech flattener, CEM26, 60 1 minute subs at ISO 400.
 

Attached Thumbnails

• 

Edited by vidrazor, 16 September 2022 - 04:55 PM.

[BQ Octantis]

One thing I read from another CN poster was that he pushed the global saturation up BEFORE any other processing. He mentioned loss of color and being very difficult to get it back after stretching, etc. I adopted that idea and I push my global saturation to 0.25 beforehand. It does seem to provide better color in some images.

 

Well that does bring up some color, but nothing like the bright reds straight out of the RawTherapee-calibrated subs.

 

 

Are you using generalized hyperbolic transformation? Although in this image below I only got 1 hour's worth of time with a blaring full moon on this target (and I was out of focus too, bad night LOL), I got the majority of this color making adjustments in the generalized hyperbolic transformation window. I have to say I am not familiar with GHT, but I pulled the two top sliders out around 2/3 out from the left, and a lot of color popped out. Unmodded Olympus E-M5 Mk II on an SVBONY 102mm f/7 with a Hotech flattener, CEM26, 60 1 minute subs at ISO 400.
 

 

No, I just used the basic stretching workflow in the Siril processing tutorial. The whole color + streching process is

 

1. Background Extraction (per sub)
2. Photometric Color Calibration
3. Asinh Transformation
4. Histogram Transformation
5. Remove Green Noise
6. Color Saturation

 

Messing with the hyperbolic transformation first seems to really boost the cyan, but it does nothing for the red.

 

 

Feel to post your stack, or a folder of subs and calibration frames.
Note the FL and Sensor in case someone wants to use photometric calibration on the colors.

 

Oooo…I've never done a "process my data" post. I might consider it. Like one of those "PI Users Only" posts, but for Siril.

[BQ Octantis]

Edit:: Further thought, that you may want to try calibrated and uncalibrated.  Don't use any bias settings at all and see what happens.  You know how I am about calibration frames, and maybe this is the secret sauce ?  I have no idea really, but not having your issues so I am throwing stuff at the wall and maybe something will stick..
Clear Skies !!

 

I was gonna say! Aren't you the guy that's tried to convince us not to use darks? I think I'm starting to see the light!

 

At least for Siril…

Edited by BQ Octantis, 16 September 2022 - 06:35 PM.

[BlueMoon]

 

Well that does bring up some color, but nothing like the bright reds straight out of the RawTherapee-calibrated subs.

I've just installed the program on my PC so I'll give it a try. Zero experience with it but it looks comprehensive.

Edited by BlueMoon, 16 September 2022 - 06:51 PM.

[vidrazor]

I was gonna say! Aren't you the guy that's tried to convince us not to use darks? I think I'm starting to see the light!

At least for Siril…

Well I used a full set of calibration frames on mine and wound up with what you see. Certainly doesn't hurt to try tho. What camera are you using?
 

[BQ Octantis]

Well I used a full set of calibration frames on mine and wound up with what you see. Certainly doesn't hurt to try tho. What camera are you using?

 

A stock Canon 600D/T3i.

 

Should I post a "Process my Stack?" thread, or a "Stack my Subs" thread?

 

Processing others' subs always felt like a chore…

Edited by BQ Octantis, 16 September 2022 - 07:12 PM.

[vidrazor]

A stock Canon 600D/T3i.
Should I post a "Process my Stack?" thread, or a "Stack my Subs" thread?
Processing others' subs always felt like a chore…

If you have online storage to upload the subs and calibration frames I would be curious to give it a whirl.

[galacticinsomnia]

Well that does bring up some color, but nothing like the bright reds straight out of the RawTherapee-calibrated subs.

 

 

 

No, I just used the basic stretching workflow in the Siril processing tutorial. The whole color + streching process is

 

1. Background Extraction (per sub)
2. Photometric Color Calibration
3. Asinh Transformation
4. Histogram Transformation
5. Remove Green Noise
6. Color Saturation

 

Messing with the hyperbolic transformation first seems to really boost the cyan, but it does nothing for the red.

 

Oooo…I've never done a "process my data" post. I might consider it. Like one of those "PI Users Only" posts, but for Siril.

You may want to try doing the photometric calibration at the end.
I don't see Color Calibration--->Color Calibration in your flow.  Doing this will bring your reds out.
It is right above the Photomentric Calibration, and by going to the green channel, selecting a broad area that represents the blacks and darkest background areas, without nebulosity, click neutralize background I  Then make a selection of a representation in the green channel again, of the Midtones, which can include nebulosity. Click apply. that should show you a corrected color, for the rest of the flow..

Do this before your photometric calibration.   I choose to do photometric as one of the last steps when I am looking for more color..
Then adjust any more of your Saturation.

Clear Skies !!

[BQ Octantis]

If you have online storage to upload the subs and calibration frames I would be curious to give it a whirl.

Of course I have online storage!

 

Here's a Google Drive share of the lights, darks, a flat, and both stacks. Bias is =2048.

 

https://drive.google...8Ap?usp=sharing

 

Lens is Canon 50mm f/1.8 Mark 1 @ f/3.5. Camera is Canon 600D/T3i@ISO800. All in the metadata of the lights. The metadata of the darks is manipulated to be uniform so RawTherapee will recognize them as a single stack.

 

BQ

[galacticinsomnia]

Is this a non modded t3i ?

 

In my stack, I get some !!

19:43:55: Bayer pattern found in header (GBRG) is different from Bayer pattern in settings (RGGB). Overriding settings.
19:43:55: Filter Pattern: GBRG
19:43:56: Saving FITS: file pp_light_00016.fit, 3 layer(s), 5202x3464 pixels
19:43:56: Saving FITS: file pp_light_00039.fit, 3 layer(s), 5202x3464 pixels
19:43:56: Reading FITS: file light_00017.fit, 1 layer(s), 5202x3464 pixels
19:43:56: Bayer pattern found in header (GBRG) is different from Bayer pattern in settings (RGGB). Overriding settings.
19:43:56: Filter Pattern: GBRG
19:43:57: Reading FITS: file light_00075.fit, 1 layer(s), 5202x3464 pixels
19:43:57: Bayer pattern found in header (GBRG) is different from Bayer pattern in settings (RGGB). Overriding settings.
19:43:57: Filter Pattern: GBRG

Not sure why some of the images are weird..   This is a stack of ONLY LIGHTS.. No darks, Flats, Or bias settings..

Hmmmmm...

Clear Skies !!

Edited by galacticinsomnia, 16 September 2022 - 09:59 PM.

[BlueMoon]

 

Bayer pattern found in header (GBRG) is different from Bayer pattern in settings (RGGB). Overriding settings.

It's a setting in Siril preferences. You can change it there to GBRG or let Siril just override it automatically as you see in the console log.

Edited by BlueMoon, 16 September 2022 - 10:45 PM.

[galacticinsomnia]

It's a setting in Siril preferences. You can change it there to GBRG or let Siril just override it automatically as you see in the console log.

Thanks, I get the setting reading, my issue with the lights are that it wasn't happening with Every light frame, just with some light frames, while Siril is handling the errors, for different from configured bayer patterns, if these lights are all from the same camera, then something is askew..

My understanding is the sensor bayer pattern for the t3i is RGBG, and not GBRG.  But my setting in Siril is RGGB.  Something isn't right.  I didn't compare header data or dump them for analysis.  Someone who is interested in solving this rutroh mystery can go for it.

Clear Skies !!

[galacticinsomnia]

I've wondered the same thing. I imaged in Cygnus quite a bit this summer with an unmodded Canon T3i and noticed subdued coloring as well.  At the time I was learning Siril so I'm not sure I was using it to it's best and may not have processed it too well. I wasn't doing a lot of subs either, mostly 30x30 which I suspect wasn't enough color data to start with.

 

One thing I read from another CN poster was that he pushed the global saturation up BEFORE any other processing. He mentioned loss of color and being very difficult to get it back after stretching, etc. I adopted that idea and I push my global saturation to 0.25 beforehand. It does seem to provide better color in some images.

 

Clear skies.

I've been shooting with canon cameras for a long time.  The t2i and t3i are almost the same.  The T3i though is the first iteration of Rebel that I skipped.  

However what I think is worth noting is the HUGE difference between emission nebulosity and reflection/reflective nebulousity.  Check your targets.  Emmission targets take hours and hours of data to get any good red out of them, and as the OP of the topic posted 30 sec exposures on an Emission nebula, umm yeah, for that target, you need tons and tons of aquisition time, and even then, you may enjoy the image, but you can get a lot more with a modded camera.  Took me 27hours of aquisition time to on an unmodified canon t6i, to get even a blip of the crescent nebula, ngc 6888. 

Sorry if this is all information you already know, but its funny because just a couple years ago I was imaging without my modded camera, and even getting bright emission nebula, I was having a problem.  No problems since the full spectrum mods.

Clear Skies !!

[vidrazor]

Of course I have online storage!

Here's a Google Drive share of the lights, darks, a flat, and both stacks. Bias is =2048.

https://drive.google...8Ap?usp=sharing

Lens is Canon 50mm f/1.8 Mark 1 @ f/3.5. Camera is Canon 600D/T3i@ISO800. All in the metadata of the lights. The metadata of the darks is manipulated to be uniform so RawTherapee will recognize them as a single stack.

BQ

Only one flat? What kind of flats cheapskate are you anyway?

 

Could you fire off 4 or 5 bias? The script I'm gonna use requires a full set of calibration frames.

 

Thanks.
 

[galacticinsomnia]

This is a repost... I think...  Don't see the other one, maybe it didn't save.. I digress...

This is what I got out of your lights.  This is not bad for 40+ minutes of aquisition time on an over 11 year old sensor.

Good job.



Clear Skies !!

[BlueMoon]

 

Thanks, I get the setting reading, my issue with the lights are that it wasn't happening with Every light frame, just with some light frames, while Siril is handling the errors, for different from configured bayer patterns, if these lights are all from the same camera, then something is askew..

Ah, I see. Yeah, that would be pretty weird. I guess all I can say is that I've never seen that show in any of my processing of .CR2 files from my T3i either.

[BQ Octantis]

Only one flat? What kind of flats cheapskate are you anyway?

 

Could you fire off 4 or 5 bias? The script I'm gonna use requires a full set of calibration frames.

 

Thanks.

 

Yes, only one flat. RawTherapee will only take one flat. And the math shows a well-exposed flat contributes very little to the noise. Both RawTherapee and Siril blur the heck out of it. And Siril doesn't even deBayer it, so it's combining all three channels.

 

As to bias, Siril sez, bias adds more noise than troubles it fixes. But as a rogue disciple of Bracken, I have bias galore—and I've uploaded my tome to the drive.

 

BQ

Edited by BQ Octantis, 17 September 2022 - 07:03 AM.

[BQ Octantis]

However what I think is worth noting is the HUGE difference between emission nebulosity and reflection/reflective nebulousity.  Check your targets.  Emmission targets take hours and hours of data to get any good red out of them, and as the OP of the topic posted 30 sec exposures on an Emission nebula, umm yeah, for that target, you need tons and tons of aquisition time, and even then, you may enjoy the image, but you can get a lot more with a modded camera.  Took me 27hours of aquisition time to on an unmodified canon t6i, to get even a blip of the crescent nebula, ngc 6888. 

Sorry if this is all information you already know, but its funny because just a couple years ago I was imaging without my modded camera, and even getting bright emission nebula, I was having a problem.  No problems since the full spectrum mods.

Now I'm really starting to see the light.

 

Siril (and seemingly Pixinsight, from Bob's result in my "Process my Data" post over in the Beginners forum) requires the light data to be waaay above the noise floor. I had certainly noticed that my light data in the unscaled stack ended at well under half the histogram. But RawTherapee had never complained—and I've gotten heaps of color data from much dimmer targets with far less data. Carina is literally the brightest Ha target in the night sky…but 30sec at f/3.5 on 4.3µm pixels in an unmodded DSLR clearly isn't enough for Siril.

 

Which is fine for me. I never had success with even stacking raws with Lynkeos, so I started with calibration in Photoshop before discovering RawTherapee. 16-bit TIFFs serve as currency between apps, and Siril also does just fine with them. It's far superior to Lynkeos for stacking—for instance, Lynkeos fails if the offset between frames is more than ~200 pixels. And all its other features that save me heaps of time in post.

 

For what it's worth, that bloke who posted over in the Experienced forum showed that DSS should also be able to handle calibrating this data in color. And you wouldn't believe how tiny of histograms I've seen in stacks out of that!

 

BQ

 

P.S. I saw the Bayer errors. Siril didn't care, so I didn't either. 

Edited by BQ Octantis, 17 September 2022 - 07:24 AM.

[BQ Octantis]

I tried another all-Siril run without any calibration frames. The problem is in the demosaicing implementation in Siril. The color it assigns seems to be luminosity-scaled. This results in less color noise in the background, for sure. But the result is highly inaccurate. So I can no longer in good conscience recommend Siril for processing raws from an unmodded DSLR.

 

Below is what I was able to pull from a single 30 sec sub at 50% scale with

 

1. RawTherapee demosaicing and calibration
2. Siril color calibration and stretching
3. RawTherapee luminosity-scaled noise control

 

Final tweaks were in Photoshop; banding control was with Astronomy Tools v1.6 (which is more effective than Siril's strongest). Even the Running Chicken is visible in such scant data.

 

Preview (click for full size @ 50% sensor scale):

Uncropped version in the Google

 

Can someone with DSS give the data a go and post the stack? I'd love to compare it to RawTherapee…

 

BQ

Edited by BQ Octantis, 17 September 2022 - 08:00 AM.

[BQ Octantis]

I downloaded the source code for v1.0.5 and looked at the LMMSE algorithm to look for any sort of color scaling. And I found the funniest thing in the header:

 

/*
*  This file is part of RawTherapee.
*
*  Copyright © 2004-2010 Gabor Horvath <hgabor@rawtherapee.com>
*
*  RawTherapee is free software: you can redistribute it and/or modify
*  it under the terms of the GNU General Public License as published by
*  the Free Software Foundation, either version 3 of the License, or
*  (at your option) any later version.
*
*  RawTherapee is distributed in the hope that it will be useful,
*  but WITHOUT ANY WARRANTY; without even the implied warranty of
*  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
*  GNU General Public License for more details.
*
*  You should have received a copy of the GNU General Public License
*  along with RawTherapee.  If not, see <http://www.gnu.org/licenses/>.
*/

 

 

BQ

Edited by BQ Octantis, 17 September 2022 - 08:17 AM.

[BlueMoon]

 

I downloaded the source code for v1.0.5 and looked at the LMMSE algorithm to look for any sort of color scaling. And I found the funniest thing in the header:

Open Source. Why reinvent the wheel?

 

 

But the result is highly inaccurate. So I can no longer in good conscience recommend Siril for processing raws from an unmodded DSLR.

This is an important consideration for me as I image exclusively with unmodded Canon cameras and Siril is my go-to software. I sort of expected to see less color from an unmodded camera but this adds another aspect to be sure.

Perhaps it would be good to get the Siril devs involved in this thread and weigh in? I know at least one, cissou8, is a CN member. I'll message him and see if we can get some insight into this from their perspective.

 

Clear skies.

 

[EDIT] Message sent.

[EDIT] Message responded to.

Edited by BlueMoon, 17 September 2022 - 10:54 AM.

[galacticinsomnia]

I've been using siril for a couple of years now I think. 
I tink I just showed that it can reveal color, maybe not as much, but like I said before for 40 minutes of aquisition time using 30 second subs, on a wide field target, with unmodified camera, you are getting an honest display of color.

I can easily run your stack in DSS, Siril, APP, etc.  The results in APP are no better, and I'm sure in Pi they are no better.

Hope Siril will chime in here, but I don't know.

LOCK042 is a dev also, but not here that often, may want to post in Siril Forum for support.

Clear Skies !!