15 replies to this topic

[T~Stew]

I've finally come full circle after about a year... revising my very first target M42. My processing skills are pretty good now, and the latest version of Siril has been great to me. But for some reason now its going haywire. I'll show my uncallibrated sub, very low vignetting even with a full frame 6D with Canon 400mm f5.6L lens.

 

I took full suite of callibration files, recently re-did my biases, and also took my flats and darks at the end of the session. Flats on the led flat panel that also has a couple layers of ND sheets on it and been working well for previous projects. 30 flats, 40 darks taken. 

 

For some reason I get severe reverse vignette, and that streak that I guess I need to clean of the sensor gets much brighter too, along with a couple other dust motes. Uncalibrated lights would be easier to process at this point, but where the heck did I go wrong?

 

 

 

 

 

Just for clarification, darks and biases stacked median, no normalization. Flats calibrated with master bias, then stacked medium - multiplicative. Lights calibrated as shown above.

 

Am I missing something simple here? I'm stumped

Edited by T~Stew, 27 January 2022 - 03:26 PM.

[dx_ron]

Double-check (in the FITS header) that you didn't accidentally set either the gain or the offset incorrectly when you took the flats. I did that once, and it drove me nuts until it was explained to me.

(or ISO, I guess, if this is dslr)

Edited by dx_ron, 27 January 2022 - 03:51 PM.

[pyrasanth]

I don't think the problem is the software. It is only working with the input you provide it with.

 

I think a close inspection of your data would be a good place to begin.

[larryjh]

You could take a look at the histogram of your light frames and see if it is getting cut off on the left side.  If so, I think the flat will over correct because pixels that are zero (or at the read noise level of the camera) can't get any lower and will therefore be over corrected.  I had this problem with my ASI294MC until I put an offset in to move the histogram to the right.  I'm not sure what the equivalent thing is with a DSLR.  I think another way of looking at it is that your background sky noise must be above your read noise.  You can see what your read noise level is in your bias frame.  If the histogram of your light frame looks a lot like your bias frame, you need more signal.

[KlausKlaus]

Does the same happen if you use SiriL‘s built-in scripts?

[T~Stew]

You could take a look at the histogram of your light frames and see if it is getting cut off on the left side.  If so, I think the flat will over correct because pixels that are zero (or at the read noise level of the camera) can't get any lower and will therefore be over corrected.  I had this problem with my ASI294MC until I put an offset in to move the histogram to the right.  I'm not sure what the equivalent thing is with a DSLR.  I think another way of looking at it is that your background sky noise must be above your read noise.  You can see what your read noise level is in your bias frame.  If the histogram of your light frame looks a lot like your bias frame, you need more signal.

There is plenty of gap on the left of the histogram. Its not a problem of signal strength, I use the same method on M42 as I have done on every image, shoot for 1200 ADU above bias level. If anything M42 should've been imaged with a bit less. My bias runs ~8200 ADU for almost every Canon I have and I shoot for 9400 ADU on the lights. 60" put me at 9732, it was close enough but I probably should've tried ~45" shots.

 

 

Does the same happen if you use SiriL‘s built-in scripts?

I just tried using the script, haven't done that in a long time, but the result was the same.

 

 

I don't think the problem is the software. It is only working with the input you provide it with.

 

I think a close inspection of your data would be a good place to begin.

I just used Siril on the Horsehead which I shot last night, had no problems with calibration. So I am sure its working. However there are some things I don't know if it necessarily does a good job with, like what is the function of the Auto Evaluate Normalization Value? Usually I prefer to do everything manually, but I have always left that on auto. 

 

 

Double-check (in the FITS header) that you didn't accidentally set either the gain or the offset incorrectly when you took the flats. I did that once, and it drove me nuts until it was explained to me.

(or ISO, I guess, if this is dslr)

I did swap to a different camera after M42, went for my 7DmkII to shoot the Leo Triplet. So I just double checked to make sure I had the correct flats, checked the fits header its all good, Right camera, ISO, etc.

SIMPLE  =                    T / file does conform to FITS standard
BITPIX  =                   16 / number of bits per data pixel
NAXIS   =                    2 / number of data axes
NAXIS1  =                 5496 / length of data axis 1
NAXIS2  =                 3670 / length of data axis 2
EXTEND  =                    T / FITS dataset may contain extensions
COMMENT   FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT   and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H
BZERO   =                32768 / offset data range to that of unsigned short
BSCALE  =                    1 / default scaling factor
CREATOR = 'ZWO ASIAIR'         / Capture software
XORGSUBF=                    0 / Subframe X position in binned pixels
YORGSUBF=                    0 / Subframe Y position in binned pixels
FOCALLEN=                  391 / Focal length of telescope in mm
XBINNING=                    1 / Camera X Bin
YBINNING=                    1 / Camera Y Bin
CCDXBIN =                    1 / Camera X Bin
CCDYBIN =                    1 / Camera Y Bin
XPIXSZ  =     6.53999996185303 / pixel size in microns (with binning)
YPIXSZ  =     6.53999996185303 / pixel size in microns (with binning)
IMAGETYP= 'Flat    '           / Type of image
EXPOSURE=      0.1249990016222 / Exposure time in seconds
EXPTIME =      0.1249990016222 / Exposure time in seconds
CCD-TEMP=                 -13. / sensor temperature in C
DATE-OBS= '2022-01-27T04:12:29.912059' / Image created time
INSTRUME= 'Canon EOS 6D'       / Camera model
BAYERPAT= 'RGGB    '           / Bayer pattern
ISO     = '1600    '           / ISO Value
TELESCOP= 'On Camera'          / Telescope name
END 

I use the autoflat on ASIair, it usually puts it right around 32000 ADU. I can see the same dust motes in the lights and flats, so I guess I know they were the matched ones, just for some reason its really over correcting.

Edited by T~Stew, 27 January 2022 - 10:49 PM.

[larryjh]

I don't know if this would make any difference for your camera, but you could try calibrating your flat with a dark flat (same exposure/ISO as your flat) instead of a bias.

 

If you want to post a link to a light sub, and your calibration images, I would be interested in taking a look.  I am interested in your problem because I also use Siril and I occasionally have problems with calibration also.

[dx_ron]

However there are some things I don't know if it necessarily does a good job with, like what is the function of the Auto Evaluate Normalization Value? Usually I prefer to do everything manually, but I have always left that on auto. 

 

 

I did swap to a different camera after M42, went for my 7DmkII to shoot the Leo Triplet. So I just double checked to make sure I had the correct flats, checked the fits header its all good, Right camera, ISO, etc.

That is from the flats. Be sure to also check that the lights do have the same iso. It's not that I am convinced that is your problem - it is just that a mismatch between flats iso and lights iso will produce exactly the same kind of overcorrection that you see.

 

The Auto Evaluate Normalization value is, as i understand it, matching the levels in the 3 color channels in the flats. I always leave it on.

[T~Stew]

Ok, sorry I didn't include the actual files in the original... I never use google so I got to figure it out each time I need to upload something like that lol.

 

Here is a light sub  https://drive.google...iew?usp=sharing

 

Here is a flat sub  https://drive.google...iew?usp=sharing

 

Here are my darks, stacked   https://drive.google...iew?usp=sharing

 

I took my darks just with the camera an intervalometer, so they are .CR2 and when I converted and stacked it looks like I didn't save the individual fits. I could run one through conversion again if important, but checking the raw file everything checks (camera, iso, etc - time was reported a tiny bit off at 59.4" but close enough).

 

That is from the flats. Be sure to also check that the lights do have the same iso. It's not that I am convinced that is your problem - it is just that a mismatch between flats iso and lights iso will produce exactly the same kind of overcorrection that you see.

 

I compared the lights and flats file header side by side, lines 1-29 are identical except of course exposure time, timestamp, and 1 degree different in temperature.

Edited by T~Stew, 28 January 2022 - 10:15 AM.

[larryjh]

I took a look at your files and here is what I see.

 

Here is a plot of the average of the first 10 rows of your flat red channel:

 

 

So it looks like about a 90% drop from the peak, which seems pretty reasonable.  Here is a plot of an average of the same 10 rows of your light sub:

 

 

So, just from these two plots it seems clear that your flat isn't going to work.  Since your light sub shows almost no drop from center to edge, I'm back to thinking, in spite of what you said, that there is insufficient background signal.  Is it possible that you need to move your median above the dark rather than the bias?  Unfortunately, I couldn't analyze your dark frame because it was converted to 32 bit and I don't know how to convert it back to do a direct comparison with your light and flat.  With your camera, would you expect a 60s dark to have significantly higher median than your bias?

 

I don't know how much any of this helps, but I would at least conclude that the problem does not lie with the Siril software.

[cissou8]

Hi,

 

I think I found the culprit, which is..... your masterdark. You said smthg interesting at some point:

"My bias runs ~8200 ADU for almost every Canon"

 

Well, if I open your dark and run a stat command, the mean level is 2048.... so looks very much like one fourth of this value.

So I used the following commands with the files you've shared:

 

# upscale the dark by a factor 4

load darks6Dminus5-60sec_stacked.fit
fmul 4.0
save upscaled_masterdark.fit

# "zero" the flat by its offset
load Flat_125.0ms_Bin1_0001.fit
offset -8192
save masterflat.fit

 

I then went ahead and calibrated your light with the scaled masterdark and this "masterflat" (well just one flat in there, so a very noisy masterflat). The debayered calibrated light is just left with what looks like LP gradient (sorry for the resolution, I had to compress a lot):

So it seems that, for some reason, the masterdark needs this multiplication. The factor 4 looks very much like a 14b to 16b conversion thing.... Did you shoot the darks to assemble your masterdark with the AsiAir? Maybe that is smthg to look into. My advice would be to keep the same imaging software/device for all the shots, lights and calibration frames. I have had surprises in the past mixing them.

 

Cheers,

 

Cecile

Edited by cissou8, 28 January 2022 - 08:00 PM.

[T~Stew]

Larryjh thanks for a taking a look.

 

If it helps I went ahead and converted one of the darks to fits, anything to try to get to the bottom of this.

 

https://drive.google...iew?usp=sharing

[larryjh]

I just took a look at your dark and I agree with Cecile's analysis.  The raw fits file level is around 2050.  I hadn't considered the fact that subtracting out the dark will make the center to edge ratio bigger.  If the dark were around 8200 instead of 2050, you'll get your roughly 90% center to edge on your light just like in your flat.  Cecile's processed image shows that better than any words I could provide!

[T~Stew]

Hi,

 

I think I found the culprit, which is..... your masterdark. You said smthg interesting at some point:

"My bias runs ~8200 ADU for almost every Canon"

 

Well, if I open your dark and run a stat command, the mean level is 2048.... so looks very much like one fourth of this value.

So I used the following commands with the files you've shared:

 

# upscale the dark by a factor 4

load darks6Dminus5-60sec_stacked.fit
fmul 4.0
save upscaled_masterdark.fit

# "zero" the flat by its offset
load Flat_125.0ms_Bin1_0001.fit
offset -8192
save masterflat.fit

 

I then went ahead and calibrated your light with the scaled masterdark and this "masterflat" (well just one flat in there, so a very noisy masterflat). The debayered calibrated light is just left with what looks like LP gradient (sorry for the resolution, I had to compress a lot):

Untitled.jpg

So it seems that, for some reason, the masterdark needs this multiplication. The factor 4 looks very much like a 14b to 16b conversion thing.... Did you shoot the darks to assemble your masterdark with the AsiAir? Maybe that is smthg to look into. My advice would be to keep the same imaging software/device for all the shots, lights and calibration frames. I have had surprises in the past mixing them.

 

Cheers,

 

Cecile

Oh maybe you're on to it... looks like those dust motes are gone! No I did not use the ASIair for the darks as I was continuing to image that night (with a different camera), I just put the lens port cover on the camera and plugged intervalometer into it to shoot the darks, saving the raws on the camera memory card. So you're saying my .CR2 out of the camera is substantially different than if the ASIair captured a .fit with it? It is about the only unusual thing I did differently that night... usually I do capture darks with the ASIair. Actually now I think about it I believe most all my bias frames were made just with camera and intervalometer too. Maybe I'll try reshooting both of those using the ASIair and see if the calibration matches up, tonights temperature might actually be close enough to redo the darks.

[T~Stew]

Thanks for the help Larry and Cecile! That was it for sure. I re made bias and darks and restacked it all and it worked as expected. Its so nice to finally have an answer to this issue that somewhat randomly popped up from time to time. Now I know to always use same acquisition software for all files.

 

 

Now I have an image to start working on...

[cissou8]

So you're saying my .CR2 out of the camera is substantially different than if the ASIair captured a .fit with it?

Indeed, the root cause is that there are two ways to write in memory 14b of information in a 16b container, which is writing at the left and pad with zeros the trailing 2b or the reverse (zero-padding the leading 2b). When you compare the 16b actual values written with both conventions, you find a multiplying factor of 4 (2b)...
And you can be sure that when there is more than one way, each software or driver editor will make their own choice, which is the exact opposite from the other software you use. Hence the advice. Same goes for the choice of writing the fits top down or up bottom. So, yes definitely, only one software or means of acquisition and you'll keep away from convention errors

If you need to "debug" that kind of problems, we have a tutorial which shows how the dark and offset levels influence over or under correction from the flats: https://siril.org/tu...rect-the-lights
There is a plot in particular which shows that inverse vignetting comes most times from under correction from the dark.

Anyway, good that it's all sorted out now!

Clear skies

C.