Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

ZWO ASI178MM Tests: Flats are Prudent

  • Please log in to reply
57 replies to this topic

#1 SgrB2

SgrB2

    Viking 1

  • -----
  • topic starter
  • Posts: 756
  • Joined: 10 Sep 2007
  • Loc: Ellicott City, MD

Posted 26 April 2021 - 02:09 PM

Observed AR12818 with a SW 80ED, ZWO tilter, Quark,
and ASI178MM to see if flats affect the grid pattern.

 
The first test used the Quark unpowered to be able
to see the photosphere.  The second test used the
Quark at power to see the chromosphere.

 

1st image:  Photosphere no flat (grid pattern seen)
2nd image:  Photosphere using flat
3rd image:  Chromosphere no flat(grid pattern seen)
4th image:  Chromosphere using flat

 

The exact same ImPPG settings were used to severly
stretch the 1st and 2nd images; similarly for the
3rd and 4th images. The photosphere shows the

grid pattern better due to less feature confusion than

the chromosphere.

 

Cheers,
SgrB2

Attached Thumbnails

  • 1st.jpg
  • 2nd.jpg


#2 SgrB2

SgrB2

    Viking 1

  • -----
  • topic starter
  • Posts: 756
  • Joined: 10 Sep 2007
  • Loc: Ellicott City, MD

Posted 26 April 2021 - 02:11 PM

3rd & 4th

Attached Thumbnails

  • 3rd.jpg
  • 4th.jpg

  • philmor56, hopskipson, BinoGuy and 3 others like this

#3 SgrB2

SgrB2

    Viking 1

  • -----
  • topic starter
  • Posts: 756
  • Joined: 10 Sep 2007
  • Loc: Ellicott City, MD

Posted 28 April 2021 - 06:11 AM

BTW, These images were taken with the ZWO driver 3.0.0.2 which was

generated in 2017.  Some may wonder if ZWO had taken care of the well-known

grid pattern problem since that time.  So yesterday I installed the latest

ZWO driver 3.16.0.0 which was generated in December 2020 and got the same

results with respect to grid pattern.  Additionally, the ASI178 mono being used

has been purchased in the last two weeks so it is unlikely that ZWO has fixed

this problem in the hardware.

 

Cheers,

SgrB2



#4 MalVeauX

MalVeauX

    Cosmos

  • *****
  • Posts: 9,137
  • Joined: 25 Feb 2016
  • Loc: Florida

Posted 28 April 2021 - 06:33 AM

Very interesting results, it seems to work out ok for your setup. It doesn't seem to do this for everyone though. So I wonder why that is. So far, there's just been no definitive response and ZWO and any other distributor, and Sony, have zero answers for why its showing up at all on a mono sensor in the first place, other than they recognize it's there and simply state not to use them for HA solar (I guess butt-coverage on their part).

 

I have reached out to this guy, as he has compelling info on this subject, as it would be great to compile his code into something that could be used to pre-treat files, but I've gotten no response.

 

https://www.cloudyni...ostly-squashed/

 

Very best,


  • SgrB2 likes this

#5 SgrB2

SgrB2

    Viking 1

  • -----
  • topic starter
  • Posts: 756
  • Joined: 10 Sep 2007
  • Loc: Ellicott City, MD

Posted 28 April 2021 - 07:17 AM

Yes, I also PM-ed jwestervelt on 10 April but got no response.  The

work he did was impressive.  Since I used the unpowered Quark

for the tests and got the grid pattern, the problem may be more

than just narrow-band HA.

 

Cheers,

SgrB2


Edited by SgrB2, 28 April 2021 - 07:32 AM.


#6 SgrB2

SgrB2

    Viking 1

  • -----
  • topic starter
  • Posts: 756
  • Joined: 10 Sep 2007
  • Loc: Ellicott City, MD

Posted 01 May 2021 - 10:41 AM

The image below is a heavily stretched white light flat taken with

a C9.25 using ND 5.0 Baader solar film and the ASI178 mono.  The

image has been blown up so that the grid pattern is easily seen.

Thus, the grid pattern is not a function of narrow-band Ha observations

and confirms that flats are prudent no matter what the wavelength and

bandwidth.

 

Cheers,

SgrB2

Attached Thumbnails

  • Stretched_WL_Flat_2021-05-01.jpg

  • hopskipson and mshedden like this

#7 jwestervelt

jwestervelt

    Viking 1

  • *****
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 01 May 2021 - 08:18 PM

Apology for the delay.

TL;DR, the sensor is shite.  It is a design flaw on Sony's behalf.  It "CAN" be fixed in firmware, but the amount of work required to do that borders on the ludicrous.  You can't calibrate this out with a flat, as the fault becomes more or less pronounced depending upon many factors, and you pretty much have to correct the image on a frame-by-frame basis.

The only positive of all of this is that you can, actually must, do your correction AFTER the data is captured from the sensor.  This means that previously-recorded data isn't complete garbage and can be corrected.

Yes, I effectively fixed this.

Yes my software solution is naive, lacks any optimization, runs in linux only, and can only support .ser files... but it works.

I still haven't released it publicly yet, mainly on account of the code not being in a state that I am happy with, and the fact that I have entirely too much other stuff going on at the moment.

The only advice I can give is that people avoid the IMX178 like the plague, and if you need a camera with a similar resolution, sensitivity, and pixel pitch you will likely need to wait until the sun runs out of hydrogen before someone brings one to market, at which point hydrogen alpha observation will probably be meaningless.


  • torsinadoc, MalVeauX and stryker66 like this

#8 MalVeauX

MalVeauX

    Cosmos

  • *****
  • Posts: 9,137
  • Joined: 25 Feb 2016
  • Loc: Florida

Posted 02 May 2021 - 06:54 AM

Apology for the delay.

TL;DR, the sensor is shite.  It is a design flaw on Sony's behalf.  It "CAN" be fixed in firmware, but the amount of work required to do that borders on the ludicrous.  You can't calibrate this out with a flat, as the fault becomes more or less pronounced depending upon many factors, and you pretty much have to correct the image on a frame-by-frame basis.

The only positive of all of this is that you can, actually must, do your correction AFTER the data is captured from the sensor.  This means that previously-recorded data isn't complete garbage and can be corrected.

Yes, I effectively fixed this.

Yes my software solution is naive, lacks any optimization, runs in linux only, and can only support .ser files... but it works.

I still haven't released it publicly yet, mainly on account of the code not being in a state that I am happy with, and the fact that I have entirely too much other stuff going on at the moment.

The only advice I can give is that people avoid the IMX178 like the plague, and if you need a camera with a similar resolution, sensitivity, and pixel pitch you will likely need to wait until the sun runs out of hydrogen before someone brings one to market, at which point hydrogen alpha observation will probably be meaningless.

If you change your mind, it would be pretty easy to spin up a distro of linux on VM ware and use your code to correct SER files for this.

 

Agree, at this point, avoid these sensors with such a flaw. But your solution is brilliant and could save a lot of trouble, if you care to share.

 

Very best,



#9 stryker66

stryker66

    Ranger 4

  • *****
  • Posts: 378
  • Joined: 31 Jul 2018

Posted 02 May 2021 - 08:11 AM

Apology for the delay.

TL;DR, the sensor is shite.  It is a design flaw on Sony's behalf.  It "CAN" be fixed in firmware, but the amount of work required to do that borders on the ludicrous.  You can't calibrate this out with a flat, as the fault becomes more or less pronounced depending upon many factors, and you pretty much have to correct the image on a frame-by-frame basis.

The only positive of all of this is that you can, actually must, do your correction AFTER the data is captured from the sensor.  This means that previously-recorded data isn't complete garbage and can be corrected.

Yes, I effectively fixed this.

Yes my software solution is naive, lacks any optimization, runs in linux only, and can only support .ser files... but it works.

I still haven't released it publicly yet, mainly on account of the code not being in a state that I am happy with, and the fact that I have entirely too much other stuff going on at the moment.

The only advice I can give is that people avoid the IMX178 like the plague, and if you need a camera with a similar resolution, sensitivity, and pixel pitch you will likely need to wait until the sun runs out of hydrogen before someone brings one to market, at which point hydrogen alpha observation will probably be meaningless.

I just order this ASI178MM but I only going to use it for FULL disc on my Lunt 60mm. Would this show up badly? Currently I use ASI174MM but the pixel is 5.86um which has pixelation after post but able to at least minimize it with drizzle. THanks



#10 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 02 May 2021 - 08:59 AM


Yes my software solution is naive, lacks any optimization, runs in linux only, and can only support .ser files... but it works.

I still haven't released it publicly yet, mainly on account of the code not being in a state that I am happy with, and the fact that I have entirely too much other stuff going on at the moment.
 

I have done a ton of work with .ser files so I am sure I could port this thing into a small windows utility. If you would like to share what you did. I probably don't even need source code - just a description of the problem and your algorithm.


  • SgrB2, chemman, BinoGuy and 1 other like this

#11 jwestervelt

jwestervelt

    Viking 1

  • *****
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 03 May 2021 - 01:33 AM

I have done a ton of work with .ser files so I am sure I could port this thing into a small windows utility. If you would like to share what you did. I probably don't even need source code - just a description of the problem and your algorithm.

Literally, all you do is take a modulo function to determine if it is an even or odd row and column...  you'll run into four pixel groups:

a. even row, even column
b. even row, odd column
c. odd row, even column
d. odd row, odd column

These translate to a bayer-mask sort of pixel layout.  You then find the mean for each group and apply a normalization value to bring them all to the same mean.  You'll likely need to also look at the peak values to make sure you don't oversaturate if you are going to boost upwards.  The easiest thing to do is find the group with the lowest mean, and then scale the other 3 downwards to match.

What you are going to find is that one group is significantly hotter than the other 3.  Two are going to be nearly equal, and one is going to be significantly low.  These correspond to red = hot, two green = medium, blue = low.  This is why I feel that there is some remnant of a bayer mask configuration in place.  It is too much of a coincidence to ignore.


  • Great Attractor and MalVeauX like this

#12 jwestervelt

jwestervelt

    Viking 1

  • *****
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 03 May 2021 - 01:52 AM

I will statically compile the linux binary and provide a copy here if anyone would like to try it out.  It has usage instructions via a -help flag which is auto-invoked if you do not provide the correct commandline arguments.

I would like to test it against data provided by someone other than myself.  If anyone has an 8bit, 16bit, or some other bitdepth .ser file which exhibits the grid pattern, feel free to put it online somewhere for download.  I don't need a lot of frames, you can truncate the .ser to something like 10 frames if you'd like to keep the uploaded data small.  If I get some data to validate against, I can release this some time during the day tomorrow (Monday).  Otherwise, I'll have to wait until I can grab some data from another camera on Thursday.

 


  • MalVeauX likes this

#13 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 03 May 2021 - 07:40 AM

Ok. That sounds pretty straightforward.  I will definitely need some test data with the grid pattern if anybody has some to share.



#14 jwestervelt

jwestervelt

    Viking 1

  • *****
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 03 May 2021 - 01:15 PM

I can provide you with a 16bit .ser for test purposes.

As for the method I'm using for correction, i'm identifying the two "green" channels, which will have similar average values, and I'm boosting the blue channel to match whilst attenuating the red channel so they are all normalized around the two green values.  If the original .ser has any overexposed pixels, you're going to be facing some potential challenges.

I ran into issues trying to statically link my binary.  It uses OpenCV, and for the life of me I couldn't get it to compile statically... i blame my inability to properly leverage threats to convince the linker to do my bidding.  I am going to just provide the github repository link here once i tidy up a couple of things.  I should have it done in the next 20-30min.


  • MalVeauX likes this

#15 jwestervelt

jwestervelt

    Viking 1

  • *****
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 03 May 2021 - 01:33 PM

OK, code is up.

https://github.com/zenmetsu/H-alfaSSed

 

Feel free to look over what I did.  I tried to comment it a bit.  I'm not responsible if this code eats your dog, opens a gateway to hell, causes earthquakes, or causes the universe to undergo vacuum decay.

Project needs openCV and cmake.  Instructions are in the INSTALL file and it should be fairly straight forward.

Eventually I'll add other functions for manipulating the .ser file, for instance changing between bit depths, changing header data, truncating a .ser to a range of frames (useful for timelapse, cropping out bad frames, etc etc), and so on.


Edited by jwestervelt, 03 May 2021 - 01:36 PM.

  • R Botero, Great Attractor and MalVeauX like this

#16 jwestervelt

jwestervelt

    Viking 1

  • *****
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 03 May 2021 - 01:56 PM

As called out in the repo's README, there is still some residual error in the red channel.  I added an animation showing a zoomed in region of the before/after images. In brighter areas, there is an overcorrection and the pixels are too dark.  In the darker regions, there is an undercorrection and the pixels are too bright.  From my experience, stacking will blow away this residual error.

Here is the image from the repo showing what I am talking about.

comparison.gif


  • MalVeauX likes this

#17 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 03 May 2021 - 02:05 PM

I can provide you with a 16bit .ser for test purposes.

As for the method I'm using for correction, i'm identifying the two "green" channels, which will have similar average values, and I'm boosting the blue channel to match whilst attenuating the red channel so they are all normalized around the two green values.  If the original .ser has any overexposed pixels, you're going to be facing some potential challenges.

I ran into issues trying to statically link my binary.  It uses OpenCV, and for the life of me I couldn't get it to compile statically... i blame my inability to properly leverage threats to convince the linker to do my bidding.  I am going to just provide the github repository link here once i tidy up a couple of things.  I should have it done in the next 20-30min.

Yep, I tried to use OpenCV for a few things and found compiling and linking so difficult that I gave up and rolled my own algorithms for various things. Maybe not the best way to do things but I definitely learned alot about how all this is done and I like understanding the low-level stuff anyway.

Regarding the .ser file: I have one already but another would be great if you can provide that somehow.


  • MalVeauX likes this

#18 jwestervelt

jwestervelt

    Viking 1

  • *****
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 03 May 2021 - 02:29 PM

I have a 20 frame 16bit .ser, it is about 120MB compressed. 

 

https://gofile.io/d/2AnVyI

 

Password is CN2021


  • MalVeauX likes this

#19 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 03 May 2021 - 02:43 PM

I have a 20 frame 16bit .ser, it is about 120MB compressed. 

 

https://gofile.io/d/2AnVyI

 

Password is CN2021

Got it. Thanks


  • MalVeauX likes this

#20 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 03 May 2021 - 02:46 PM

I have a 20 frame 16bit .ser, it is about 120MB compressed. 

 

https://gofile.io/d/2AnVyI

 

Password is CN2021

Can you compress with something besides xz? I don't have access to linux box right now


  • MalVeauX likes this

#21 jwestervelt

jwestervelt

    Viking 1

  • *****
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 03 May 2021 - 02:54 PM

I just found out that the residual error shown in the .gif above is due to clipped pixels.  Many of the "red" channel pixels were at max value (65535).  As such, it is not possible to correct this after capture, and more care is needed to ensure that exposure is set correctly at time of capture.  Either way, the utility fixes the problem sufficiently enough to extract more detail during image stacking.


  • MalVeauX likes this

#22 jwestervelt

jwestervelt

    Viking 1

  • *****
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 03 May 2021 - 03:02 PM

xz is kinda force of habit since it offers better compression.

Here's the .gz
https://gofile.io/d/KzhEOO

 

Same password


  • MalVeauX likes this

#23 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 03 May 2021 - 03:30 PM

Thanks. Got it


  • MalVeauX likes this

#24 jwestervelt

jwestervelt

    Viking 1

  • *****
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 03 May 2021 - 04:22 PM

that file has overexposed pixels, so it can provide you with something for that use case.  i'll go through my other data to find something a bit cleaner.  it was from the same .ser that produced this image, so it isn't completely worthless.

LMLZrOc.jpg


  • MalVeauX likes this

#25 jwestervelt

jwestervelt

    Viking 1

  • *****
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 04 May 2021 - 01:17 PM

I may have made a breakthrough on this issue.  I'll need to collect more video to verify.  There might be a way to prevent this after all...


  • SgrB2 and MalVeauX like this


CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics