Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

ZWO ASI178MM Tests: Flats are Prudent

  • Please log in to reply
57 replies to this topic

#26 dhkaiser

dhkaiser

    Apollo

  • *****
  • Posts: 1,146
  • Joined: 23 Dec 2019

Posted 04 May 2021 - 01:36 PM

Late to the thread here.  I have used my ASI178MM since March of 2020.  I had one image with the grid pattern early on, but not since.  Can't find it, must have tossed it.  I speculate that I have avoided the problem due to intentional poor tracking.  I let the image slowly drift during video capture.  Seems to work, have not had a problem in a long time.  Strictly speculation on my part.

 

https://www.flickr.c...157713689506148


Edited by dhkaiser, 04 May 2021 - 01:43 PM.

  • hopskipson and MalVeauX like this

#27 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 04 May 2021 - 04:17 PM

Hey all,

I have a working windows utility that removes the grid pattern. It isn't ready for prime time and I need more ser files to test with but here is a screenshot of my one test example.

Thanks to Jason for getting me started on this.

Basically, I figure out what rows and columns the grid pattern is overlayed on, then I back-fill those pixels with the median value of the surrounding 4 pixels (N,S,E and W, not diagonals).

Pretty simple really.

 

The left side shows a raw frame from the original ser file and the right is a frame from the exported/repaired ser file.

While the results are good, the fact is that this may be destroying data even though the mean value is probably very close the actual value. Hard to say.

 

Jasons' previously described method of adjusting those pixels up or down by the difference in the means of each population of pixels is probably better and may preserve the original data a little better. I will try to experiment with that method as well.

 

Attached Thumbnails

  • Screen Shot 2021-05-04 at 5.06.25 PM.jpg

  • SgrB2, BinoGuy, AJamesB and 2 others like this

#28 TheCrimson_King

TheCrimson_King

    Explorer 1

  • -----
  • Posts: 92
  • Joined: 14 Nov 2020

Posted 04 May 2021 - 04:30 PM

A Windows utility would be huge! Let me check if I have some data I can share with grid artifacts.
  • MalVeauX likes this

#29 jwestervelt

jwestervelt

    Viking 1

  • *****
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 04 May 2021 - 06:03 PM

I tried to get some more data today, but i fought a losing battle between my camera and the trees to my west.  TL;DR is that sometimes i have to unplug/replug my camera a few times to get it to recognize properly.  Today "a few times" literally meant over 100 plug insertions... over an hour of wasted time.  It has always been cranky, but today it was especially so.  I should have made a warranty claim on it day one since i always had this issue.  Live and learn I guess.  If anyone else has this problem, try to get a warranty return, it isn't your cable or your computer, it's something whack within the QHY5III series.


  • AJamesB and MalVeauX like this

#30 SgrB2

SgrB2

    Viking 1

  • -----
  • topic starter
  • Posts: 753
  • Joined: 10 Sep 2007
  • Loc: Ellicott City, MD

Posted 04 May 2021 - 06:20 PM

Here's another view of Dave's results.  Good work, Dave!

Attached Thumbnails

  • Annotation 2021-05-04 191648.jpg

  • MalVeauX likes this

#31 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 04 May 2021 - 06:31 PM

Here's another view of Dave's results.  Good work, Dave!

That is a nice analysis tool. I should mention that the screenshot I posted has a lot of artifacts from the way it was captured, reduced and compressed into a jpeg. The raw image frame is alot cleaner.


  • AJamesB and MalVeauX like this

#32 jwestervelt

jwestervelt

    Viking 1

  • *****
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 04 May 2021 - 10:31 PM

I am processing my data now.  I found out two things today.

First, the grid pattern defect is more pronounced when the image is overexposed.  By keeping the top of the histogram between 90% and 95%, the problem is almost completely eliminated.
Second, overexposure isn't the only trigger.  Each time the camera is plugged in, it seems to exhibit a different bias.  If I analyze the data in realtime, I can continuously unplug/replug the camera until I get a good data set, and then make sure i don't oversaturate the camera during my capture session.

I will get more data tomorrow and put together some charts.


  • SgrB2, AJamesB and MalVeauX like this

#33 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 07 May 2021 - 04:22 PM

Hey all,
Any more grid data I can use to test this utility??

Thanks


  • SgrB2 likes this

#34 jwestervelt

jwestervelt

    Viking 1

  • *****
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 07 May 2021 - 06:33 PM

Hey all,
Any more grid data I can use to test this utility??

Thanks

I have some older data which was really bad... should give your utility a workout lol.gif


  • SgrB2 likes this

#35 SgrB2

SgrB2

    Viking 1

  • -----
  • topic starter
  • Posts: 753
  • Joined: 10 Sep 2007
  • Loc: Ellicott City, MD

Posted 07 May 2021 - 06:39 PM

Hi Dave!

 

The weather is the culprit here and I will send some solar data as soon as

the weather lets up but it doesn't look good for about a week.  In the meantime

please specify what size frames and numbers of frames in  SER files

that you would want as well as any preference as to ZWO gain settings or

any other camera parameters.

 

Cheers,

SgrB2



#36 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 07 May 2021 - 06:45 PM

Hi Dave!

 

The weather is the culprit here and I will send some solar data as soon as

the weather lets up but it doesn't look good for about a week.  In the meantime

please specify what size frames and numbers of frames in  SER files

that you would want as well as any preference as to ZWO gain settings or

any other camera parameters.

 

Cheers,

SgrB2

Smaller files for now but all the other parameters should not be restricted in any way as that is up to the imager to decide what to use and it wouldn't be appropriate for me to dictate that in any way..  The utility is not optimized for speed in any way and it is a little slow so having ser files with maybe 5-10 frames should be fine for testing.


  • SgrB2 likes this

#37 SgrB2

SgrB2

    Viking 1

  • -----
  • topic starter
  • Posts: 753
  • Joined: 10 Sep 2007
  • Loc: Ellicott City, MD

Posted 07 May 2021 - 06:57 PM

Dave,

 

I  can run off some flat SER files from indoor lighting tomorrow and send those to you

if that would help.  They will show the grid pattern.  For example:

 

ZWO Gain       dB          Vout/Vin

 

0                       0             1.000

50                     5             1.778

100                   10           3.162

 

In each case the frame size would be 1024 x 768 with 10 frames in a SER file.

 

Cheers,

SgrB2



#38 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 07 May 2021 - 07:12 PM

Dave,

 

I  can run off some flat SER files from indoor lighting tomorrow and send those to you

if that would help.  They will show the grid pattern.  For example:

 

ZWO Gain       dB          Vout/Vin

 

0                       0             1.000

50                     5             1.778

100                   10           3.162

 

In each case the frame size would be 1024 x 768 with 10 frames in a SER file.

 

Cheers,

SgrB2

As long as it has the grid pattern visible, then that will be helpful. Thanks


  • SgrB2 likes this

#39 SgrB2

SgrB2

    Viking 1

  • -----
  • topic starter
  • Posts: 753
  • Joined: 10 Sep 2007
  • Loc: Ellicott City, MD

Posted 07 May 2021 - 07:25 PM

As long as it has the grid pattern visible, then that will be helpful. Thanks

The grid pattern is always detectable by stretching the data.  If you

 want the pattern visible without stretching, I will try to find settings

 that will show it. 

 

 Cheers,

 SgrB2



#40 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 07 May 2021 - 07:58 PM

The grid pattern is always detectable by stretching the data.  If you

 want the pattern visible without stretching, I will try to find settings

 that will show it. 

 

 Cheers,

 SgrB2

It makes it much easier to test if I can see the grid when displaying and stepping thru the ser file. The ser viewer that I use does not allow for any stretching so it will be harder to test.



#41 jwestervelt

jwestervelt

    Viking 1

  • *****
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 07 May 2021 - 08:15 PM

oof, i thought i had more bad data, but i looked and i think i deleted it.  the files i just uploaded are clean.

i can generate more bad data, i know it can be done by using certain versions of the SDK under linux.



#42 Tom M

Tom M

    Messenger

  • *****
  • Posts: 448
  • Joined: 24 Apr 2005
  • Loc: Indiana

Posted 07 May 2021 - 10:36 PM

It makes it much easier to test if I can see the grid when displaying and stepping thru the ser file. The ser viewer that I use does not allow for any stretching so it will be harder to test.

I'm using this SER player on windows and it has the tools to do stretching and create/save clips.

 

https://sites.google...layer#downloads



#43 SgrB2

SgrB2

    Viking 1

  • -----
  • topic starter
  • Posts: 753
  • Joined: 10 Sep 2007
  • Loc: Ellicott City, MD

Posted 08 May 2021 - 07:46 AM

Dave,

 

I did some indoor testing with Gain = 300
and could not see any grid pattern on the SharpCap
preview and capture screen while the camera
was running. I even tried to use the gamma
feature for focusing to enhance the view even
tho the capture ignores that feature.


I used AS3! to stack 5, 25, and 125 images
and observed them in ImPPG initially with
its default setting and all three looked just
fine.  However, stretching these 3 cases
show that the grid pattern gets worse the
more stacking you do.  This confirms what
JWestervelt said in 2020: " As I suspected,
stacking software was aligning to the grid
artifacts on a small scale and wrecking the
resolution of the image."

 

I think it unlikely that anyone can provide
data that show a grid pattern in each SER
frame (without enhancement) if my ASI178 is any

indication.  Altho, when observing the sun itself

I was able on one occasion to see the pattern on

the preview screen when using the gamma

enhancement focus aid in SharpCap.

 

Cheers,
SgrB2


Edited by SgrB2, 08 May 2021 - 08:02 AM.


#44 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 08 May 2021 - 07:55 AM

Dave,

 

I did some indoor testing with Gain = 300
and could not see any grid pattern on the SharpCap
preview and capture screen while the camera
was running. I even tried to use the gamma
feature for focusing to enhance the view even
tho the capture ignores that feature.


I used AS3! to stack 5, 25, and 125 images
and observed them in ImPPG initially with
its default setting and all three looked just
fine.  However, stretching these 3 cases
show that the grid pattern gets worse the
more stacking you do.  This confirms what
JWestervelt said in 2020: " As I suspected,
stacking software was aligning to the grid
artifacts on a small scale and wrecking the
resolution of the image."

 

I think it unlikely that anyone can provide
data that show a grid pattern in each SER
frame if my ASI178 is any indication.  Altho, when
observing the sun itself I was able on one occasion

to see the pattern on the preview screen when using
the gamma enhancement focus aid in SharpCap.

 

Cheers,
SgrB2

Ah ok. Then send me what you have and I will work with that. I didn't realize it was so difficult to reproduce but now that I think about it, the few occasions that I did see it were after stacking.



#45 MalVeauX

MalVeauX

    Cosmos

  • *****
  • Posts: 9,134
  • Joined: 25 Feb 2016
  • Loc: Florida

Posted 08 May 2021 - 07:58 AM

Dave,

 

I did some indoor testing with Gain = 300
and could not see any grid pattern on the SharpCap
preview and capture screen while the camera
was running. I even tried to use the gamma
feature for focusing to enhance the view even
tho the capture ignores that feature.


I used AS3! to stack 5, 25, and 125 images
and observed them in ImPPG initially with
its default setting and all three looked just
fine.  However, stretching these 3 cases
show that the grid pattern gets worse the
more stacking you do.  This confirms what
JWestervelt said in 2020: " As I suspected,
stacking software was aligning to the grid
artifacts on a small scale and wrecking the
resolution of the image."

 

I think it unlikely that anyone can provide
data that show a grid pattern in each SER
frame if my ASI178 is any indication.  Altho, when
observing the sun itself I was able on one occasion

to see the pattern on the preview screen when using
the gamma enhancement focus aid in SharpCap.

 

Cheers,
SgrB2

I've seen the grid on both an IMX178 color and mono sensor (I had both) and IMX183 color and mono sensor (had both, still have one) in preview with sub-angstrom HA, real time, not even stacking. For whatever reason, the more narrow the bandpass, it more easily it becomes apparent. I can see it somewhat at 2.4A bandpass, but at 0.6A it's obvious to me. I got rid of the cameras due to this. I kept my IMX183 cooled-color camera, for other applications, but the mono IMX178 & IMX183 I sold off because of this issue. It's not unique to solar, so much as its an issue with really tight narrowband. And it seemed to be more and more obvious with really long focal-ratios with sub-angstrom bandpass filter systems. There's lots of anecdotal stuff out there about one person using it without the grid and others have the grid, but I don't thinks its variance of the sensors or controllers or cables, I think its variance of their filter systems and focal-ratios and how they processed it to either exacerbate the issue or not.

 

For example, if you do a full disc with an IMX178 mono and left the sun drift a bit during capture (just turn off solar rate and use standard side rail rate), you can pretty much stack out the grid as the features move pixel to pixel and gets realigned and so people with poorly aligned EQ or rotating AltAz during the day are probably experiencing drift due to poor alignment and so they don't see the grid thanks to the drift and how that results in stacking out the grid. But people with very tight alignment are likely not drifting and they get the grid right away from stacking. This is my hunch for the variability people are seeing.

 

At the end of the day, ZWO publically stated to not use their IMX183 sensor for solar (implying sub-angstrom HA use) for this very reason as they didn't know what it was but acknowledged it's a problem and not just user error. Most camera distributors are not going to say something like that for fear of backlash or sales issues, PR nightmare, etc. So I'm actually happy ZWO decided to acknowledge it publically instead of trying to blow it off like its the user's fault when its an issue with the sensor and it's Sony's fault.

 

Very best,



#46 SgrB2

SgrB2

    Viking 1

  • -----
  • topic starter
  • Posts: 753
  • Joined: 10 Sep 2007
  • Loc: Ellicott City, MD

Posted 08 May 2021 - 08:33 AM

Hi Marty,

 

Yes, the time I saw the grid pattern on the preview screen I

was using the Quark at power.

 

Cheers,

SgrB2



#47 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 08 May 2021 - 08:37 AM

I'm using this SER player on windows and it has the tools to do stretching and create/save clips.

 

https://sites.google...layer#downloads

Ah...That is the program I used but I never noticed the processing menu! 

Thanks



#48 SgrB2

SgrB2

    Viking 1

  • -----
  • topic starter
  • Posts: 753
  • Joined: 10 Sep 2007
  • Loc: Ellicott City, MD

Posted 11 May 2021 - 04:18 PM

 Today I managed to get a detailed image of the grid pattern of

 the ASI178 mono using a Quark with no tilter and the image

 below is a blow-up of one portion of it.  It looks like the grid

 is ~10 x ~10 pixels.  I also took a bias frame and did not see

 the grid pattern.  So the sensor must have light to generate

 the grid.

 

 Cheers,

 SgrB2

Attached Thumbnails

  • Grid Pattern.jpg


#49 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 11 May 2021 - 04:24 PM

 Today I managed to get a detailed image of the grid pattern of

 the ASI178 mono using a Quark with no tilter and the image

 below is a blow-up of one portion of it.  It looks like the grid

 is ~10 x ~10 pixels.  I also took a bias frame and did not see

 the grid pattern.  So the sensor must have light to generate

 the grid.

 

 Cheers,

 SgrB2

Can you provide a portion of the ser file? There appears to a be a one-pixel grid and a 10 pixel "super grid". I have not seen that before.


  • SgrB2 likes this

#50 jwestervelt

jwestervelt

    Viking 1

  • *****
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 11 May 2021 - 04:24 PM

The larger grid pattern is not the issue, and I have not seen this myself.  It does look like the underlying circuitry.. the "hot" pixels represent 1/4 of the available photosites, and the larger grid pattern seems to be an 8x8 arrangement of the hot pixels, which represents a total 16x16 array.  There are traces that run beneath the photo sensor layer and this is likely what you are seeing... very possibly these channels are cooler than the pixels in between and thus they don't collect as much thermal noise.  These are supposed to be "back-illuminated" sensors, so this shouldn't be a shadow of the circuitry.


Edited by jwestervelt, 11 May 2021 - 04:26 PM.



CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics