Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

IMX178 grid artifact bug mostly squashed

  • Please log in to reply
18 replies to this topic

#1 jwestervelt

jwestervelt

    Viking 1

  • *****
  • topic starter
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 22 March 2020 - 09:03 AM

After being told by QHY that the grid artifacts that plague users of the QHY5III178mono could not be fixed, I decided to take a stab at it myself.  To be fair, this is a problem of Sony's IMX178 sensor, and cameras from other brands have the same problem.  The issue is that the camera appears to have a Bayer mask, despite being mono.  The pixels that correspond to "red" on a color camera are overbiased and will appear brighter than surrounding pixels.  For reasons unknown, this seems to affect monochromatic sources with longer wavelengths more than other subjects, thus H.alpha users are particularly screwed when it comes to this camera. 

While undertaking a project to write an autoguider utility for linux for use with solar, lunar, and planetary imaging, I found myself needing to rip apart the .ser video files to grab individual frames to guide upon. For whatever reason, the images acquired this way were particularly bad with regard to the IMX178 bug, so I decided to take a crack at correcting the image.  It took all of 3 minutes.

Here is an example image before correction.   This was a shot, albeit upside-down, of my kitchen (the sink, and the faucet specifically), and you can clearly see the grid artifacts if you zoom in:  KCnaVKs.png

 

By doing a simple modulo to identify pixels that are in both even numbered rows and odd numered columns, a correction factor can be applied.

Here is the result:

MoDMlWQ.png

 

The correction isn't perfect as it is slightly non-linear, but that can be fixed with a little extra effort.  That's the good news.

The bad news... depending on the bias, these "red" pixels will lose a fair amount of the high end data.  For example, in the image shown here, the defective pixels were 1.74 times more luminous than they should have been.  This means that for a value exceeding around 150, the defective pixels record 255 in 8bit mode, and everything over 150 is clamped to 255. 

The not-so-bad news is that in most .ser files, the bias is much lower than this 1.74 value.  Additionally, since these "red" pixels are particularly hot, most people will end up reducing exposure to avoid blowing out the image.  This has the side-effect of preventing the clamping mentioned before, but it simultaneously makes the other 75% of the pixels darker than they need to be, resulting in a loss of dynamic range.  Regardless of how it's done in post processing, the camera will not be true 8bit unless the underlying issue is solved. 

Anyway, once I get some testing knocked out, I will release the utility.  I expect that it will be sometime in the next week or two, but it will likely remain a linux-only utility just like the autoguider unless someone wants to port it to Windows.  I'm going to finish up the autoguider first since that is more critical for me at the moment. 

Regarding the autoguider, it works by pointing the utility at a .ser file that is in the process of being written.  This will work with the imaging utility I currently use, but I don't know if other utilities (Ekos, FireCapture, etc) write the .ser file incrementally or not.  I would assume so, given the size.  It calculates the center of contour of the subject in the most recent frame, and then spits out an X,Y coordinate for the center.  It then works to keep the mount aimed such that the X,Y coordinate remains stationary with some filtering to smooth out atmospheric seeing effects and scope vibration. 


  • StephenW, eros312, gustavo_sanchez and 2 others like this

#2 gustavo_sanchez

gustavo_sanchez

    Apollo

  • *****
  • Moderators
  • Posts: 1,427
  • Joined: 30 Dec 2010
  • Loc: Puerto Rico, US

Posted 22 March 2020 - 10:48 AM

Yeah, I noticed that grid with my solar scope mainly. For some reason, increased magnification made the effect worse. Removing the barlow eliminated the grid effect... Go figure.

#3 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 22 March 2020 - 10:48 AM

Very nice work. I have written a tool to enhance live solar views by stacking ser files (done but not released yet...). I was amazed at how simple the ser file format is. Quite a nice surprise.



#4 jwestervelt

jwestervelt

    Viking 1

  • *****
  • topic starter
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 22 March 2020 - 05:17 PM

.SER is simple...  .FITS... ugh.   I still don't know if I understand that format. 

The issue I have is that i'm probably just going to use OpenCV to find the center, so I need to extract the frame of data into a format that OpenCV can use.  .FITS seemed to be the best fit, but I must be missing some crucial info on how these files are structured.  I'm just relying upon the library to do the heavy lifting for me, but when i need to debug the file with a hex editor, I get a bit confused with what I'm looking at. 

Right now my utility will work on 8bit .SER files just fine, but 16bit .SER files tend to produce in binarized output.  I tried swapping byte order to account for an endianness issue, but that wasn't it.  Oh well, it wouldn't be fun if problems were not encountered.  smile.gif



#5 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 22 March 2020 - 06:05 PM

Are you having trouble with reading the 16 bit SER? Because it is pretty simple. From upper left in image, across rows, read either 1 or 2 bytes per pixel. This is for mono files.  I don't currently deal with RGB files as they aren't used for solar imaging.

 

some C code from my app

 

// read the next frame
i = 0;
for (y=0; y<mHeader.height; y++)
{
  for (x=0; x<mHeader.width; x++ )
  {
   fread((void *)&mData[i++],1,mBytesPerPixel,mFh);
  }
}

 

mData[] is an array of unsigned longs

mBytesPerPixel is either 1 or 2

 

This reference was really helpful for me

http://www.grischa-h...SER Doc V3b.pdf



#6 jwestervelt

jwestervelt

    Viking 1

  • *****
  • topic starter
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 22 March 2020 - 06:34 PM

I can read the 16bit .SER just fine.  The issue is that I need to capture one of the frames to a .FITS file.  My code works just fine if the .SER is an 8bit, but using example code for cfitsio, it either segfaults or returns with an overflow condition when i try to write a 16bit .FITS file.  What's more retarded, the examples use a 'long' for the array, which is 32bit, and despite the .FITS clearly specifying that it is 16bit, when I view my data with a viewer, I get pixel values exceeding 16bit max, some something is wonky.  I banged my head on this for 8 hours last night and I need a break.



#7 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 22 March 2020 - 06:42 PM

I can read the 16bit .SER just fine.  The issue is that I need to capture one of the frames to a .FITS file.  My code works just fine if the .SER is an 8bit, but using example code for cfitsio, it either segfaults or returns with an overflow condition when i try to write a 16bit .FITS file.  What's more retarded, the examples use a 'long' for the array, which is 32bit, and despite the .FITS clearly specifying that it is 16bit, when I view my data with a viewer, I get pixel values exceeding 16bit max, some something is wonky.  I banged my head on this for 8 hours last night and I need a break.

Sorry, I misunderstood your problem. I know nothing about FITS other than I started to review the format and was really turned off by the complexity. Like a DICOM file - pure nonsense written by an anti-engineer to make them impossible to work with.



#8 jwestervelt

jwestervelt

    Viking 1

  • *****
  • topic starter
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 22 March 2020 - 07:07 PM

... I started to review the format and was really turned off by the complexity ... pure nonsense written by an anti-engineer to make them impossible to work with.

This, exactly.

I need to find a non-compressed, rasterized image format that doesn't take a PhD in rocket surgery to figure out... and one that OpenCV supports.  I'm about to give up on .FITS over this. 
 



#9 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 22 March 2020 - 07:14 PM

This, exactly.

I need to find a non-compressed, rasterized image format that doesn't take a PhD in rocket surgery to figure out... and one that OpenCV supports.  I'm about to give up on .FITS over this. 
 

Can you not just feed it an array of pixel values (either 1 or 2 bytes)?  I don't use OpenCV ( could never get it to compile) but the image display library I use allows me to feed each ser frame's data almost directly to it with very little or no manipulation.  

 

I think from a library standpoint, all image files formats should become the same once loaded into memory - just an array of pixel values of a specific bit depth and number of color channels.



#10 jwestervelt

jwestervelt

    Viking 1

  • *****
  • topic starter
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 22 March 2020 - 07:23 PM

Can you not just feed it an array of pixel values (either 1 or 2 bytes)?  I don't use OpenCV ( could never get it to compile) but the image display library I use allows me to feed each ser frame's data almost directly to it with very little or no manipulation.  

 

I think from a library standpoint, all image files formats should become the same once loaded into memory - just an array of pixel values of a specific bit depth and number of color channels.

Yes and no... because that array of data needs some context... you can't tell the image's dimensions unless you use a 2-dimensional array, which gets to be kinda messy to create since you'll need to be doing it in the heap instead of the stack due to size.  It's easier to manage a single-dimensional array, which is what cfitsio appears to be doing.



#11 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 22 March 2020 - 07:41 PM

Yes and no... because that array of data needs some context... you can't tell the image's dimensions unless you use a 2-dimensional array, which gets to be kinda messy to create since you'll need to be doing it in the heap instead of the stack due to size.  It's easier to manage a single-dimensional array, which is what cfitsio appears to be doing.

Exactly. For my library, I pass it a 1D array, a width in pixels and a number of bytes per pixel. So simple. I can't imagine OpenCV doesn't have a simple interface like that.



#12 jwestervelt

jwestervelt

    Viking 1

  • *****
  • topic starter
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 22 March 2020 - 08:33 PM

Maybe it does, I just haven't seen any mention of it yet.  I'm new to OpenCV so i'm just trying to wade through the documentation at the moment.  Either way, I need to make sure that i'm extracting the data in a meaningful way before trying to troubleshoot issues downstream, and writing to a standard file format and verifying with a viewer is a good way of doing that.  Maybe I'll just skip that step and hope to get lucky.  :D



#13 jwestervelt

jwestervelt

    Viking 1

  • *****
  • topic starter
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 22 March 2020 - 09:39 PM

Yeah.... I got it figured out.  My data parsing was fine... my FITS-fu needs work, but it isn't needed in this case so I'm going to skip it.  Thanks for the help.  :D



#14 jwestervelt

jwestervelt

    Viking 1

  • *****
  • topic starter
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 22 March 2020 - 11:10 PM

.ser : down

OpenCV : down

Next Up : indilib

I'll still need to figure out cfitsio for the image correction utility, but I'll do that after I get this limb tracker working.

 

:D

dTM1Ste.png



#15 jwestervelt

jwestervelt

    Viking 1

  • *****
  • topic starter
  • Posts: 748
  • Joined: 01 Sep 2012
  • Loc: Phoenix, Arizona

Posted 01 April 2020 - 04:27 AM

A small update...

First, regarding the solar tracker, I managed to get my code working, and it more than compensated for my poor polar alignment, keeping the sun dead center in the frame for about 260 minutes.

Second, and more importantly, I got my code working to "debayer" the IMX178 data.  The result was nothing short of night and day.  As I suspected, stacking software was aligning to the grid artifacts on a small scale and wrecking the resolution of the image.  How well did the code work?  Well, here's a side-by-side comparison, with the source .ser on the left and the corrected .ser on the right.  You might need to zoom in to see the difference:

5s1R9wk.png

 

 

As a result of this, stacking software such as Stackistry had a MUCH better alignment.  Same with AutoStakkert.  The detail was much better in the end.

Speaking of detail, here's the final output of my processing entirely within linux, from the corrected .ser data shown above.  I tried for several minutes to find any hint of the grid artifact and was unable to do so.

Due to the improved stacking, I am now able to see spicules easily in my image.  It is almost like I got a free upgrade to my hardware.  I can't wait to process a capture with better seeing conditions.


LMLZrOc.jpg

 


Edited by jwestervelt, 01 April 2020 - 04:31 AM.

  • Der_Pit and adios like this

#16 adios

adios

    Lift Off

  • -----
  • Posts: 24
  • Joined: 07 Jun 2019

Posted 04 September 2020 - 11:35 AM

Could you please share how to do it?



#17 FlankerOneTwo

FlankerOneTwo

    Viking 1

  • *****
  • Posts: 947
  • Joined: 30 Aug 2017
  • Loc: Vegas, baby!

Posted 05 September 2020 - 03:30 PM

Very interesting! Have you tried using AutoStakkert's Gaussian blur (under Experimental features)? It seems like perhaps that might effective against this. I would be interested in seeing a comparison of the results.



#18 descott12

descott12

    Gemini

  • *****
  • Posts: 3,262
  • Joined: 28 Sep 2018
  • Loc: Charlotte, NC

Posted 05 September 2020 - 03:45 PM

In an earlier post, I think I mis-spoke as I said I had seen this problem once or twice in about 18  months. But that was incorrect, I like some others, have not had this problem with the 178MM so I wonder if some batches of sensors didn't get the odd bayer mask put on for some reason??

 

I had seen a grid pattern/pixelation that was due to my less-that-great-quality LCD monitor that sometimes had trouble with images at various zoom levels. It can often create a pixelated look that goes away if I increase or decrease the zoom.



#19 FlankerOneTwo

FlankerOneTwo

    Viking 1

  • *****
  • Posts: 947
  • Joined: 30 Aug 2017
  • Loc: Vegas, baby!

Posted 07 September 2020 - 10:36 AM

Very interesting! Have you tried using AutoStakkert's Gaussian blur (under Experimental features)? It seems like perhaps that might effective against this. I would be interested in seeing a comparison of the results.

Clarification - the blur might be effective against the alignment degradation from the grid pattern. You would need dithering or image drift during the sequence to eliminate the visual result.




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics