240 replies to this topic

[NGC3031]

This is a very nice software for creating star masks indeed. Thanks for sharing, this will probably become my main tool for creating star masks
For anyone wondering how to create the star mask: In Pixelmath, subtract the starless from the original version and extract the grayscale image.

 

[jdj]

Hi Nikita -

 

Thanks for making this available to the community, it looks like an amazing tool.  Unfortunately, the dropbox link for the pre-compiled version is no longer working (something about too many requests, a good sign I suppose!). If you could post the pre-compiled version elsewhere that would be fantastic.

 

Thank you!

John

[nekitmm]

Hi Nikita -

 

Thanks for making this available to the community, it looks like an amazing tool.  Unfortunately, the dropbox link for the pre-compiled version is no longer working (something about too many requests, a good sign I suppose!). If you could post the pre-compiled version elsewhere that would be fantastic.

 

Thank you!

John

I am glad it is helpful!

 

The DB link indeed was banned! I moved the program to SourceForge, here is the link:

 

https://sourceforge....ojects/starnet/

 

I didn't find the way to edit the original post though....

[jdj]

Thanks much for posting the new link Nikita.  I tried out your program and it works like a charm!

 

Here are original and starless versions of a recent Rosette Nebula image of mine:

 

 

Congrats on a fine piece of software and thanks again for making this available to the community!!

 

John

[Stevous]

Dear Nikita,

 

I am adding my deepest thanks to you as well, this line command is simply outstanding.

 

Best wishes,

 

Steve

[nekitmm]

Dear Nikita,

 

I am adding my deepest thanks to you as well, this line command is simply outstanding.

 

Best wishes,

 

Steve

Thanks, Steve!

 

I am glad that you like it!

[pfile]

came across this on instagram this morning. got the python version running on OSX relatively easily... i don't have a 64-bit wine environment to run starnet++ but if i understand it right the python version is equivalent. with 12 threads it processes images from an 8300M in 1-2 minutes, which is totally fine... and the results are incredible. easily the best star removal software i've ever used, and trivially easy to run! thanks nikita.

 

p.s. ever thought about training a GAN like this for noise reduction? i know its being done but i wonder if astronomical images need a different training set.

[pfile]

wow. i'm totally blown away, check out what starnet did here. just took me about 10 minutes of very simple cleanup with clonestamp + some noise reduction:

 

 

you can blink back and forth between the original and the starless and see just how clean it is.

 

rob

[nekitmm]

wow. i'm totally blown away, check out what starnet did here. just took me about 10 minutes of very simple cleanup with clonestamp + some noise reduction:

 

 

you can blink back and forth between the original and the starless and see just how clean it is.

 

rob

Thanks for the kind words, Rob! I am glad that it worked for you! The result looks good!

 

 

p.s. ever thought about training a GAN like this for noise reduction? i know its being done but i wonder if astronomical images need a different training set.

That actually might be my next direction to investigate! Would be interesting to see what GANs can achieve here!

[MattC867]

This is really amazing, thanks!

I am looking to train the model on some of my own data, but I can't find the ./log folder that the program is looking for. Keep getting an error. Same issue trying to run the plot sub function - what am I doing wrong?

Edited by MattC867, 17 March 2019 - 11:03 PM.

[nekitmm]

This is really amazing, thanks!

I am looking to train the model on some of my own data, but I can't find the ./log folder that the program is looking for. Keep getting an error. Same issue trying to run the plot sub function - what am I doing wrong?

Hi, Matt!

 

Sorry about that! I added the missing folder into git repo! It should work just fine now!

 

Let me know if you will have other problems! Also, if possible, please let me know results of your experiments!

 

Thank you for kind words!

[airscottdenning]

I cloned the python version from git and it works very well for me on a recent mac.

 

Tried it out on an image of the Rosette that I shot with one-shot color on a 488 mm refractor. Results are interesting: stars are gone, but lots of artifacts are left behind. 

 

I understand this would be better if I trained the program on my own image. Do you have any suggestions about how to better clean it up for training purposes?

 

Here's a link to a jpg of the starless image (small preview attached).

 

Edited by airscottdenning, 18 March 2019 - 11:26 PM.

[nekitmm]

I cloned the python version from git and it works very well for me on a recent mac.

 

Tried it out on an image of the Rosette that I shot with one-shot color on a 488 mm refractor. Results are interesting: stars are gone, but lots of artifacts are left behind. 

 

I understand this would be better if I trained the program on my own image. Do you have any suggestions about how to better clean it up for training purposes?

 

Here's a link to a jpg of the starless image (small preview attached).

 

Presentation2.jpg

Hi!

 

It would be nice to also see full size of original image! I think that possible cause of all these artifacts is that original image was heavily processed and star profiles became too distorted.

 

In most cases you will get significantly better results if you try on a less processed version of the image. Can you try that before getting into all the troubles of training?

Edited by nekitmm, 18 March 2019 - 11:36 PM.

[airscottdenning]

Nikita,

 

Thank you for the suggestion. I redid my Rosette image with just a light stretch and re-ran StarNet transform (using your weights) with MUCH better results!

 

Here's a Dropbox link to the "before" image: gentle.stretch.tif

and here's a Dropbox link to the "after" image: gentle.stretch.tif_starless.tif

 

Your software is WONDERFUL!

 

Would you consider modifying it to allow higher bit depths of the "before" image?  It seems that I am losing some dynamic range by having to save my image as 8 bits per channel before calculating doing the star removal.

 

At any rate, thank you VERY MUCH for your hard work on this, and for making this amazing tool available for our community!

 

Have you shared this in the PixInsight forums?

 

Best regards,

Scott Denning

 

 

Hi!

 

It would be nice to also see full size of original image! I think that possible cause of all these artifacts is that original image was heavily processed and star profiles became too distorted.

 

In most cases you will get significantly better results if you try on a less processed version of the image. Can you try that before getting into all the troubles of training?

[pfile]

+1 higher bit depths, but looking at the code it seems like internally the images might be internally represented by 32b floats already?

 

transform.py:        input = np.array(img.open(image), dtype = np.float32)

 

so is it just a matter of using a different library to read/write the file?

 

also, another question - how many images did you have to use to train the network to get satisfactory results?

 

rob

Edited by pfile, 19 March 2019 - 04:12 PM.

[nekitmm]

Nikita,

 

Thank you for the suggestion. I redid my Rosette image with just a light stretch and re-ran StarNet transform (using your weights) with MUCH better results!

 

Here's a Dropbox link to the "before" image: gentle.stretch.tif

and here's a Dropbox link to the "after" image: gentle.stretch.tif_starless.tif

 

Your software is WONDERFUL!

 

Would you consider modifying it to allow higher bit depths of the "before" image?  It seems that I am losing some dynamic range by having to save my image as 8 bits per channel before calculating doing the star removal.

 

At any rate, thank you VERY MUCH for your hard work on this, and for making this amazing tool available for our community!

 

Have you shared this in the PixInsight forums?

 

Best regards,

Scott Denning

 

 

+1 higher bit depths, but looking at the code it seems like internally the images might be internally represented by 32b floats already?

Lol, I also vote for higher bit depths!)

 

I am not sure right now, but when I was developing the code, something didn't let me do that. I think PIL image module that I use to read and write files does not support reading (or writing?) 16 bit files. I don't remember exactly the cause right now. Internally it is not a problem, as you noted.

 

Anyway, I will look into this a bit later. Right now I am working on a new release of binary files for both Windows and Mac. You will be able to work with 16bit files using those! Stay tuned!

 

Thanks for kind words!

[nekitmm]

Hi, everyone!

 

I just released new version of starnet! See here for downloads: https://sourceforge....ojects/starnet/

 

For some additional description please see https://www.astrobin.com/397290/. Don't want to repeat all the text here!

 

There are two major changes in this release:

 

1. There are now two executables: for Grayscale images and for RGB images. Both still work with 16bit per channel TIF images. Previous release was created with only color images in mind and performance on Grayscale images was often rather poor. This should not be the case anymore.

 

2. Both Windows and MacOS are now supported (make sure you download proper version).

 

Hope it will work well for you! All feedback is welcome (good or bad)!

 

Thanks!

Edited by nekitmm, 24 March 2019 - 11:02 PM.

[pfile]

seems like there might be some problems with the OSX build.

 

1) dyld complains that libtiff.5.dyld is not found in @rpath

     - this seems to be fixed by making a symlink from libtiff.5.dyld to the provided libtiff.dyld .

     - but i suspect installing libtiff from homebrew might also fix this.

 

2) dyld then complains:

    dyld: Library not loaded: /usr/local/opt/jpeg/lib/libjpeg.9.dylib

 

installing libjpeg from brew solves that.

 

then 3) dyld complains:

 

dyld: Library not loaded: /usr/local/opt/webp/lib/libwebp.7.dylib

 

again installing webp from brew solves that.

 

so in short i think these binaries were built assuming that the user has HomeBrew and has those liibraries installed.

 

rob

[nekitmm]

seems like there might be some problems with the OSX build.

 

1) dyld complains that libtiff.5.dyld is not found in @rpath

     - this seems to be fixed by making a symlink from libtiff.5.dyld to the provided libtiff.dyld .

     - but i suspect installing libtiff from homebrew might also fix this.

 

2) dyld then complains:

    dyld: Library not loaded: /usr/local/opt/jpeg/lib/libjpeg.9.dylib

 

installing libjpeg from brew solves that.

 

then 3) dyld complains:

 

dyld: Library not loaded: /usr/local/opt/webp/lib/libwebp.7.dylib

 

again installing webp from brew solves that.

 

so in short i think these binaries were built assuming that the user has HomeBrew and has those liibraries installed.

 

rob

Thanks for the info. That's pretty weird though, the build was tested on a few MacOS machines and there were no problems running it without installing anything...

 

Why does it try to load jpeg libs? That's weird. Is your input TIF file compressed?

 

I will try to look into it!

 

Thanks!

Edited by nekitmm, 24 March 2019 - 11:55 PM.

[pfile]

dyld: Library not loaded: /usr/local/opt/webp/lib/libwebp.7.dylib
  Referenced from: /Users/pfile/Downloads/StarNet_MacOS/libtiff.dylib

 

dyld: Library not loaded: /usr/local/opt/jpeg/lib/libjpeg.9.dylib
  Referenced from: /Users/pfile/Downloads/StarNet_MacOS/libtiff.dylib

 

so they would appear to be dependencies of libtiff for whatever reason...

 

/usr/local/ seems to be where homebrew installs all its stuff. i don't think there's anything in that directory on a stock OSX installation but i'm not sure.

 

rob

[airscottdenning]

I get the exact same errors when I try to run this on my mac(s).

[pfile]

if you install homebrew and then install webp and libjpeg (as well as making the symbolic link i described above) you can get around the problem. then again i guess the point of starnet++ is so that users don't have to mess around with installing stuff from Fink, MacPorts or Homebrew as you do with the python version.

 

rob

[nekitmm]

I get the exact same errors when I try to run this on my mac(s).

Were you able to follow instructions from Rob and get it running?

[nekitmm]

if you install homebrew and then install webp and libjpeg (as well as making the symbolic link i described above) you can get around the problem. then again i guess the point of starnet++ is so that users don't have to mess around with installing stuff from Fink, MacPorts or Homebrew as you do with the python version.

 

rob

Yes, that's the plan, but you have to keep in mind that I don't have Mac and have never owned one, so my abilities are limited here.

 

I compiled libtiff myself and I don't remember linking it against any other libraries. On Windows that's just stand alone library, so I am surprised that it needs some third party files on Mac... And I can not even reproduce this error, since on the Mac I used to compile it all these libraries were there to begin with.

 

Trying to look into it, anyway! Thanks a lot for your input!

[pfile]

Yes, that's the plan, but you have to keep in mind that I don't have Mac and have never owned one, so my abilities are limited here.

 

I compiled libtiff myself and I don't remember linking it against any other libraries. On Windows that's just stand alone library, so I am surprised that it needs some third party files on Mac... And I can not even reproduce this error, since on the Mac I used to compile it all these libraries were there to begin with.

 

Trying to look into it, anyway! Thanks a lot for your input!

 

no problem, yes it is difficult doing this stuff blind.

 

for whatever reason, the libtiff shipped with StarNet_MacOS has the following undefined symbols, which i guess compels dyld to look for them:

                 U _WebPConfigInitInternal

                 U _WebPEncode

                 U _WebPFreeDecBuffer

                 U _WebPIAppend

                 U _WebPIDecGetRGB

                 U _WebPIDelete

                 U _WebPINewDecoder

                 U _WebPInitDecBufferInternal

                 U _WebPPictureFree

                 U _WebPPictureImportRGB

                 U _WebPPictureImportRGBA

                 U _WebPPictureInitInternal

                 U _WebPValidateConfig
 
                 U _jpeg_CreateCompress

                 U _jpeg_CreateDecompress

                 U _jpeg_abort

                 U _jpeg_destroy

                 U _jpeg_finish_compress

                 U _jpeg_finish_decompress

                 U _jpeg_has_multiple_scans

                 U _jpeg_read_header

                 U _jpeg_read_raw_data

                 U _jpeg_read_scanlines

                 U _jpeg_resync_to_restart

                 U _jpeg_set_colorspace

                 U _jpeg_set_defaults

                 U _jpeg_set_quality

                 U _jpeg_start_compress

                 U _jpeg_start_decompress

                 U _jpeg_std_error

                 U _jpeg_suppress_tables

                 U _jpeg_write_raw_data

                 U _jpeg_write_scanlines

                 U _jpeg_write_tables

among others, but they look like stdlib, etc. stuff.

 

so the /usr/lib path i indicated above is just peculiar to that machine where i have homebrew installed - dyld is looking there, which makes sense. on my desktop machine where i have MacPorts installed, i could not satisfy all the dependencies for whatever reason, but if i just make a symlink to /opt/local/lib/libtiff.5.dylib then that libtiff gets loaded and all it's dependencies are satisfied by dylibs in /opt/local/lib.

 

which brings me to this - the undefined symbols in /opt/local/libtiff.5.dylib are different than the ones shown above. notably libtiff from macports has no WebP symbols... which says to me that WebP support must be a compile-time option for libtiff, as could be jpeg support. so maybe if you turned off those ./configure flags for libtiff and rebuilt the deps would go away? that is, if they really are compile time options...

 

rob