Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

StarNet: star removing software

astrophotography
  • Please log in to reply
240 replies to this topic

#26 NGC3031

NGC3031

    Vendor - Sattelite Chaser

  • -----
  • Vendors
  • Posts: 49
  • Joined: 29 Oct 2016

Posted 03 March 2019 - 07:35 AM

This is a very nice software for creating star masks indeed. Thanks for sharing, this will probably become my main tool for creating star masks :)
For anyone wondering how to create the star mask: In Pixelmath, subtract the starless from the original version and extract the grayscale image.

 

xGbnMRo.gif


  • darkstar3d, futuneral, nekitmm and 1 other like this

#27 jdj

jdj

    Lift Off

  • -----
  • Posts: 7
  • Joined: 17 Jun 2011
  • Loc: Silver Spring, MD

Posted 03 March 2019 - 06:29 PM

Hi Nikita -

 

Thanks for making this available to the community, it looks like an amazing tool.  Unfortunately, the dropbox link for the pre-compiled version is no longer working (something about too many requests, a good sign I suppose!). If you could post the pre-compiled version elsewhere that would be fantastic.

 

Thank you!

John



#28 nekitmm

nekitmm

    Explorer 1

  • *****
  • topic starter
  • Posts: 64
  • Joined: 25 Apr 2018

Posted 03 March 2019 - 07:45 PM

Hi Nikita -

 

Thanks for making this available to the community, it looks like an amazing tool.  Unfortunately, the dropbox link for the pre-compiled version is no longer working (something about too many requests, a good sign I suppose!). If you could post the pre-compiled version elsewhere that would be fantastic.

 

Thank you!

John

I am glad it is helpful!

 

The DB link indeed was banned! I moved the program to SourceForge, here is the link:

 

https://sourceforge....ojects/starnet/

 

I didn't find the way to edit the original post though....


  • Oleg Astro and jdj like this

#29 jdj

jdj

    Lift Off

  • -----
  • Posts: 7
  • Joined: 17 Jun 2011
  • Loc: Silver Spring, MD

Posted 04 March 2019 - 07:31 PM

Thanks much for posting the new link Nikita.  I tried out your program and it works like a charm!

 

Here are original and starless versions of a recent Rosette Nebula image of mine:

 
Rosette Nebula (OSC)
Rosette Nebula (OSC) - Starless version using Starnet++
 

Congrats on a fine piece of software and thanks again for making this available to the community!!

 

John


  • nekitmm likes this

#30 Stevous

Stevous

    Mariner 2

  • -----
  • Posts: 272
  • Joined: 01 Sep 2009
  • Loc: Melbourne, Victoria, Australia

Posted 10 March 2019 - 02:26 PM

Dear Nikita,

 

I am adding my deepest thanks to you as well, this line command is simply outstanding.

 

Best wishes,

 

Steve


  • nekitmm likes this

#31 nekitmm

nekitmm

    Explorer 1

  • *****
  • topic starter
  • Posts: 64
  • Joined: 25 Apr 2018

Posted 10 March 2019 - 10:49 PM

Dear Nikita,

 

I am adding my deepest thanks to you as well, this line command is simply outstanding.

 

Best wishes,

 

Steve

Thanks, Steve!

 

I am glad that you like it!



#32 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5178
  • Joined: 14 Jun 2009

Posted 14 March 2019 - 04:28 PM

came across this on instagram this morning. got the python version running on OSX relatively easily... i don't have a 64-bit wine environment to run starnet++ but if i understand it right the python version is equivalent. with 12 threads it processes images from an 8300M in 1-2 minutes, which is totally fine... and the results are incredible. easily the best star removal software i've ever used, and trivially easy to run! thanks nikita.

 

p.s. ever thought about training a GAN like this for noise reduction? i know its being done but i wonder if astronomical images need a different training set.


  • nekitmm likes this

#33 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5178
  • Joined: 14 Jun 2009

Posted 14 March 2019 - 11:10 PM

wow. i'm totally blown away, check out what starnet did here. just took me about 10 minutes of very simple cleanup with clonestamp + some noise reduction:

 

get.jpg?insecure

 

you can blink back and forth between the original and the starless and see just how clean it is.

 

rob


  • bilgebay, futuneral, nekitmm and 1 other like this

#34 nekitmm

nekitmm

    Explorer 1

  • *****
  • topic starter
  • Posts: 64
  • Joined: 25 Apr 2018

Posted 15 March 2019 - 04:14 PM

wow. i'm totally blown away, check out what starnet did here. just took me about 10 minutes of very simple cleanup with clonestamp + some noise reduction:

 

get.jpg?insecure

 

you can blink back and forth between the original and the starless and see just how clean it is.

 

rob

Thanks for the kind words, Rob! I am glad that it worked for you! The result looks good!

 

 

p.s. ever thought about training a GAN like this for noise reduction? i know its being done but i wonder if astronomical images need a different training set.

That actually might be my next direction to investigate! Would be interesting to see what GANs can achieve here!


  • RossW likes this

#35 MattC867

MattC867

    Mariner 2

  • -----
  • Posts: 222
  • Joined: 18 Jun 2016

Posted 17 March 2019 - 11:02 PM

This is really amazing, thanks!

I am looking to train the model on some of my own data, but I can't find the ./log folder that the program is looking for. Keep getting an error. Same issue trying to run the plot sub function - what am I doing wrong?

Edited by MattC867, 17 March 2019 - 11:03 PM.

  • nekitmm likes this

#36 nekitmm

nekitmm

    Explorer 1

  • *****
  • topic starter
  • Posts: 64
  • Joined: 25 Apr 2018

Posted 18 March 2019 - 09:30 PM

This is really amazing, thanks!

I am looking to train the model on some of my own data, but I can't find the ./log folder that the program is looking for. Keep getting an error. Same issue trying to run the plot sub function - what am I doing wrong?

Hi, Matt!

 

Sorry about that! I added the missing folder into git repo! It should work just fine now!

 

Let me know if you will have other problems! Also, if possible, please let me know results of your experiments!

 

Thank you for kind words!



#37 airscottdenning

airscottdenning

    Mariner 2

  • *****
  • Posts: 205
  • Joined: 22 Aug 2008
  • Loc: Colorado

Posted 18 March 2019 - 11:26 PM

I cloned the python version from git and it works very well for me on a recent mac.

 

Tried it out on an image of the Rosette that I shot with one-shot color on a 488 mm refractor. Results are interesting: stars are gone, but lots of artifacts are left behind. 

 

I understand this would be better if I trained the program on my own image. Do you have any suggestions about how to better clean it up for training purposes?

 

Here's a link to a jpg of the starless image (small preview attached).

 

Presentation2.jpg


Edited by airscottdenning, 18 March 2019 - 11:26 PM.

  • nekitmm likes this

#38 nekitmm

nekitmm

    Explorer 1

  • *****
  • topic starter
  • Posts: 64
  • Joined: 25 Apr 2018

Posted 18 March 2019 - 11:34 PM

I cloned the python version from git and it works very well for me on a recent mac.

 

Tried it out on an image of the Rosette that I shot with one-shot color on a 488 mm refractor. Results are interesting: stars are gone, but lots of artifacts are left behind. 

 

I understand this would be better if I trained the program on my own image. Do you have any suggestions about how to better clean it up for training purposes?

 

Here's a link to a jpg of the starless image (small preview attached).

 

attachicon.gif Presentation2.jpg

Hi!

 

It would be nice to also see full size of original image! I think that possible cause of all these artifacts is that original image was heavily processed and star profiles became too distorted.

 

In most cases you will get significantly better results if you try on a less processed version of the image. Can you try that before getting into all the troubles of training?


Edited by nekitmm, 18 March 2019 - 11:36 PM.


#39 airscottdenning

airscottdenning

    Mariner 2

  • *****
  • Posts: 205
  • Joined: 22 Aug 2008
  • Loc: Colorado

Posted 19 March 2019 - 10:34 AM

Nikita,

 

Thank you for the suggestion. I redid my Rosette image with just a light stretch and re-ran StarNet transform (using your weights) with MUCH better results!

 

Here's a Dropbox link to the "before" image: gentle.stretch.tif

and here's a Dropbox link to the "after" image: gentle.stretch.tif_starless.tif

 

Your software is WONDERFUL!

 

Would you consider modifying it to allow higher bit depths of the "before" image?  It seems that I am losing some dynamic range by having to save my image as 8 bits per channel before calculating doing the star removal.

 

At any rate, thank you VERY MUCH for your hard work on this, and for making this amazing tool available for our community!

 

Have you shared this in the PixInsight forums?

 

Best regards,

Scott Denning

 

 

Hi!

 

It would be nice to also see full size of original image! I think that possible cause of all these artifacts is that original image was heavily processed and star profiles became too distorted.

 

In most cases you will get significantly better results if you try on a less processed version of the image. Can you try that before getting into all the troubles of training?


  • nekitmm likes this

#40 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5178
  • Joined: 14 Jun 2009

Posted 19 March 2019 - 04:11 PM

+1 higher bit depths, but looking at the code it seems like internally the images might be internally represented by 32b floats already?

 

transform.py:        input = np.array(img.open(image), dtype = np.float32)

 

so is it just a matter of using a different library to read/write the file?

 

also, another question - how many images did you have to use to train the network to get satisfactory results?

 

rob


Edited by pfile, 19 March 2019 - 04:12 PM.

  • nekitmm likes this

#41 nekitmm

nekitmm

    Explorer 1

  • *****
  • topic starter
  • Posts: 64
  • Joined: 25 Apr 2018

Posted 19 March 2019 - 09:34 PM

Nikita,

 

Thank you for the suggestion. I redid my Rosette image with just a light stretch and re-ran StarNet transform (using your weights) with MUCH better results!

 

Here's a Dropbox link to the "before" image: gentle.stretch.tif

and here's a Dropbox link to the "after" image: gentle.stretch.tif_starless.tif

 

Your software is WONDERFUL!

 

Would you consider modifying it to allow higher bit depths of the "before" image?  It seems that I am losing some dynamic range by having to save my image as 8 bits per channel before calculating doing the star removal.

 

At any rate, thank you VERY MUCH for your hard work on this, and for making this amazing tool available for our community!

 

Have you shared this in the PixInsight forums?

 

Best regards,

Scott Denning

 

 

+1 higher bit depths, but looking at the code it seems like internally the images might be internally represented by 32b floats already?

Lol, I also vote for higher bit depths!)

 

I am not sure right now, but when I was developing the code, something didn't let me do that. I think PIL image module that I use to read and write files does not support reading (or writing?) 16 bit files. I don't remember exactly the cause right now. Internally it is not a problem, as you noted.

 

Anyway, I will look into this a bit later. Right now I am working on a new release of binary files for both Windows and Mac. You will be able to work with 16bit files using those! Stay tuned!

 

Thanks for kind words!


  • futuneral likes this

#42 nekitmm

nekitmm

    Explorer 1

  • *****
  • topic starter
  • Posts: 64
  • Joined: 25 Apr 2018

Posted 24 March 2019 - 11:00 PM

Hi, everyone!

 

I just released new version of starnet! See here for downloads: https://sourceforge....ojects/starnet/

 

For some additional description please see https://www.astrobin.com/397290/. Don't want to repeat all the text here!

 

There are two major changes in this release:

 

1. There are now two executables: for Grayscale images and for RGB images. Both still work with 16bit per channel TIF images. Previous release was created with only color images in mind and performance on Grayscale images was often rather poor. This should not be the case anymore.

 

2. Both Windows and MacOS are now supported (make sure you download proper version).

 

Hope it will work well for you! All feedback is welcome (good or bad)!

 

Thanks!


Edited by nekitmm, 24 March 2019 - 11:02 PM.


#43 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5178
  • Joined: 14 Jun 2009

Posted 24 March 2019 - 11:39 PM

seems like there might be some problems with the OSX build.

 

1) dyld complains that libtiff.5.dyld is not found in @rpath

     - this seems to be fixed by making a symlink from libtiff.5.dyld to the provided libtiff.dyld .

     - but i suspect installing libtiff from homebrew might also fix this.

 

2) dyld then complains:

    dyld: Library not loaded: /usr/local/opt/jpeg/lib/libjpeg.9.dylib

 

installing libjpeg from brew solves that.

 

then 3) dyld complains:

 

dyld: Library not loaded: /usr/local/opt/webp/lib/libwebp.7.dylib

 

again installing webp from brew solves that.

 

so in short i think these binaries were built assuming that the user has HomeBrew and has those liibraries installed.

 

rob



#44 nekitmm

nekitmm

    Explorer 1

  • *****
  • topic starter
  • Posts: 64
  • Joined: 25 Apr 2018

Posted 24 March 2019 - 11:43 PM

seems like there might be some problems with the OSX build.

 

1) dyld complains that libtiff.5.dyld is not found in @rpath

     - this seems to be fixed by making a symlink from libtiff.5.dyld to the provided libtiff.dyld .

     - but i suspect installing libtiff from homebrew might also fix this.

 

2) dyld then complains:

    dyld: Library not loaded: /usr/local/opt/jpeg/lib/libjpeg.9.dylib

 

installing libjpeg from brew solves that.

 

then 3) dyld complains:

 

dyld: Library not loaded: /usr/local/opt/webp/lib/libwebp.7.dylib

 

again installing webp from brew solves that.

 

so in short i think these binaries were built assuming that the user has HomeBrew and has those liibraries installed.

 

rob

Thanks for the info. That's pretty weird though, the build was tested on a few MacOS machines and there were no problems running it without installing anything...

 

Why does it try to load jpeg libs? That's weird. Is your input TIF file compressed?

 

I will try to look into it!

 

Thanks!


Edited by nekitmm, 24 March 2019 - 11:55 PM.


#45 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5178
  • Joined: 14 Jun 2009

Posted 24 March 2019 - 11:56 PM

dyld: Library not loaded: /usr/local/opt/webp/lib/libwebp.7.dylib
  Referenced from: /Users/pfile/Downloads/StarNet_MacOS/libtiff.dylib

 

dyld: Library not loaded: /usr/local/opt/jpeg/lib/libjpeg.9.dylib
  Referenced from: /Users/pfile/Downloads/StarNet_MacOS/libtiff.dylib

 

so they would appear to be dependencies of libtiff for whatever reason...

 

/usr/local/ seems to be where homebrew installs all its stuff. i don't think there's anything in that directory on a stock OSX installation but i'm not sure.

 

rob



#46 airscottdenning

airscottdenning

    Mariner 2

  • *****
  • Posts: 205
  • Joined: 22 Aug 2008
  • Loc: Colorado

Posted 25 March 2019 - 03:22 PM

I get the exact same errors when I try to run this on my mac(s).



#47 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5178
  • Joined: 14 Jun 2009

Posted 25 March 2019 - 03:32 PM

if you install homebrew and then install webp and libjpeg (as well as making the symbolic link i described above) you can get around the problem. then again i guess the point of starnet++ is so that users don't have to mess around with installing stuff from Fink, MacPorts or Homebrew as you do with the python version.

 

rob



#48 nekitmm

nekitmm

    Explorer 1

  • *****
  • topic starter
  • Posts: 64
  • Joined: 25 Apr 2018

Posted 25 March 2019 - 06:00 PM

I get the exact same errors when I try to run this on my mac(s).

Were you able to follow instructions from Rob and get it running?



#49 nekitmm

nekitmm

    Explorer 1

  • *****
  • topic starter
  • Posts: 64
  • Joined: 25 Apr 2018

Posted 25 March 2019 - 06:05 PM

if you install homebrew and then install webp and libjpeg (as well as making the symbolic link i described above) you can get around the problem. then again i guess the point of starnet++ is so that users don't have to mess around with installing stuff from Fink, MacPorts or Homebrew as you do with the python version.

 

rob

Yes, that's the plan, but you have to keep in mind that I don't have Mac and have never owned one, so my abilities are limited here.

 

I compiled libtiff myself and I don't remember linking it against any other libraries. On Windows that's just stand alone library, so I am surprised that it needs some third party files on Mac... And I can not even reproduce this error, since on the Mac I used to compile it all these libraries were there to begin with.

 

Trying to look into it, anyway! Thanks a lot for your input!


  • pfile likes this

#50 pfile

pfile

    Fly Me to the Moon

  • -----
  • Posts: 5178
  • Joined: 14 Jun 2009

Posted 25 March 2019 - 06:21 PM



Yes, that's the plan, but you have to keep in mind that I don't have Mac and have never owned one, so my abilities are limited here.

 

I compiled libtiff myself and I don't remember linking it against any other libraries. On Windows that's just stand alone library, so I am surprised that it needs some third party files on Mac... And I can not even reproduce this error, since on the Mac I used to compile it all these libraries were there to begin with.

 

Trying to look into it, anyway! Thanks a lot for your input!

 

no problem, yes it is difficult doing this stuff blind.

 

for whatever reason, the libtiff shipped with StarNet_MacOS has the following undefined symbols, which i guess compels dyld to look for them:

                 U _WebPConfigInitInternal

                 U _WebPEncode

                 U _WebPFreeDecBuffer

                 U _WebPIAppend

                 U _WebPIDecGetRGB

                 U _WebPIDelete

                 U _WebPINewDecoder

                 U _WebPInitDecBufferInternal

                 U _WebPPictureFree

                 U _WebPPictureImportRGB

                 U _WebPPictureImportRGBA

                 U _WebPPictureInitInternal

                 U _WebPValidateConfig
 
                 U _jpeg_CreateCompress

                 U _jpeg_CreateDecompress

                 U _jpeg_abort

                 U _jpeg_destroy

                 U _jpeg_finish_compress

                 U _jpeg_finish_decompress

                 U _jpeg_has_multiple_scans

                 U _jpeg_read_header

                 U _jpeg_read_raw_data

                 U _jpeg_read_scanlines

                 U _jpeg_resync_to_restart

                 U _jpeg_set_colorspace

                 U _jpeg_set_defaults

                 U _jpeg_set_quality

                 U _jpeg_start_compress

                 U _jpeg_start_decompress

                 U _jpeg_std_error

                 U _jpeg_suppress_tables

                 U _jpeg_write_raw_data

                 U _jpeg_write_scanlines

                 U _jpeg_write_tables

among others, but they look like stdlib, etc. stuff.

 

so the /usr/lib path i indicated above is just peculiar to that machine where i have homebrew installed - dyld is looking there, which makes sense. on my desktop machine where i have MacPorts installed, i could not satisfy all the dependencies for whatever reason, but if i just make a symlink to /opt/local/lib/libtiff.5.dylib then that libtiff gets loaded and all it's dependencies are satisfied by dylibs in /opt/local/lib.

 

which brings me to this - the undefined symbols in /opt/local/libtiff.5.dylib are different than the ones shown above. notably libtiff from macports has no WebP symbols... which says to me that WebP support must be a compile-time option for libtiff, as could be jpeg support. so maybe if you turned off those ./configure flags for libtiff and rebuilt the deps would go away? that is, if they really are compile time options...

 

rob




CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics





Also tagged with one or more of these keywords: astrophotography



Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics