241 replies to this topic

[Linwood]

what you are seeing with the stock tensorflow is normal. the DLL distributed with SNv2 runs tensorflow on the CPU. it's going to be extremely slow on a 2x drizzled image from a 6200. the 65% utilization is what i was talking about above; on smaller thread count machines i saw 100% CPU usage but adding more threads/cores does not speed things up past some point. that's not necessarily unusual for parallel codes.

 

rob

Ah.... Why does it include a stock tensorflow, wouldn't it already be present?   Or maybe not... maybe there's nothing in PI that needs it. 

 

But got it... thanks.  Then is explained. 

[pfile]

i'm pretty sure that SNv2 needs a newer version of the TF library than what is distributed with PI. so it comes with another copy. CPU is the lowest common denominator here as far as hardware is concerned so there's no GPU-based library distributed with either SNv1, PI's SN or SNv2. people figured out after the fact how to substitute the GPU-ful version of tensorflow on machines/GPUs that support it. for now i think that's only linux and windows with Nvdia GPUs. nothing else built in to PI has tensorflow as a dependency.

 

the only kink with the CPU-based stuff is that juan compiled TensorFlow with the AVX2 and FMA vector instructions turned on in the compiler. that means that PI's SN won't run on older Xeon CPUs or the apple M1's x86-64 emulator (Rosetta2) out of the box as it doesn't emulate those instructions. the TF libraries that came with SN1 (sourceforge) and SN2 don't have these vector instructions so they are compatible with more CPUs and Rosetta2. if you can find a version of the TensorFlow libraries that is the same as what comes with PI, you can replace the files just like you do to get GPU support to get SN running on the M1 or older intel CPUs.

 

rob

[Linwood]

the only kink with the CPU-based stuff is that juan compiled TensorFlow with the AVX2 and FMA vector instructions turned on in the compiler. that means that PI's SN won't run on older Xeon CPUs...

I may annoy a lot of people but when people invest thousands and tens of thousands of dollars in optics, they should also buy a CPU from the last 10 years.  

 

I have this great beefy GPU in the hopes PI will one day make use of it, by the way... hint... hint... subtle hint.... 

[pfile]

as it turns out for whatever reason FMA/AVX2 does not bring much to the table, at least for the graph that SN uses. i don't see any difference at all...

 

robyx was tasked with the GPU stuff in PI but WBPP has apparently been taking up all of his time.

 

the GPU stuff is gnarly with respect to cross-platform compatibility. its not going to be easy.

 

rob

[MAT_Blue]

I just replaced an old GeForce GT1030 card with a new GeForce RTX 3060 card.  I had StarNet2 2.0.0-34 with the CUDA modules running in PixInsight 1.8.9-1 (Win10 x64) with the old card (replace tensorflow dll, install CUDA 10.1 and copy the CUDNN \bin and \lib files) and it was working (100% on the GPU in Starnet2).

After installing the new card, I installed the latest version of the nVidia driver (516.59)  and checked that the same tensorflow, CUDA and CUDNN files were still there.

I ran Starnet2 and it took a few minutes before it did anything (busy cursor) then flew through the process but generated junk.  See attachment.

I tried uninstalling and reinstalling CUDA 10.1 but that didn’t help.

It was using a bunch of GPU memory but very little GPU processor time.

Any suggestions?

 

 

***UPDATE***

I managed to get it working.  I found a copy of tensorflow.dll version 2.8.0 for GPUs.  I used that with version 11.2 of CUDA and 8.1.1 of CUDNN.  Maybe it was because my nVidia driver is the latest version (516.59).  I was hesitant to downgrade the driver and I didn't have to with these versions of tensorflow, CUDA and CUDNN.  When I run nvidia-smi it says I have CUDA version 11.7.  Maybe that's the version in the driver???

I don't understand why it's so hard to find a compiled version of tensorflow.dll.  I still haven't found a compiled version of 2.9.0.

I spent all day trying different versions of tensorflow, CUDA and CUDNN before I finally found a set that worked.  Kind of frustrating.  The whole thing seems totally arcane to me.

Attached Thumbnails

• 

Edited by MAT_Blue, 17 July 2022 - 03:55 PM.

[MAT_Blue]

Update to my previous post (it won't let me edit that one for some reason).

 

Sorry about the rant at the end regarding tensorflow, CUDA and CUDNN.  I REALLY, REALLY appreciate all the effort that's been put into creating StarNet2 and for making it a free product.  It's a very good product.  My rant was not directed at the developer of StarNet2.  It took a while to get it working with the new RTX3060 card but it's well worth it.  I can now run StarNet2 at a stride of 256 on a 12 megapixel debayered image in under 15 sec.

 

I also managed to find where to get compiled versions of tensorflow for GPU:

https://storage.goog...86_64-2.8.0.zip

Copy the link and change the "2.8.0" at the end to whatever version you want.

 

So, with this card (which is supposed to be able to support CUDA 11.7), what are the recommended versions of CUDA and CUDNN to use with tensorflow GPU version 2.9.0?

[MAT_Blue]

I just double-checked and I think all the necessary tensorflow-related objects are deleted at the end of the session. I tend to keep PI open all the time, never restart it and have not noticed any potential leaks. Just made a run and memory usage was back to pre-starnet level.

 

This said, tensorflow is very bad at using GPU memory. Unlike CPU runs, it always allocates all the available GPU memory even if actual need is a tiny fraction of it. It could be that for some reason it fails to clean afterwards, but you would think that by now more people would notice this... 

 

 

How bad is it? What do you mean by threads spawned by tensorflow hanging around?

I noticed the same thing.  I have a RTX3060 with 12GB of GPU memory.  When I start StarNet2, the "Dedicated GPU Memory" goes to 12GB and stays there until I exit PixInsight.  Same thing happens if I run StarNet.

[jimmyjamesearl]

I just downloaded (7/22/22) StarNet2 and tried to install it and it didn't work.

 

The download files include only:

 

starnet2_weights.pb

tensorflow.dll

tiff.dll

 

Which according STARNET in their README file are the only three files to be loaded to PIXINSIGHT BIN folder

 

There are no longer any DYLIB files.

 

Has the installation change?

[pfile]

you must have downloaded the standalone version? you need the pixinsight version. PI requires a special dll/dylib containing the process.

 

also dylib is for macos, dll is for windows. so make sure you are downloading the right one.

 

rob

[jimmyjamesearl]

Thank you for your Help.

[jeffweiss9]

    Dumb question but I just tried to install Starnetv2 on my Win10 64b (PI 1.8.9-1) field laptop, after running it just fine for month(s) along with Starnet (v1), StarXterminator and NoiseXterminator on my desktop at home. However, after copying all the files in the readme into the ProgramFiles/Pixinsight/bin directory and seeing them there to be sure, when I try to install the module, Pixinsight says there are no modules to install. Starnet (v1) no longer runs on the laptop either as it says the AI (.pb) files don't exist.  I installed StarXterminator and NoiseXterminator first on the laptop and they are running fine.

    Any suggestions on what might be wrong?                

Thanks.

CS/Jeff

P.S. When I look in 'manage modules', I see the Starnet module is installed and Starnet-pxm.dll is in the bin directory but I didn't see any Starnet or Starnet2_weights.pb  AI files in the /Library directory. I did find Starnet2.pb in the /bin directory and copied it into the /Library directory but that didn't change anything described above.   After loading 3 missing Starnet V1 .pb files from my desktop to the laptop, I was able to get Starnet  (v1) to work on the laptop, but still no module and no luck with Starnet V2.

Edited by jeffweiss9, 30 July 2022 - 08:03 PM.

[MAT_Blue]

On my system, Starnet2_weights.pb is in the \bin folder.  The two files, mono_starnet_weights.pb and rgb_starnet_weights.pb are in the \library folder.  I think you can put the two starnet weights whereever you want, just use the wrench icon on the Starnet process window to tell it where they are.  There's no wrench icon for Starnet2 so the weights for it probably have to stay in the \bin folder.

 

When I go into Manage Modules, it lists StarNet-pxm.dll in the \bin folder for Starnet and StarNet2-pxm.dll for Starnet2.  I don't have Star Exterminator or NoiseExterminator.

 

Hope that helps.

[pfile]

one thing i've never been clear on is if the tensorflow libraries for v1 (which ship with PI) and the tensorflow libraries for v2 are actually compatible with one another.

 

since both libraries probably define the same symbols, if they are both present in the PI bin directory then what happens when windows tries to load them is kind of undefined. of course if the files have the same name, the act of copying the new ones from V2 into PI's application directory will erase the old ones, and so you're left with the newer tensorflow that ships with v2. i don't use windows so i don't know what it all looks like. i guess i'm saying that this might be a problem, but perhaps not.

 

PI is very opaque about modules that fail to load - it just silently ignores any modules that won't start when you search for them with the add module dialog. however, if you're brave enough, you can install the module by editing PixInsight.ini and putting it at the end of the modules section and giving it a number that's one higher than the last one that's currently shown in the file - just copy the entire last line of the modules section and fix it up to point at the StarNet dll. you should do this with PI not running. then when you start PI, it will fail to load the module and delete it from the .ini file, but because it failed at startup you'll get a dialog box saying what the error was. this is how i found out, for instance, that way back in the day certain versions of PI on OSX wouldn't load starnet because it was built in such a way that it required all dynamically linked libraries to have a signature. it's also how i figured out that for whatever reason, on M1 macs, i had to put the v2 tensorflow libraries in /Applications/PixInsight/PixInsight.app/Contents/Frameworks/ - PI actually prints in the dialog what list of directories it is searching for dynamic libraries in, so i knew i could put it there and PI would find it.

 

hopefully on windows you'll get enough info in that dialog box to figure out just what is going wrong when PI tries to load the module. just back up your PixInsight.ini file first so you can restore an unedited one, in case you mess up!

 

rob

[jeffweiss9]

Thanks very much, Mat-BLUE and Rob-
I ended up getting all 4 programs to work fine on the desktop by putting the assorted files in the following directories:

 

/bin:  StarNet2_weights.pb, NoiseXterminator-pm.dll, StarNet2-pm.dll, StarNet-pm.dll, StarXTerminator-pm.dlll
/library: StarNet2_weights.pb, mono_starnet_weights.pb,rgb_starnet_weights.pb, StarXterminator.2.pb and three StarXterminatorY.pb files where Y=6,7 and 10.

 

At this point, I don’t know who put the tensorflow.dll in /bin but it is dated 11/8/2021 and is 186,300kB.
Thanks again.
CS/Jeff

Edited by jeffweiss9, 31 July 2022 - 11:03 PM.

[cloudynites]

Hello
I am just an old newbie - never got much better at it, maybe until now!

Anyway, I installed the GUI version on Windows PC and I am not using Pixinsight at present - just stand-alone StarNet.

On my home PC with Windows 10 Pro 21H2 64bit works fine, but on my older laptop with Windows 10 Pro 1903 64bit  I keep getting an error with either of them GUI or Command Line.

I also tried 'compatibility settings' - I believe it did work once, maybe weeks ago.

Error is as if it cannot run on my computer:

 

Application Error
The application was unable to start correctly (0xc0000142).

My initial thoughts were, maybe the Graphics (GPU) driver might be a problem.

This windows installation - which works fine - is on a ATI Mobility Radeon HD 4650 - cannot be updated any more and I managed to make it work using a driver offered by Microsoft (as they did for many other graphics to make WIN 10 work).

I guess it is a Vista versions driver ... any way - I do not think it has anything to do with it.

 

Any idea?

Thanks in advance
M

Edited by cloudynites, 01 August 2022 - 03:55 AM.

[pfile]

what kind of CPU is in the laptop? some versions of the tensorflow library require AVX2 and FMA instructions.

 

rob

[jeffweiss9]

Unfortunately, now on vacation trying to use Starnet2 on the same old field laptop, PI 1.8.9-1 has forgotten Starnet2 once again. Erasing the PixInsight/updates.xri to trigger PI Update to reinstall modules did not help.  Starnet2 is lost again with no new modules in manage modules even though all the files needed appear to be in the correct directories.  Fortunately I can use StarXTerminator, which gave better results last time I tried them both anyway.

 

I don't know if this is a PI Updating issue or something strange about Starnet2, but repeated disappearences suggests the latter.

 

CS/Jeff

Edited by jeffweiss9, 03 August 2022 - 07:09 PM.

[pfile]

starnet2 isn't distributed via the pixinsight distribution center... you need to install starnetv2 by hand.

 

nice thing about SXT is that it uses the GPU out of the box both on macos and windows. so it's fast.

 

rob

[jeffweiss9]

Hi, Rob-

I had installed starnet2 and got it working fine after the exchange a few messages above on 31 July, The only thing I did after that was fly to  Kona, HI and try to use it again today, but it had disappeared in PI.  All the files are there in either /bin or /lib (I tried StarNet2_weights.pb in BOTH /bin and /lib), yet PI 1.8.9-1 does show it in manage modules and there are new new modules available. I've checked that all the files are the same date and file size as those in the StarNet2PI_Win.zip EXCEPT the tensorflow.dll which is newer on my system (probably subsequently installed by one of Russ Croman's codes).   The zip file tensorflow.dll is 145kb uncompressed and dates 11/27/20.  The tensorflow.dll in my /bin file has file size 186kB and is dated 11/9/21, a year later. Since I believe I reinstalled one of Russ's codes after getting StarNet2 to work (again) on 31 July, the difference in tensorflow.dll files seems to be the only difference I've found from the files in the StarNet2PI_Win.zip.  Incompatible tensorflow.dll's ??  (Garden variety Win10 Pro on the laptop with Intel® Core™ i7-6500U CPU @ 2.50GHz   2.60 GHz).

CS/Jeff

Edited by jeffweiss9, 03 August 2022 - 10:12 PM.

[pfile]

could be incompatible tensorflow, yeah. if PI can't start the module it just silently disappears, which is what sounds like happened to you. but you say you see it in Manage Modules if you search for modules again?

 

it's a terrible hack but you can quit PI and add the StarNetV2 dll to the PixInsight.ini file by copying the last line of the modules section, incrementing the module number, and changing the name to point to the StarNetV2 dll. then when you start PI, if it fails to load, PI should put up a dialog box saying what was wrong (and then it once again deletes the line you added to the ini file) but that might give you a clue about what is wrong with starnetV2. also you should probably back up the PixInsight.ini file before making your edits just in case you miss a bracket or something and cause the file to be un-parseable.

 

rob

[cloudynites]

what kind of CPU is in the laptop? some versions of the tensorflow library require AVX2 and FMA instructions.

 

rob

Sorry for late reply - busy at work and although Laptop is old, it still works very well!

SONY VGN-FW41M/H
CPU: Intel Core 2 Duo P8700 @ 2.53GHz

Thanks

 

ADDITION: ... and I did not even read the README file!

It states:
Second, the tensorflow.dll was compiled to support AVX instructions
and will only work if your CPU supports these AVX instructions.
Which means that CPUs made before 2011 will not work.

Mine is before 2011, as PC was from 2011 !
Lovely - just my luck!

Go here and download the 'installer'.
Run it and then click run. It will give you your CPU capability - GREEN is good!

Mine is GREY !!!

Edited by cloudynites, 05 August 2022 - 04:10 AM.

[jeffweiss9]

could be incompatible tensorflow, yeah. if PI can't start the module it just silently disappears, which is what sounds like happened to you. but you say you see it in Manage Modules if you search for modules again?

 

it's a terrible hack but you can quit PI and add the StarNetV2 dll to the PixInsight.ini file by copying the last line of the modules section, incrementing the module number, and changing the name to point to the StarNetV2 dll. then when you start PI, if it fails to load, PI should put up a dialog box saying what was wrong (and then it once again deletes the line you added to the ini file) but that might give you a clue about what is wrong with starnetV2. also you should probably back up the PixInsight.ini file before making your edits just in case you miss a bracket or something and cause the file to be un-parseable.

 

rob

Neither tensorflow or StarNet2 is visible in Manage Modules. I'll just keep things as they are for now and maybe try again when I'm back home and can see exactly which tensorflow.dll seems to allow all of those to work simultaneously.

Tnx,

Jeff

[cloudynites]

Anywhere I can download version 1 to try or was it already using AVX AVX2 ?

I vaguely remember it was working on this laptop or maybe I am imagining it!

I have not done any imaging in the last 2 years apart from a couple of tests I think.
M

 

[pfile]

you know, i've been so focused on AVX2/FMA that i'm not even sure if the original starnet's tensorflow libs were compiled with AVX.

 

you might be able to find some compatible dlls on tensorflow.org. but you have to make sure to use the right version of tensorflow and off the top of my head i don't remember what starnetv1 or starnetv2 expected.

 

rob

[cloudynites]

Never mind, will try with another laptop I will keep where I work - when I have no work I can play with it!