108 replies to this topic

[RedLionNJ]

Hi Grant,

 

I'm sorry that you wasted your time in getting my code to do something meaningful on your computer. uage with good parallel support.

 

Btw.: This weekend I wrote a detailed documentation of the algorithmic details of my code and put it on Github. If you want to see how the stacking program works (without having to read the source code), perhaps this PDF is interesting.

 

All the best,

 Rolf

No worries, Rolf - it was hardly a waste - I learned a few things along the way. Always a good experience. Probably better for the rest of the day's chores I did fail, actually. Else I'd likely have spent the entire Sunday sitting here at my computer, experimenting with it!

 

I just read the documentation - nicely-done.

 

One question, though - when handling RGB image data, is the stacking done in a similar way to Emil's method, where you take advantage of the Bayer matrix to build up a full-resolution image, only adding in data of one specific color for each pixel per frame?

 

Cheers,

 

Grant

[John Duchek]

After installing several of these I am getting the same error as the other user.

Even then, after all that, I get an error about (rightly) not being able to locate a DLL (mkl_rt.dll).

Since I am in linux, I don't think I will be installing a dll soon. 

 

As to my comment about siril having the stack button greyed out, next time I tried the program it worked fine.  go figure.

 

John

[Rolf]

Hi John,

 

After installing several of these I am getting the same error as the other user.

Even then, after all that, I get an error about (rightly) not being able to locate a DLL (mkl_rt.dll).

Since I am in linux, I don't think I will be installing a dll soon. 

 

As to my comment about siril having the stack button greyed out, next time I tried the program it worked fine.  go figure.

 

John

I really cannot help you in getting the Python libraries installed correctly on your Linux system. I have installed all the libraries I listed in my previous note on an Ubuntu 16.04 LTS, and everything worked without any problems.

 

Best wishes,

 Rolf

[Rolf]

Hi Grant,

 

One question, though - when handling RGB image data, is the stacking done in a similar way to Emil's method, where you take advantage of the Bayer matrix to build up a full-resolution image, only adding in data of one specific color for each pixel per frame?

 

Cheers,

 

Grant

Thank you very much for this suggestion! At the moment stacking is done pixelwise for all three colors. That's also how all three colors of an image are stored. Of course one could do as you suggested. This could save operations at the cost of a more complicated data access pattern. If this really helps in terms of performance would have to be tried out. (This part of the algorithm is not performance-critical, anyway.)  It could help to reduce RAM usage, though, because the color images would only occupy a third of the memory compared to the current scheme. We will think about it!

 

Best wishes,

 Rolf

[RedLionNJ]

Hi John - I found the Intel MKL functionality is available via an "add on" download here:

 

https://software.see...mance-libraries

 

 

It looks like there's a version for Linux as well as one for Windows.

 

Grant

[TorstenEdelmann]

 For performance reasons I certainly would not choose Java. Stacking is really performance-critical, and an improvement over Python + Numpy seems to me possible only if I use a compiled language with good parallel support.

 

 

Hi Rolf,

 

First of all I'm quite impressed how far your project has evolved since the last time we emailed.

 

I can't contribute much but just a note regarding performance: when I started with FC I did a lot of performance testing between Java and C++ just to figure out when and which I should run parts of the code on native side using JNI. On those tests it was obvious that the difference in performance was very little.

 

When it comes to performance you really should consider using GPU acceleration as this IMO offers by far the most potential, much much more than just the decision to choose C++ over Java or whatever language. So one idea whould be making your stacking tool CUDA compatible offering NVIDIA users an option to run the code on the GPU. There are a couple of tools available for reimplementing algorithms on GPU (e.g. OpenCV CUDA). At the end it needs to be tested how good the stacking code could be adapted to running efficiently on the GPU but in case it works speed improvement is astounding.

 

Torsten

Edited by TorstenEdelmann, 17 January 2019 - 01:40 AM.

[Rolf]

Hi Torsten,

 

Hi Rolf,

 

First of all I'm quite impressed how far your project has evolved since the last time we emailed.

 

I can't contribute much but just a note regarding performance: when I started with FC I did a lot of performance testing between Java and C++ just to figure out when and which I should run parts of the code on native side using JNI. On those tests it was obvious that the difference in performance was very little.

 

When it comes to performance you really should consider using GPU acceleration as this IMO offers by far the most potential, much much more than just the decision to choose C++ over Java or whatever language. So one idea whould be making your stacking tool CUDA compatible offering NVIDIA users an option to run the code on the GPU. There are a couple of tools available for reimplementing algorithms on GPU (e.g. OpenCV CUDA). At the end it needs to be tested how good the stacking code could be adapted to running efficiently on the GPU but in case it works speed improvement is astounding.

 

Torsten

Thank you very much for your comments!

 

I think that with Python and the underlying Numpy framework I can get quite close to C performance inside those library functions for array operations. I see the main performance problem in unneccessary memory copies and temporary objects kept by Python between such operations. Since Python (as well as Java) does not have an explicit memory management, at the application programming level I can hardly avoid that. Similarly, multi-core control is of course provided by Numpy, but possibly one could do it more efficiently outside the innermost operation blocks. Again, this is sonething I would like to look at in a language like C or Fortran.

 

I already considered exploiting the power of graphics cards. Here at my institute we have several projects where that has been done successfully. (So I have the experts at hand who can tell me how to do it.) On the negative side, however, it makes code deployment and portability more complicated. So, this topic is not of highest priority right now, but I will keep an eye on it.

 

All the best,

 Rolf

[Rolf]

Hi,

 

Today I finished the last module of the PlanetarySystemStacker prototype. It deals with the postprocessing of the stacked image. I invested a lot of effort on the user interface to make the sharpening process as easy and effective as possible. Here are some features of the module:

• An arbitrary number of vesions of the sharpened image can be created.
• For each version up to four sharpening layers can be defined. The kernel radius and the effect is set individually for each layer.
• For color images sharpening can be applied to all color channels, or to the luminosity channel only.
• All computations are in 16 bit full color mode.
• Different versions can be compared in the image viewer. The viewer supports zooming and panning.
• In switching between versions, the same part of the image is displayed for easy comparison.

Additionally there  is a "blink comparator". Here you can select two versions for direct comparison. The viewer alternately shows both versions. Panning and zooming also works while the blink comparator is running.

 

To give you an idea how all this looks like, here is an animation of the blink comparator in action:

 

 

(To see the animation, you have to click on the image.)

 

In this example a single sharpened version of the image, using two sharpening layers, has been created. It is compared with the original image (which resulted from the stacking process). The original image always is counted as "version 0". To keep the versions apart during blinking, the version manager displays the version number currently displayed in red.

 

In the next few days I will integrate the module into the PSS software and start with the final system tests.

 

All the best,

 Rolf

Edited by Rolf, 01 May 2019 - 09:47 AM.

[NGC3031]

The progress you have made on this software is astounding. Have you done comparisons with Wavelets/Deconvolution?

[Rolf]

Hi,

 

The progress you have made on this software is astounding. Have you done comparisons with Wavelets/Deconvolution?

Yes, I compared the results to the wavelet sharpening in AviStack2, and I do not see any difference. Actually, the multi-layer option allows to adapt the deconvolution kernel in a very general way, very similar to how it is done with wavelets. Usually, however, one or at most two layers are completely sufficient.

 

All the best,

 Rolf

[Tom Glenn]

Rolf, thanks for providing the update.  My initial reaction is that incorporating sharpening into the program seems unnecessary, as most people have settled upon particular sharpening routines they are comfortable with using other software, and will only use the stacking software for stacking.  But your version, as described here, does have certain features that will make it worth testing.  I like the option of having a viewer that allows multiple versions to be compared simultaneously.  And convenience does count for something in this respect.  For example, I have stopped using Registax to sharpen my lunar images, because it has become very slow and cumbersome when dealing with large files.  There can be a 10-20 second lag (or more) in between making any adjustment, and there is no convenient way to compare results (such as a simple toggle feature).  This makes comparisons nearly impossible, unless you truly have a photographic memory.  In AstraImage, there is a preview window that provides instantaneous feedback on each sharpening setting, which is nice, although in this case there is still no way to compare different versions simultaneously (you can only compare a proposed version to the base layer).  In order to do this, I have to save several versions and then load them in Photoshop, at which point I can easily compare alternate versions by toggling layers on and off.  Saving this step would be nice, although in the end the utility of the sharpening will come down to the results.  I have grown quite accustomed to using Lucy-Richardson deconvolution, but will be interested to compare the results in your program.  

[airscottdenning]

Wow, this is very impressive!  Thank you very much for taking this on, and especially for choosing to open the project to others in the community.

 

We have long needed a project that can develop and evolve over the years, even after the original developer loses interest.  

 

Congratulations!

[TareqPhoto]

Wow, this is very impressive!  Thank you very much for taking this on, and especially for choosing to open the project to others in the community.

 

We have long needed a project that can develop and evolve over the years, even after the original developer loses interest.  

 

Congratulations!

I agree and support that!

[Rolf]

Hi Tom,

 

Rolf, thanks for providing the update.  My initial reaction is that incorporating sharpening into the program seems unnecessary, as most people have settled upon particular sharpening routines they are comfortable with using other software, and will only use the stacking software for stacking.  But your version, as described here, does have certain features that will make it worth testing.  I like the option of having a viewer that allows multiple versions to be compared simultaneously.  And convenience does count for something in this respect.  For example, I have stopped using Registax to sharpen my lunar images, because it has become very slow and cumbersome when dealing with large files.  There can be a 10-20 second lag (or more) in between making any adjustment, and there is no convenient way to compare results (such as a simple toggle feature).  This makes comparisons nearly impossible, unless you truly have a photographic memory.  In AstraImage, there is a preview window that provides instantaneous feedback on each sharpening setting, which is nice, although in this case there is still no way to compare different versions simultaneously (you can only compare a proposed version to the base layer).  In order to do this, I have to save several versions and then load them in Photoshop, at which point I can easily compare alternate versions by toggling layers on and off.  Saving this step would be nice, although in the end the utility of the sharpening will come down to the results.  I have grown quite accustomed to using Lucy-Richardson deconvolution, but will be interested to compare the results in your program.  

Thank you very much for your interesting feedback!

 

My original plan was not to incorporate sharpening in my program. I changed my mind for the following reasons:

• The alternatives I used so far (mainly AviStack2, Giotto and Registax) are dead. All this software has not been maintained for many years.
• In my opinion all existing user interfaces are unsatisfactory. I fully agree with what you wrote about your experiences. Since sharpening is no "exact science" where you see immediately if you have reached the optimum, an essential feature is the possibility to compare different versions. Giotto provided that to some extent. You could process your image in up to four different windows side by side. But even then I find it difficult to compare subtle changes. So, I came up with the idea of the blink comparator, combined with the possibility to zoom into detail areas and to navigate in the picture by simply dragging with the mouse. You can even change the sharpening parameters during blinking, and you can change the versions being compared.

That said, I didn't want to include postprocessing in the standard workflow. This is my solution:

• At program startup, you can define a list of jobs which PSS will handle one after the other. PSS decides what to do depending on the input which you specify for the job:
  1. If you select a video file, PSS will stack it.
  2. If you select a directory containing image files (tiff, jpg, png, ...), PSS will stack all images in that directory.
  3. If you select a single image file (tiff, jpg, png, ...), PSS will postprocess it.
• The job list can contain jobs with different type in arbitrary order. In cases 1. and 2. the primary result is a summation image. In the configuration viewer you can set an option saying that the program should include postprocessing. In that case both the stacked and the sharpened images are written to disk.
• I thought a lot about how best to combine manual and batch processing. I did not find any convincing solution among the existing programs. In PSS I implemented the following solution:
  • You start with the input specification. This can be a single job (see above) or many jobs. When you press the "start" button, PSS starts processing all jobs in succession.
  • A single checkbox in the main GUI decides between manual and batch mode. If it is checked, after each processing step PSS gives control back to the user. For example, it provides a view of the image where you can define the alignment points manually, or modify the automatic choice provided by PSS. PSS records all parameter changes you do in manual mode.
  • If you like, you can process the entire workflow in manual mode. Or, at some point you check the "automatic" checkbox, and PSS from that point on will continue in batch mode. From this point on there is no user interaction via the graphical user interface. PSS processes the jobs with the current parameter set. The process details are written in a protocol which you can inspect in the end. For the protocol, you can choose among three detail levels.
  • Since in the case of hundreds of jobs the protocol file gets a little confusing, there is the option for each job to store the protocol section related to this job together with the stacked image. This way it is easy to look at the processing details if there is something wrong with a particular stacking result.
  • At any time during batch mode, you can switch back to manual mode just by unchecking the "automatic" checkbox. After finishing the current workflow step (even in the middle of a job), PSS gives control back to the user. From this point on you can proceed in manual mode. Of course you can switch back to batch mode any time.

As you see, I pay a lot of attention to the aspect of "usability". At my institute  at the German Aerospace Center, software engineering, and in particular software usability, are among our core research topics. This is one reason. The other one is that as an amateur astronomer I have been suffering from the inadequate user interfaces of the existing programs. So at least for me my program will make the processing of my moon images much more convenient. And I hope that others will find my software useful as well.

 

All the best,

 Rolf

[eekeek77]

Hi Rolf,

 

Very interested in your progress. Again I can't add much other than moral support... 

 

I think there is a great need for one package, with a simple interface, that will do everything 'quite well'. I believe there are plenty of people who do not want to get into Photoshop or Pixinsight. I heartily agree with what has been said about user interfaces and having to jump from AS! to Registax to whatever...

 

I love the blink comparator!

 

re: GPUs. My understanding is that OpenCL is hardware agnostic. It just sees CPUs, iGPUs and GPUs as resources. It is nearly as efficient as CUDA on Nvidia but also works on AMD without any extra coding for you.

[Rolf]

Hi,

 

Hi Rolf,

 

Very interested in your progress. Again I can't add much other than moral support... 

 

I think there is a great need for one package, with a simple interface, that will do everything 'quite well'. I believe there are plenty of people who do not want to get into Photoshop or Pixinsight. I heartily agree with what has been said about user interfaces and having to jump from AS! to Registax to whatever...

 

I love the blink comparator!

 

re: GPUs. My understanding is that OpenCL is hardware agnostic. It just sees CPUs, iGPUs and GPUs as resources. It is nearly as efficient as CUDA on Nvidia but also works on AMD without any extra coding for you.

Thank you very much for your positive comments!

 

Regarding the use of GPUs, I am not so sure any more that they could help a lot during stacking. The problem is the bottleneck between main memory and GPU memory. If the number of operations per memory item is too low (which I suspect is the case), you don't get any speedup. This would be different for postprocessing. There basically you only have to load the original image once, and then the memory traffic is negligable. If speed becomes an issue in postprocessing, I will investigate this option.

 

Of couse I know OpenCL, and I like the portability aspect. Nevertheless its use would make the deployment of my software somewhat more complicated.

 

Today I'm working on the integration of postprocessing in the configuration handling. The last step then will be the integration into the workflow control.

 

All the best,

 Rolf

[RedLionNJ]

+1 on the GPU being of minimal (if any) help during the alignment or stacking aspects. I've been a lot of years in the software industry, including graphical applications and these two operations in particular just don't seem to be well-attuned to typical GPU capabilities.  Raw CPU, yes. Even multi-processors, multi-cores or multi-threads.  But I just don't see how anyone could utilize the GPU for this.

 

Keep going, Rolf - I'd love to play with a version I can get to run.

 

Any possibility of publishing at least a list of dependency package VERSIONS? That would make it so much easier, rather than trial and error.

 

Cheers.

 

Grant

[Rolf]

Hi Grant,

 

+1 on the GPU being of minimal (if any) help during the alignment or stacking aspects. I've been a lot of years in the software industry, including graphical applications and these two operations in particular just don't seem to be well-attuned to typical GPU capabilities.  Raw CPU, yes. Even multi-processors, multi-cores or multi-threads.  But I just don't see how anyone could utilize the GPU for this.

 

Keep going, Rolf - I'd love to play with a version I can get to run.

 

Any possibility of publishing at least a list of dependency package VERSIONS? That would make it so much easier, rather than trial and error.

 

Cheers.

 

Grant

 

I agree that multi-cores / multi-threads are important. Since I'm using Numpy and OpneCV for all compute intensive tasks, my code is using multithreading implicitly. I will look into ways to speed up the performance critical code sections further. But before that I want to complete a first working version of the entire SW.

 

Unfortunately I did not create a virtual Python environment with exactly the modules required by PSS. In my above post of January 13, however, you see a list of packages and their versions which I use to run PSS.

 

All the best,

 Rolf

[Rolf]

Hi,

 

Last weekend I finished the integration of postprocessing with the PlanetarySystemStacker software. It works as I described in my post of May 4 above.

 

Now that everything was in place, I performed the first large-scale test: I created the Windows installer on my development PC and moved it to a PC with no Python or other programming environment installed. There I ran the Windows installer (to test how it would work at your site). Then I went to a directory with AVI files of a complete moon panorama, 75 files in total with about 1800 frames each. I let PSS process all those 75 files in fully automatic batch mode, and afterwards I batch-post-processed all 75 summation files for sharpening. My PC needed 91 minutes in total for stacking the 75 videos. Sharpening the 75 images just took 24 seconds to complete.

 

Finally, I copied the 75 sharpened images into a folder, together with the corresponding reference files produced with AutoStakkert!3 and AviStack2 (for wavelet sharpening). For a fair comparison I had chosen similar numbers of alignment points in both programs. Now, switching back and forth in an image viewer, for every panorama tile I could directly compare both versions. The result: For all tiles the PSS version was consistently better than the reference version. There were much fewer artifacts (like vertical banding, checkboard-like patterns, distortions of the moon rim, gray patches outside the rim), and in many places the definition of small detail was better (in particular in craters with shadows on the terminator). To make sure that this was not just my biased opinion, I showed the images to friends who confirmed my observations.

 

I produced this panorama from the 75 tiles. The very first panorama produced with my own software! I was so relieved to see that everything worked as expected: No program crash, not even a warning, even in those cases where the moon only covered a small fraction of the view, no artifact on the moon rim caused by improper frame alignment etc.

 

Yesterday I started writing the user guide. I will publish the Windows installer only when that is finished, so I have to ask you for some more patience. Stay tuned!

 

All the best,

 Rolf

[airscottdenning]

Rolf, wow incredibly impressive! If you want a "friendly user" to try this out on non-windows platforms (linux and/or Mac), please let me know. I'm happy to install the underlying software support and libraries. 

 

Really delighted to see this happening!

[aeroman4907]

Looks pretty good Rolf, particularly the seamlessness of the panorama.  Since I presume you are looking for critical feedback, the one thing that jumped out to me was the noise level which was fairly strong.  If I read correctly you used 1800 frames each stack, the noise level seems quite high for well lit, or even shadow areas of the moon.  Perhaps that is also associated with the sharpening method?

[jp_oc]

Well I was able to get PSS 0.5.0 running on MacOS 10.13. It was pretty straight forward, just a Python 3 install and a bunch of libraries:

PyQt5
numpy
matplotlib
	-add backend: TkAgg to ~/.matplotlib/matplotlibrc
scipy
scikit-image
opencv

I can start the main program "planetary_system_stacker.py", modify the config, load the "short_video.avi" test file and start processing but it crashes:

**************************************************************
17-47-24.0 Start processing /Users/joe/Desktop/short_video.avi
**************************************************************
17-47-24.0 +++ Start ranking images +++
17-47-25.6 +++ Initializing frame alignment +++
17-47-25.6 +++ Start aligning all frames +++
17-47-25.8 +++ Start computing the average frame +++
2019-05-13 17:47:25.947 python[46205:14881093] -[QNSApplication _setup:]: unrecognized selector sent to instance 0x7fbfcb7703b0

Any ideas? I'd love to see this running on MacOS. There aren't any good planetary stackers for Mac.

 

Joe

[RedLionNJ]

Looks pretty good Rolf, particularly the seamlessness of the panorama.  Since I presume you are looking for critical feedback, the one thing that jumped out to me was the noise level which was fairly strong.  If I read correctly you used 1800 frames each stack, the noise level seems quite high for well lit, or even shadow areas of the moon.  Perhaps that is also associated with the sharpening method?

Not trying to be the proverbial "nasse decke" here, but I kind of have to agree with aeroman4907 - with up to 1800 frames per individual tile, I'd be expecting a far superior signal-to-noise ratio on the moon. is ti possible the individual tiles are only composed of a handful of stacked frames? Or perhaps the exposure/gain was extremely low during capture?

 

Just curious, more than anything. There is no doubt the mosaic worked out seamlessly.

[Rolf]

Hi,

 

Looks pretty good Rolf, particularly the seamlessness of the panorama.  Since I presume you are looking for critical feedback, the one thing that jumped out to me was the noise level which was fairly strong.  If I read correctly you used 1800 frames each stack, the noise level seems quite high for well lit, or even shadow areas of the moon.  Perhaps that is also associated with the sharpening method?

The videos are some 1800 frames each, of which I stacked 10%, i.e. about 180 frames. I admit that at 100% zoom the picture is a little grainy. Probably I should have set the sharpening level a little lower.

 

The point, though, is that I didn't want to impress you with a super-high-quality panorama. I just wanted to compare my program with AutoStakkert!3. Therefore, I used the same stacking percentage (10%) as I had used with the other program before, and I tried to set the sharpening control as similar as possible to what I had done with AviStack2 before. In a direct comparison between the resulting tiles (before panorama construction) the noise level and the amount of other artifacts are lower with PSS than with AS3/AviStack2. For me this was extremely good news, since it shows that my algorithms do their job pretty well.

 

All the best,

 Rolf

[Rolf]

Hi Joe,

 

Well I was able to get PSS 0.5.0 running on MacOS 10.13. It was pretty straight forward, just a Python 3 install and a bunch of libraries:

PyQt5
numpy
matplotlib
	-add backend: TkAgg to ~/.matplotlib/matplotlibrc
scipy
scikit-image
opencv

I can start the main program "planetary_system_stacker.py", modify the config, load the "short_video.avi" test file and start processing but it crashes:

**************************************************************
17-47-24.0 Start processing /Users/joe/Desktop/short_video.avi
**************************************************************
17-47-24.0 +++ Start ranking images +++
17-47-25.6 +++ Initializing frame alignment +++
17-47-25.6 +++ Start aligning all frames +++
17-47-25.8 +++ Start computing the average frame +++
2019-05-13 17:47:25.947 python[46205:14881093] -[QNSApplication _setup:]: unrecognized selector sent to instance 0x7fbfcb7703b0

Any ideas? I'd love to see this running on MacOS. There aren't any good planetary stackers for Mac.

 

Joe

It is very difficult for me to do any troubleshooting on a system I have no access to. And I do not have any experience with MacOS. I googled the error message "Python unrecognized selector sent to instance", though, and the first hit was a report on a Matplotlib problem on MacOS. So, my best guess is that your problem is related to that one. In my code, I use Matplotlib in a couple of places.

 

All the best,

 Rolf