26 replies to this topic

[Dan Finnerty]

I'm planning a new Threadripper 3 system for running PI. Currently thinking of getting the 16-core 3950 chip. I'm looking in to DDR4 memory speeds and wondering what to get. Speeds range from 2133 to 4266 MT/s. Prices go up exponentially for the highest speeds.

 

So the question is how sensitive is memory speed for PI processing? Looking at the price range, 3600 MT/s seems reasonable, but is the price difference compared to 3200 worth it? Is it worth spending more for even higher speed memory?

 

At this point, I'm thinking of getting 64 GB of memory initially, with a possibility of upgrading to 128 GB eventually. It looks like all the new TRX40 mother boards provide 8 memory slots, so 4x16 chipsets seems to do the trick.

 

These chips and boards are brand new, so I doubt anyone has any real world experience yet, but has anyone tested out PI performance vs. memory speed on other systems with Threadrippers?

Edited by Dan Finnerty, 30 November 2019 - 08:22 PM.

[ImNewHere]

I have been running 64gb DDR4 at 2667mhz since I built this one I am on in the summer of 2015. The RAM doesn't seem to be the bottleneck. Right now it is my 6 core i7 that is the bottleneck, after that would be the SSD.

If you can go with more cores I'd say that is more worth it. I know my next build will be the 64 core threadripper as soon as it is released, looks like in January.

[D_talley]

If you want speed, go with Linux. Windows is a big bottle neck when it comes to PI.  I saw my fastest performance with my 56 core system using Linux. So if you need speed go with that. 

[Dan Finnerty]

I have been running 64gb DDR4 at 2667mhz since I built this one I am on in the summer of 2015. The RAM doesn't seem to be the bottleneck. Right now it is my 6 core i7 that is the bottleneck, after that would be the SSD.

If you can go with more cores I'd say that is more worth it. I know my next build will be the 64 core threadripper as soon as it is released, looks like in January.

Yew! The 32 core TR is $2k, that 64 core beauty is going to be very high rent! I’ll be interested in seeing benchmarks for it when it hits the streets. I wonder if there is a peak to the cores vs. PI speed curve. You will be one of the first to find out! 

[Dan Finnerty]

If you want speed, go with Linux. Windows is a big bottle neck when it comes to PI.  I saw my fastest performance with my 56 core system using Linux. So if you need speed go with that. 

So I’ve heard. My brother got a used dual core Xeon server. Windows was a dog relative to Linux for PI processing. Do you know why the performance difference?

 

There are a number of things that affect PI performance. I’m primarily interested in memory speed in this post. I have some other system configuration questions I’ll ask in follow on threads.

Edited by Dan Finnerty, 01 December 2019 - 03:28 AM.

[Oleg Astro]

Look at The PixInsight Benchmark

 

Current best result has Total time    17.17 s

Attached Thumbnails

• 

Edited by Oleg Astro, 01 December 2019 - 05:05 AM.

[brave_ulysses]

any details on your pc? what is your pi benchmark score?

 

nice website!

 

 

If you want speed, go with Linux. Windows is a big bottle neck when it comes to PI.  I saw my fastest performance with my 56 core system using Linux. So if you need speed go with that. 

[D_talley]

Thanks.  Here is the results from my Linux benchmark.  The fastest time reported above was 17 seconds.  Mine was 30 seconds with 2ghz cpus.  

 

Below the Linux performance is the Windows performance on the same computer.  Marked difference!

 

Benchmark version ...... 1.00.08
CPU Identification
CPU vendor ............. GenuineIntel
CPU model .............. Intel® Xeon® CPU E5-2683 v3 @ 2.00GHz
System Information
Platform ............... Linux
Operating system ....... 4.13.9-300.fc27.x86_64 #1 SMP Mon Oct 23 13:41:58 UTC 2017 GNU/Linux
Core version ........... PixInsight Core 01.08.05.1352 (x64)
Logical processors ..... 56
Total memory size ...... 62.800 GiB
Execution Times
Total time ............. 00:30.53
CPU time ............... 00:21.13
Swap time .............. 00:09.36
Swap transfer rate ..... 1771.613 MiB/s
Performance Indices
Total performance ...... 15408
CPU performance ........ 17913
Swap performance ....... 9812

Now here are the number from this same computer running under windows 10

Total performance ...... 4179
CPU performance ........ 10173
Swap performance ....... 1219

Marked increased in speed for the Fedora system.  Almost a 4 times speed increase in total performance.  Wow.  I also noticed that only 32 of the 56 logical processors were being used in Windows 10.  In linux all 56 processors were being used.  Too bad Photoshop can't be used in Linux.

 

My PI computer is a ASUS Dual XEON motherboard with 64gb ram. 

[Oleg Astro]

This is my experiment with different desktops:

 

*******************************************************************************
The Official PixInsight Benchmark version 1.0
Copyright © 2014-2015 Pleiades Astrophoto. All Rights Reserved.
*******************************************************************************

Benchmark version ...... 1.00.08
Input checksum ......... 2cd72b67e12fff2812ef5b5da054ab2a70a25e23
Serial number .......... J6R2R9S56X1Q8L4YTDUW57V9W2Q97JMQ

CPU Identification
CPU vendor ............. GenuineIntel
CPU model .............. Intel® Core™ i7-3930K CPU @ 3.20GHz

System Information
Platform ............... Linux / Ubuntu 19.04
Operating system ....... 5.0.0-36-generic #39-Ubuntu SMP Tue Nov 12 09:46:06 UTC 2019 GNU/Linux
Core version ........... PixInsight Core 1.8.8-2 (x64)
Logical processors ..... 12
Total memory size ...... 58.915 GiB

 

1) ****** iceWM + RAM disk 10GB ******

Execution Times
Total time ............. 00:51.00
CPU time ............... 00:48.01
Swap time .............. 00:02.96
Swap transfer rate ..... 5601.434 MiB/s

Performance Indices
Total performance ......  9223
CPU performance ........  7883
Swap performance ....... 31024

 

2) ****** LXQT + RAM disk 10GB *****

Execution Times
Total time ............. 00:51.10
CPU time ............... 00:48.10
Swap time .............. 00:02.96
Swap transfer rate ..... 5605.288 MiB/s

Performance Indices
Total performance ......  9206
CPU performance ........  7869
Swap performance ....... 31045

 

3) ***** Openbox + RAM disk 10GB ******

Execution Times
Total time ............. 00:51.21
CPU time ............... 00:48.21
Swap time .............. 00:02.98
Swap transfer rate ..... 5571.279 MiB/s

Performance Indices
Total performance ......  9185
CPU performance ........  7851
Swap performance ....... 30857

 

4) ***** Wayland + RAM disk 10GB ******

Execution Times
Total time ............. 00:51.26
CPU time ............... 00:48.22
Swap time .............. 00:03.01
Swap transfer rate ..... 5513.529 MiB/s

Performance Indices
Total performance ......  9177
CPU performance ........  7849
Swap performance ....... 30537

Edited by Oleg Astro, 02 December 2019 - 03:54 PM.

[brave_ulysses]

i have a dual cpu 2680 v2 that runs ~32 seconds; the fastest windows benchmark i saw was ~21 seconds

 

it may be sad, but i love watching all the cores working in htop!

[Dan Finnerty]

Oleg, that is outrageous swap performance. You show 10GB RAM disk. How are you setting up swap files within that RAM Disk.

 

I know that swap performance is only a part of total performance, and the benchmark test does not duplicate real interactive processing,  but it is nice to see a way to optimize that component of total performance.

 

Dwight, your Linux vs. Windows results are stunning. Looking at the benchmarks on the PI site, it looked to me like people were getting 25% to 50% improvements, while you are getting 400%. It seems out of family. Do you understand what might be driving your differences?

 

Another thing I noticed from the PI Benchmark site Oleg linked to, is that there is a significant improvement in going from 32GB to 64GB of memory, but relatively modest In the 4% to 8% range. And people were not consistent in noting how much total memory they used for RAM disk. So not enough information to draw any clear conclusions beyond "more is better" and "significant but not huge" improvements.

 

And there appears to be no mineable information on memory speed impact on PI performance. Guess I'll have to just get some really fa$t memory, and try benchmarking it at various speeds. I would still appreciate if anyone knows of any such tests. It appears to be a rather esoteric question at this point.

 

I'm still inclined to go for 64GB of 3600 MT/s memory as a compromise between speed and cost, but I have no technical basis for that choice yet. I'd love to go for 4000 MT/s memory, but that is getting nose bleed high on the cost scale.

[D_talley]

My thought on the marked speed difference between Linux and Windows is Linux was using all of the cores of my system, where windows 10 Pro was not.  I guess you have to tell Windows or windows based programs to use all available cores.  While Linux says lets party!!!

[D_talley]

I ran a test on a slightly faster computer with only 32gb ram, half of my first system.  Results were almost identical.  So the amount of memory did not make an impact on the benchmark.  This system only had 48 cores and the drop from 56 did not impact the processing of the benchmark. 

So I would not be too concerned about the memory. I would look at the number of cores and speed of the processors. 

 

*******************************************************************************
The Official PixInsight Benchmark version 1.0
Copyright © 2014-2015 Pleiades Astrophoto. All Rights Reserved.
*******************************************************************************

Benchmark version ...... 1.00.08
Input checksum ......... 2cd72b67e12fff2812ef5b5da054ab2a70a25e23
Serial number .......... X095B36TPE500H0IG59GJA03SJ5E4YY1

CPU Identification
CPU vendor ............. GenuineIntel
CPU model .............. Intel® Xeon® CPU E5-2690 v3 @ 2.60GHz

System Information
Platform ............... Linux
Operating system ....... 4.14.16-300.fc27.x86_64 #1 SMP Wed Jan 31 19:24:27 UTC 2018 GNU/Linux
Core version ........... PixInsight Core 01.08.05.1352 (x64)
Logical processors ..... 48
Total memory size ...... 31.293 GiB

Execution Times
Total time ............. 00:30.40
CPU time ............... 00:21.72
Swap time .............. 00:08.61
Swap transfer rate ..... 1924.620 MiB/s

Performance Indices
Total performance ...... 15475
CPU performance ........ 17424
Swap performance ....... 10660

*******************************************************************************

[Oleg Astro]

Oleg, that is outrageous swap performance. You show 10GB RAM disk. How are you setting up swap files within that RAM Disk.

 

How to Create and Use a Ramdisk on Ubuntu 18.04

 

Open the terminal an run:

sudo mkdir /mnt/ramdisk
sudo mount -t tmpfs -o rw,size=10G tmpfs /mnt/ramdisk

You also need to change the settings in PixInsight! (look at the attached picture)

 

P.S. "To unmount the RAMDISK, run the following command:"

sudo umount /mnt/ramdisk

Attached Thumbnails

• 

Edited by Oleg Astro, 03 December 2019 - 02:59 AM.

[Dan Finnerty]

Thanks Oleg, I'm bookmarking this post for future reference. I'm going to start with Win 10, but will experiment with Linux in a dual boot configuration when the system is up and running. The performance advantage for PI looks overwhelming.

[Cfreerksen]

I am of the mind that the benchmark in PI is pretty narrow minded. It doesn't take into account stacking like 100 subs. When I doubled my RAM (64GB now) the benchmark didn't even budge. In real life though the stacking and pre-processing script is running much faster. It's when you get to the available memory boundary and have to start swapping. Then everything slows way down. If you have more than enough ram then no swapping.

 

I am not convinced that a RAMdisk is helpful either. I would need to see it in practical use. The RAMdisk is created from RAM and served up through software. If you reserve a portion of your RAM to a RAMdisk then you have reduced that amount of RAM available to the program. Then the need to swap may arise. Now you go to swapping to the RAMdisk. The very RAM you took away in the first place. Now managed by your RAMdisk software. It sounds much more efficient to just let PI have the RAM.

 

I created extra swapfiles in PI. Six of them. One for each core as someone recommended. After my upgrade to 64GB nothing even goes to them so far. They are untouched (PI seems to leave the last used files there). Well so far. Maybe if I have a larger stack of subs.

 

The whole benchmark my be nice to see how fast the processor crunches the numbers but beyond that it is not effective in reflecting actual processing and how RAM availability affects processing time.

 

Chris

Edited by Cfreerksen, 06 December 2019 - 12:12 AM.

[Oleg Astro]

Chris,
my experience is that a large amount of RAM>=64GB  is needed for large mosaics.

For example, look at this mosaic The VST captures three spectacular nebulae in one image

It has size 93031 x 35412 px = 3294Mpx or 3.3Gpx.

 

But this image is the crop of full mosaic with dimensions 101761x67536 px = 6.872Gpx (see attached pictures).

Similar work needs a big RAM

 

So, we can use RAM disk for PixInSight, if our computer has RAM 32...64GB.

Attached Thumbnails

• 

Edited by Oleg Astro, 06 December 2019 - 05:13 AM.

[Cfreerksen]

BTW. I have 32GB on the classifieds for $60 if anybody wants to upgrade.

 

Chris

[Cfreerksen]

Chris,
my experience is that a large amount of RAM>=64GB  is needed for large mosaics.

For example, look at this mosaic The VST captures three spectacular nebulae in one image

It has size 93031 x 35412 px = 3294Mpx or 3.3Gpx.

 

But this image is the crop of full mosaic with dimensions 101761x67536 px = 6.872Gpx (see attached pictures).

Similar work needs a big RAM

 

So, we can use RAM disk for PixInSight, if our computer has RAM 32...64GB.

But does RAMdisk get you anything? You steal system RAM for the RAMdisk.

 

Chris

[Oleg Astro]

PI needs more special tests.

 

For example, operation ImageIntegration for 48 images (w=4524 h=3624) needs these times for my case:

1. 01:18.53 (or 78.53 s)  with one Swap storage directories in /tmp

2. 21.362 s  with twelve Swap storage directories in /mnt/ramdisk

 

We can see the big difference: 78.53 s vs 21.36 s.

 

On the other hand, operation StarAlignment has the modest difference:

1. 01:29.06 with one Swap storage directories in /tmp

2. 01:28.40 with twelve Swap storage directories in /mnt/ramdisk

 

But operation LRGBCombination showed the most "interesting" results:

1. 3.745 s with one Swap storage directories in /tmp

2. 3.746 s with twelve Swap storage directories in /mnt/ramdisk

Edited by Oleg Astro, 07 December 2019 - 07:29 AM.

[Dan Finnerty]

Based on the lack of feedback on memory speed being a significant factor in PI performance, I've gone ahead and ordered 128GB of DDR4 3600 GB/s memory/ 64GB would probably have been quite sufficient for the task, but I've decided to pull out all the stops and go for the ultimate system.

 

I ordered a Gigabyte TRX40 AORUS Extreme mother board. I'm awaiting availability of the 3970x 32 core Threadripper 3 chip. They seem to be backordered everywhere.

 

When I have the system up and running I will have a great platform for experimenting with various settings and their effect on performance. Then I can begin to answer my own questions.

 

Maybe I'll even process some real astrophotos.

 

It would be interesting to put together a more realistic real-world test case for benchmarking performance. The existing standard benchmark does seem to have its limitations, particularly in maybe overemphasizing swap performance on total system performance.

[Tony Finnerty]

Hey bro'.  I just stumbled across this thread.  I built my dual Xeon PI machine from dumpster parts plus a new mobo, case and power supply for about $2400.  It's slower than some Threadripper hot rods, but the price was right.  Specs at:

 

http://pixinsight.co...THYS98263QOJRX2

 

I installed 128 GB of used ECC RAM, carving out a 64 GB RAM  drive for swap.  Testing different numbers of swap storage directories in the RAM drive, 12 directories gave highest benchmark scores, with a swap transfer rate of 9346 MiB/s.  That's nearly 1 GiB/s.  Total performance index: 21287.

 

I can't wait to see how your no-holds-barred Threadripper PI box will perform.

[Dan Finnerty]

Yo, Tony, I was wonder if you were lurking hereabouts. That dustbin daemon sounds like it is a performance per dollar winner. Obviously I’m taking a little bit of a different route.

 

And its not quite no holds barred. I went for DDR4 3600 GBPs vs 4000 GBPs which gets very pricey. Other than that though, its higher end. I got a Samsung Evo Plus m.2 NVME drive which has 3500 Mbps Read/ 3200 Mbps write speeds, so a nice boost on write speed.

 

The MB has 3 nvme slots, which I will eventually populate and get away from HDs and SSDs entirely except for backup and bulk archive storage. 

 

Threadrippers are in short supply so no idea when they will come in and I can complete the build.

[brave_ulysses]

hi dan,

 

any thoughts of an nvme array for your linux setup? i setup a zfs/raidz array at work for a fileserver and am happy with the performance

 

some info here to see if it fits your use case:

 

https://itsfoss.com/what-is-zfs/

 

good luck; looking forward to your benchmark results!

[Dan Finnerty]

nvme is so fast I"ve not really thought of arraying it. The mother board can hold 3 of them, so maybe down the line after I've assessed speeds and where bottlenecks lie. With 128 GB of memory, I can afford to make a big ram disk and see how much that helps compared to scratch files on the nvme. It will be fun over time playing with the various components to see what matters. Memory speed for instance. I got 3600 Mbps DDR4; I can play with clocking it at various speeds to see how much that matters. Finally able answer my initial question.

 

The ZFS file system sounds very interesting. Robust! We still have some old Solaris workstations at work, I wonder what file system they are running. I will ask our SAs about that.

 

In the near-term, I'm installing Win 10 Pro. Once I have everything running there, I'll try Linux on a dual boot configuration.Probably Fedora as that is similar to the RHEL environment at work. I need to think about file systems then. Win default is NTFS. I'm presuming Linux can work with NTFS. Are there other file systems that Win 10 can use?