Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

New PC build. Should I have a separate boot drive?

  • Please log in to reply
26 replies to this topic

#1 BobNY

BobNY

    Ranger 4

  • *****
  • topic starter
  • Posts: 322
  • Joined: 30 Nov 2004
  • Loc: Long Island, NY

Posted 22 January 2020 - 09:52 AM

I am building new PC for use with Pixinsight. For the past 20+ years I have used off the shelf computers with one drive and loaded it up with everything. I bought a 1T Samsung 970 Evo+ and a 2T HDD for the new build. . My plans were to run a 1T SSD for my OS and all programs, then use the 2T to store all my Pixinsight files after I am done with processing. I have been reading that people use dedicated boot drives for their OS and programs and use a second SSD as a swap drive while working on images. Do I want to do this? In a setup like this, do you still load all programs to the C drive, or do you keep that just for the OS and install programs on the second SSD? Im starting the build today and I am trying to decide if I should go get another SSD before I get too far into the build. 

 

Thanks 



#2 Xeroid

Xeroid

    Surveyor 1

  • *****
  • Posts: 1,901
  • Joined: 21 Sep 2013
  • Loc: USA

Posted 22 January 2020 - 10:02 AM

BobNY

 

Your plans  "..run a 1T SSD for my OS and all programs, then use the 2T to store all my Pixinsight files"  are reasonable.

 

FYI: Usually Boot drives do not need to be very large as USER DATA / Programs are often stored on a "D" drive in case one has to "refresh" the boot drive.

 

Insuch a case above, you would then use an "E" 2T drive for your image files.

Easier to Backup the above C and D drives


Edited by Xeroid, 22 January 2020 - 10:05 AM.

  • BobNY likes this

#3 Jim Waters

Jim Waters

    Skylab

  • *****
  • Posts: 4,420
  • Joined: 21 Oct 2007
  • Loc: Phoenix, AZ USA

Posted 22 January 2020 - 10:45 AM

There are long posts on CN and the PI Forum site on PC build recommendations.  Do a search.



#4 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 21,035
  • Joined: 27 Oct 2014

Posted 22 January 2020 - 10:47 AM

I am building new PC for use with Pixinsight. For the past 20+ years I have used off the shelf computers with one drive and loaded it up with everything. I bought a 1T Samsung 970 Evo+ and a 2T HDD for the new build. . My plans were to run a 1T SSD for my OS and all programs, then use the 2T to store all my Pixinsight files after I am done with processing. I have been reading that people use dedicated boot drives for their OS and programs and use a second SSD as a swap drive while working on images. Do I want to do this? In a setup like this, do you still load all programs to the C drive, or do you keep that just for the OS and install programs on the second SSD? Im starting the build today and I am trying to decide if I should go get another SSD before I get too far into the build. 

 

Thanks 

I use two SSDs, I think most who use SSD for swap do.  It's not so much for the OS as it is for the swap files.  I don't really care if the OS drive has other things on it, I try to keep the swap drive (500 GB Samsung 970PRO) clean.  I use a 1 TB Samsung 970Plus for the OS and general files.  A 4 TB spinning drive for archiving.  It's a dedicated PI machine, although I have Teamviewer on it, in case I want to run the observatory computer remotely.  Usually use my general purpose desktop for that.

 

An alternative many use is to have a very large amount of RAM, and use some as a RAM drive for the swap files.  When I built mine, RAM was incredibly expensive.

 

I don't _think_ there's much speed difference between the two alternatives.  The SSD has significantly more space, obviously.  The RAM is probably a bit faster.

 

Other things.  I think the single most important thing is cores/threads.  PI is very effectively multithreaded.  I use a 1950X Threadripper, 16/32.  Linux provides a useful speed boost, this was my first Linux machine.

 

Data here, somewhat hard to wade through.  Note that many cores and Linux dominate among the fastest machines.  There are a variety of strategies for swap.  I bench in the low 20s.

 

http://www.pixinsight.com/benchmark/


Edited by bobzeq25, 22 January 2020 - 10:51 AM.

  • BobNY likes this

#5 BobNY

BobNY

    Ranger 4

  • *****
  • topic starter
  • Posts: 322
  • Joined: 30 Nov 2004
  • Loc: Long Island, NY

Posted 22 January 2020 - 10:56 AM

I use two SSDs, I think most who use SSD for swap do.  It's not so much for the OS as it is for the swap files.  I don't really care if the OS drive has other things on it, I try to keep the swap drive (500 GB Samsung 970PRO) clean.  I use a 1 TB Samsung 970Plus for the OS and general files.  A 4 TB spinning drive for archiving.  It's a dedicated PI machine, although I have Teamviewer on it, in case I want to run the observatory computer remotely.  Usually use my general purpose desktop for that.

 

An alternative many use is to have a very large amount of RAM, and use some as a RAM drive for the swap files.  When I built mine, RAM was incredibly expensive.

 

I don't _think_ there's much speed difference between the two alternatives.  The SSD has significantly more space, obviously.  The RAM is probably a bit faster.

 

Other things.  I think the single most important thing is cores/threads.  PI is very effectively multithreaded.  I use a 1950X Threadripper, 16/32.  Linux provides a useful speed boost, this was my first Linux machine.

 

Data here, somewhat hard to wade through.  Note that many cores and Linux dominate among the fastest machines.  There are a variety of strategies for swap.  I bench in the low 20s.

 

http://www.pixinsight.com/benchmark/

Thanks Bob, Your posts in all the previous pixinsight PC threads I have searched and read have been tremendously helpful. In fact they were a big help in choosing the parts for my build. I went with the 16 core, 32 thread Ryzen 9 3950x for my processor.  I appreciate the help!



#6 DennisK

DennisK

    Vostok 1

  • -----
  • Posts: 128
  • Joined: 13 Jan 2019
  • Loc: Willowick, OH

Posted 22 January 2020 - 03:51 PM

I do DSLR photography as well as astro-imaging, I use 3 drives:

 

a boot drive for the OS and apps;

 

a scratch drive for data handling during processing;

 

a file storage drive.

 

 

My boot drive and scratch drive are SSD's, the data drive is a conventional hard drive.  Some people use a conventional drive for boot because once the system is up & running and the app(s) loaded, the speed is not as important.  Also, depending on the app, a lot of memory can minimize the use of the scratch drive - but this usually becomes a Ford/Chevy argument....


  • BobNY likes this

#7 rockethead26

rockethead26

    Fly Me to the Moon

  • *****
  • Posts: 6,910
  • Joined: 21 Oct 2009
  • Loc: Northern Arizona, USA

Posted 22 January 2020 - 07:05 PM

I do DSLR photography as well as astro-imaging, I use 3 drives:

 

a boot drive for the OS and apps;

 

a scratch drive for data handling during processing;

 

a file storage drive.

 

 

My boot drive and scratch drive are SSD's, the data drive is a conventional hard drive.  Some people use a conventional drive for boot because once the system is up & running and the app(s) loaded, the speed is not as important.  Also, depending on the app, a lot of memory can minimize the use of the scratch drive - but this usually becomes a Ford/Chevy argument....

I do this as well with one addition. Along with the SSD drive for the OP and apps, a second SSD for the swap drive and a RAID array of hard drives for data, I also have a third SSD with a clone of my OS/Apps drive. If I have an OS/apps drive fail, I simply swap on bootup, remove the failed drive, replace it and clone again. I reclone after every OS update or after the addition of a new app. A fail hasn't happened yet with SSD drives, but it did happen previously when the OS and apps were on a hard drive. It's good insurance.



#8 BobNY

BobNY

    Ranger 4

  • *****
  • topic starter
  • Posts: 322
  • Joined: 30 Nov 2004
  • Loc: Long Island, NY

Posted 23 January 2020 - 01:54 AM

Thanks for all of the advice everyone. I picked up a 2nd SSD today to use as a scratch drive. My set up will be the same as DennisK's. SSD boot drive, SSD scratch drive and HDD for storage. 



#9 jdupton

jdupton

    Mercury-Atlas

  • *****
  • Posts: 2,730
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 23 January 2020 - 02:17 PM

BobNY,

 

   There have been some good examples of system configurations and recommendations already. I will add my thoughts on the subject.

 

   I am currently building a new PC very similar to yours. I have it nearly assembled but am still waiting on the DRAM memory modules before I can boot it, install the OS and applications, and start my performance tuning. I am using the Ryzen 9 3950X, 64 GB of 3600 MHz DRAM, two 1 TB PCIe 4.0 4 lane M.2 NVMe SSDs all on an ASUS X570 MB. I also have a 6 TB slow (5400 rpm) backing HDD in the configuration.

 

   I am planning to use one M.2 SSD (the one driven directly from the CPU) as the C: Boot Drive. It will contain the Windows 10 Pro OS and all key programs and applications. The second M.2 SSD (driven from the X570 chipset) will contain my data which is used most frequently as well as sundry seldom used programs and utility applications. The slower, larger HDD will be used for "online backups" of data. This will contain image data from older or completed imaging projects. Finally, an external similarly sized HDD will be used for "offline backups" of older data and periodic backups.

 

   In my working career before retirement, I worked on design, bring-up, test, and debug of large very high performance computer systems (think supercomputer building blocks). The one take-away from those years is when it comes to performance, the key is getting data latency as low as possible. In many to most cases, a computer system that can get the data into the processor's cache fastest will win even against systems with faster CPUs but longer latency. I am assembling this system with those things and more in mind.

 

   In configuring the system for PixInsight, we need to make a distinction between PixInsight "swap directories" and OS system paging / swap files. PI uses the swap directories for temporary data storage during processing operations. With multiple directories defined, different execution threads can overlap storage I/O Operations better. These "swap files" are different from the Windows (or Linux or Mac) OS swap files. Those OS controlled swap files are used as "virtual memory" when the OS needs more RAM than is physically present in the hardware. As physical RAM fills up and applications require even more, the OS saves some of the lesser used RAM from running applications to the storage system (SSDs and HDDs) and uses the newly freed space to allocate the freed RAM to whatever thread is requesting use of more.

 

   These OS swap files can be fixed in size or can be managed by the OS to increase or free up storage automatically as needed to allow for virtual memory operation. Those size choices / limits can be changed or configured from the Control Panel in Windows. (PS: if you have ever gotten the message "Out of Memory" inside PI, it often means that not only was all physical RAM completely filled up but Windows had run out of storage space on your drives and could not supply the needed RAM request even by moving stuff from RAM to virtual memory. The out of memory condition can be helped by freeing up storage space on your drives if they are running low or by allowing the OS to use more available drive space for storage of virtual memory. The added space can be allocated across multiple drives.)

 

   I will note something about systems specifically intended to run PixInsight also. Many advocate RAM Disks for improving PixInsight Benchmark scores. They do and it works well. However, those swap files / directories are only used by PI. When they fill up as can happen on large Image Integrations, Local Normalization, and Drizzle Integrations, PI loses the advantage of parallelism for the normal temp files on your storage devices. At the same time, PI is likely allocating very large amounts of virtual memory to perform its processing functions. Once PI's need for RAM exceeds available physical RAM memory, then the OS will begin to page data out to storage and back in as needed. This adds additional loads to the storage bandwidth and negates some of the advantages of using RAM as temporary storage.

 

   On large integrations on some of my imaging data, I found PI ran through my current system's 32 GB of RAM very quickly. Had I used RAM Disk, it would have filled up even faster and begun swapping virtual memory even sooner. Once swapping of virtual memory starts, storage bandwidth to the PI swap directories drops. For small integrations, this is not a concern and RAM Disks can help a lot. As a counter-case in point, I once used the Windows Task Manager to watch virtual memory allocations as I integrated ~850 images of 134 MB each. As ImageIntegration worked, total allocated memory grew to over 150 GB. That was all being paged in and out of the Windows OS page space. RAM Disk can actually hinder processing speed in such cases if it consumes too much of the physical RAM. (In fact, it is possible that the PI swap directory files placed on a RAM Disk could be swapped out of RAM to storage anyway as virtual memory needs increase.)

 

   Back to how I will configure my new system -- here are my plans:

  • SSD #1 (CPU) = Windows OS, Key Programs, Windows Managed Swap File.
  • SSD #2 (X570) = Lesser used Programs, Windows Managed Swap File, Data storage
  • HDD (SATA-3) = Accessible backup data and / or seldom used data files.

    PixInsight Configuration:
  • SSD #1 = 8 to 12 swap directory entries
  • SSD #2 = 8 to 12 swap directory entries

  I will run both the PI Benchmark and a few medium size ImageIntegration runs to tune the PI swap directory counts. From past use of my older i7-3930K (6 core) system, my instinct is telling me that 12 may be close to the optimum number of PI swap directories on each SSD for this system but I will likely try as many as 16. It would be nice if we could allocate one per core (or thread) but even the SSD probably cannot handle that many I/O Operations even if it had enough transfer bandwidth.

 

   If I were building the system solely for PixInsight, I might be tempted to put the boot drive SSD on the X570 PCIe connections and run the Data SSD from the CPU PCIe bus. The thought there is that once the programs are loaded, lower latency to the data might squeeze a little more performance out the system. There are probably a lot of configuration tuning tricks that could be tried when tuning a system for a single program. I may be able to report initial results later next week.

 

 

John


Edited by jdupton, 23 January 2020 - 05:19 PM.

  • BobNY, bobzeq25 and lucam like this

#10 BobNY

BobNY

    Ranger 4

  • *****
  • topic starter
  • Posts: 322
  • Joined: 30 Nov 2004
  • Loc: Long Island, NY

Posted 25 January 2020 - 11:08 AM

BobNY,

 

   There have been some good examples of system configurations and recommendations already. I will add my thoughts on the subject.

 

   I am currently building a new PC very similar to yours. I have it nearly assembled but am still waiting on the DRAM memory modules before I can boot it, install the OS and applications, and start my performance tuning. I am using the Ryzen 9 3950X, 64 GB of 3600 MHz DRAM, two 1 TB PCIe 4.0 4 lane M.2 NVMe SSDs all on an ASUS X570 MB. I also have a 6 TB slow (5400 rpm) backing HDD in the configuration.

 

   I am planning to use one M.2 SSD (the one driven directly from the CPU) as the C: Boot Drive. It will contain the Windows 10 Pro OS and all key programs and applications. The second M.2 SSD (driven from the X570 chipset) will contain my data which is used most frequently as well as sundry seldom used programs and utility applications. The slower, larger HDD will be used for "online backups" of data. This will contain image data from older or completed imaging projects. Finally, an external similarly sized HDD will be used for "offline backups" of older data and periodic backups.

 

   In my working career before retirement, I worked on design, bring-up, test, and debug of large very high performance computer systems (think supercomputer building blocks). The one take-away from those years is when it comes to performance, the key is getting data latency as low as possible. In many to most cases, a computer system that can get the data into the processor's cache fastest will win even against systems with faster CPUs but longer latency. I am assembling this system with those things and more in mind.

 

   In configuring the system for PixInsight, we need to make a distinction between PixInsight "swap directories" and OS system paging / swap files. PI uses the swap directories for temporary data storage during processing operations. With multiple directories defined, different execution threads can overlap storage I/O Operations better. These "swap files" are different from the Windows (or Linux or Mac) OS swap files. Those OS controlled swap files are used as "virtual memory" when the OS needs more RAM than is physically present in the hardware. As physical RAM fills up and applications require even more, the OS saves some of the lesser used RAM from running applications to the storage system (SSDs and HDDs) and uses the newly freed space to allocate the freed RAM to whatever thread is requesting use of more.

 

   These OS swap files can be fixed in size or can be managed by the OS to increase or free up storage automatically as needed to allow for virtual memory operation. Those size choices / limits can be changed or configured from the Control Panel in Windows. (PS: if you have ever gotten the message "Out of Memory" inside PI, it often means that not only was all physical RAM completely filled up but Windows had run out of storage space on your drives and could not supply the needed RAM request even by moving stuff from RAM to virtual memory. The out of memory condition can be helped by freeing up storage space on your drives if they are running low or by allowing the OS to use more available drive space for storage of virtual memory. The added space can be allocated across multiple drives.)

 

   I will note something about systems specifically intended to run PixInsight also. Many advocate RAM Disks for improving PixInsight Benchmark scores. They do and it works well. However, those swap files / directories are only used by PI. When they fill up as can happen on large Image Integrations, Local Normalization, and Drizzle Integrations, PI loses the advantage of parallelism for the normal temp files on your storage devices. At the same time, PI is likely allocating very large amounts of virtual memory to perform its processing functions. Once PI's need for RAM exceeds available physical RAM memory, then the OS will begin to page data out to storage and back in as needed. This adds additional loads to the storage bandwidth and negates some of the advantages of using RAM as temporary storage.

 

   On large integrations on some of my imaging data, I found PI ran through my current system's 32 GB of RAM very quickly. Had I used RAM Disk, it would have filled up even faster and begun swapping virtual memory even sooner. Once swapping of virtual memory starts, storage bandwidth to the PI swap directories drops. For small integrations, this is not a concern and RAM Disks can help a lot. As a counter-case in point, I once used the Windows Task Manager to watch virtual memory allocations as I integrated ~850 images of 134 MB each. As ImageIntegration worked, total allocated memory grew to over 150 GB. That was all being paged in and out of the Windows OS page space. RAM Disk can actually hinder processing speed in such cases if it consumes too much of the physical RAM. (In fact, it is possible that the PI swap directory files placed on a RAM Disk could be swapped out of RAM to storage anyway as virtual memory needs increase.)

 

   Back to how I will configure my new system -- here are my plans:

  • SSD #1 (CPU) = Windows OS, Key Programs, Windows Managed Swap File.
  • SSD #2 (X570) = Lesser used Programs, Windows Managed Swap File, Data storage
  • HDD (SATA-3) = Accessible backup data and / or seldom used data files.

    PixInsight Configuration:
  • SSD #1 = 8 to 12 swap directory entries
  • SSD #2 = 8 to 12 swap directory entries

  I will run both the PI Benchmark and a few medium size ImageIntegration runs to tune the PI swap directory counts. From past use of my older i7-3930K (6 core) system, my instinct is telling me that 12 may be close to the optimum number of PI swap directories on each SSD for this system but I will likely try as many as 16. It would be nice if we could allocate one per core (or thread) but even the SSD probably cannot handle that many I/O Operations even if it had enough transfer bandwidth.

 

   If I were building the system solely for PixInsight, I might be tempted to put the boot drive SSD on the X570 PCIe connections and run the Data SSD from the CPU PCIe bus. The thought there is that once the programs are loaded, lower latency to the data might squeeze a little more performance out the system. There are probably a lot of configuration tuning tricks that could be tried when tuning a system for a single program. I may be able to report initial results later next week.

 

 

John

John,

 

Thank you for this reply. That all makes perfect sense. I have been building my system over the last few nights. It is all together and running with Windows 10 pro installed. I just need to spend some time cleaning up wires. I should have Pixinsight loaded tonight or tomorrow. I will definitely look at setting up the swap directories. I like your idea of spreading hem between the two drives. My experience with Pixinsight so far has been on my old Macbook pro with whatever defaults are loaded during install. I will do a benchmark after I get Pixinsight loaded on the new PC before I make any changes to the swap directories. I have a lot of reading to do about setting all of that up still. Thanks for the great explanation of how you are planning to configure your system. That helped me to understand things a lot better.



#11 BobNY

BobNY

    Ranger 4

  • *****
  • topic starter
  • Posts: 322
  • Joined: 30 Nov 2004
  • Loc: Long Island, NY

Posted 25 January 2020 - 09:42 PM

I installed PI on the new PC and played with swap directories and ran some benchmarks.

 

Here is the result just after install, before adding any more directories:

 

Default 1 Directory

*******************************************************************************
The Official PixInsight Benchmark version 1.0
Copyright © 2014-2015 Pleiades Astrophoto. All Rights Reserved.
*******************************************************************************

Benchmark version ...... 1.00.08
Input checksum ......... 2cd72b67e12fff2812ef5b5da054ab2a70a25e23
Serial number .......... YWDUTXR2MU4LF298IP0K3LOT9FP0B19P

CPU Identification
CPU vendor ............. AuthenticAMD
CPU model .............. AMD Ryzen 9 3950X 16-Core Processor

System Information
Platform ............... Windows
Operating system ....... Microsoft Windows 10 Pro
Core version ........... PixInsight Core 1.8.8-4 (x64)
Logical processors ..... 32
Total memory size ...... 31.949 GiB

Execution Times
Total time ............. 00:30.52
CPU time ............... 00:19.62
Swap time .............. 00:10.87
Swap transfer rate ..... 1524.482 MiB/s

Performance Indices
Total performance ...... 15413
CPU performance ........ 19292
Swap performance .......  8443

*******************************************************************************

 

Here is after setting it to 8 directories:

 

8 Swap Directories

*******************************************************************************
The Official PixInsight Benchmark version 1.0
Copyright © 2014-2015 Pleiades Astrophoto. All Rights Reserved.
*******************************************************************************

Benchmark version ...... 1.00.08
Input checksum ......... 2cd72b67e12fff2812ef5b5da054ab2a70a25e23
Serial number .......... DK96YGO82YW49WSW55688PKASVEVHYV0

CPU Identification
CPU vendor ............. AuthenticAMD
CPU model .............. AMD Ryzen 9 3950X 16-Core Processor

System Information
Platform ............... Windows
Operating system ....... Microsoft Windows 10 Pro
Core version ........... PixInsight Core 1.8.8-4 (x64)
Logical processors ..... 32
Total memory size ...... 31.949 GiB

Execution Times
Total time ............. 00:25.29
CPU time ............... 00:20.29
Swap time .............. 00:04.98
Swap transfer rate ..... 3328.291 MiB/s

Performance Indices
Total performance ...... 18598
CPU performance ........ 18655
Swap performance ....... 18434

*******************************************************************************

 

After that I went to 12 directories with no improvement. Then I added another 12 on my 2nd SSD and also saw no improvement so I went back to the 8 directories on 1 drive for now. This is without using any ram drives. I may play with that a bit, but I am happy with the result as it is I think. Now I have to get better at processing smile.gif

 

 

 

 



#12 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 21,035
  • Joined: 27 Oct 2014

Posted 26 January 2020 - 12:54 AM

Just FYI.  I'd guess (it's just that) you could get from 25 seconds to 20 with Linux.


  • BobNY likes this

#13 BobNY

BobNY

    Ranger 4

  • *****
  • topic starter
  • Posts: 322
  • Joined: 30 Nov 2004
  • Loc: Long Island, NY

Posted 26 January 2020 - 04:03 PM

Just FYI.  I'd guess (it's just that) you could get from 25 seconds to 20 with Linux.

 Yes, I took a look after you mentioned Linux performance earlier and it does seem that it tops the charts. I would like to try Linux someday, but I opted for the familiar Windows 10. I have added so many new pieces of equipment and software to learn my way around lately not having to learn a new OS is a good thing.

I have had some benchmarks dip down into the mid 24's. I can live with that for sure.

 

I bought a 34" 21:9 monitor to go with the new PC. Having all that screen space to work with in Pixinsight is awesome. Screen real estate was a definite issue when processing on my 15" MacBook Pro. 



#14 jdupton

jdupton

    Mercury-Atlas

  • *****
  • Posts: 2,730
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 26 January 2020 - 08:06 PM

BobNY,

 

   You are way ahead of me at this point. I have been struggling with problems on the new PC build. I'm on my sixth load of Windows 10 Pro. It keeps eating itself and then refusing to boot and also refusing to allow the installation media to repair the installed copy. The only option given is to reinstall from scratch. Just when I think I am making progress, it commits suicide again. I think I have narrowed it down to an incompatibility between the NVMe drives and SATA. If I disable SATA, the system seems pretty solid. I have found that Dell systems have run into this before but so far have found nothing about ASUS motherboard based systems having this particular problem.

 

   All that aside, I have had it running enough to install and benchmark PixInsight. The numbers I got were close to those you posted in Post #11 above. My swap numbers also peaked at eight directory entries. Seven entries was only slightly lower than eight but like you, adding more up to twelve didn't help at all. Also like you, adding more on the second SSD did not improve the result any.

 

   My Swap Bandwidth was reported at 3930 MB/s. I measured the SSDs as capable of higher. With two NVMe drives, I thought the Swap bandwidth should be almost twice as much. I have some thoughts about that and if I can get the system stabilized, I will try something a little different. My results were about ~19,800 total, ~21,000 CPU and ~18,500 Swap. I don't recall the total time.

 

   I am still running at the default memory speed of 2133 MHz. If I can get past the problems with NVMe and SATA coexisting, I will enable the XMP profile on the memory and try running at the full speed of 3600 MHz. That should raise the CPU number in the benchmark and also help the Swap number a little as well.

 

   Good luck on any further tests you do.

 

 

John



#15 BobNY

BobNY

    Ranger 4

  • *****
  • topic starter
  • Posts: 322
  • Joined: 30 Nov 2004
  • Loc: Long Island, NY

Posted 26 January 2020 - 10:18 PM

John,

 

My OS install wasn't without issue either. After telling myself 100 times not to install Window on the wrong drive I wend ahead and accidentally installed it on my HDD instead of my SSD. Then the install to my SSD somehow got messed up and wouldn't boot, or let me fix it. I ended up deleting all partitions on all drives from the screen where the windows install asks you pick your drive. After that I reinstalled and everything went well. 

 

As far as the benchmark performance. I'm sure with some tweaking I may be able to find another second or two. I have decided not to chase the top of the charts and go with things as they are. Although I am curious to see what adding swap directories to ram would do.

Even though I probably don't need it, I have been thinking of adding 32gb more Ram. Those 2 empty slots are bothering me :(.



#16 jdupton

jdupton

    Mercury-Atlas

  • *****
  • Posts: 2,730
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 26 January 2020 - 10:27 PM

Bob,

 

   Deja vu...

 

   I tried just reinstalling to the existing partitions a couple of times after Windows refused to boot anymore. That never seemed to last very long until it crashed again. After that, I started doing clean installs by going to the Command Prompt window from the installation media and manually deleting all partitions using the "diskpart" subsystem.

 

   For now, the system is stable but then I haven't yet partitioned the second SSD nor the SATA drive. I am almost afraid to do that lest I end up back with the no-boot, non-repairable installation again...

 

 

John



#17 BobNY

BobNY

    Ranger 4

  • *****
  • topic starter
  • Posts: 322
  • Joined: 30 Nov 2004
  • Loc: Long Island, NY

Posted 26 January 2020 - 11:00 PM

John,

 

I also had some issues with my SSD's not showing up in disk management after the first install. I tried to use diskpart to fix that using some instructions I found on line. I finally gave up and posted in the troubleshooting forum on pcpartpicker.com. I followed the advice of one of the replies and things worked perfectly. I cant tell you why it didn't at first, but once it was messed up it was tough to figure out how to get back to the beginning. 



#18 Dan Finnerty

Dan Finnerty

    Viking 1

  • *****
  • Posts: 910
  • Joined: 11 Sep 2011
  • Loc: Pasadena, CA

Posted 27 January 2020 - 12:04 AM

John,

 

My OS install wasn't without issue either. After telling myself 100 times not to install Window on the wrong drive I wend ahead and accidentally installed it on my HDD instead of my SSD. Then the install to my SSD somehow got messed up and wouldn't boot, or let me fix it. I ended up deleting all partitions on all drives from the screen where the windows install asks you pick your drive. After that I reinstalled and everything went well. 

 

As far as the benchmark performance. I'm sure with some tweaking I may be able to find another second or two. I have decided not to chase the top of the charts and go with things as they are. Although I am curious to see what adding swap directories to ram would do.

Even though I probably don't need it, I have been thinking of adding 32gb more Ram. Those 2 empty slots are bothering me frown.gif.

 

 

Bob,

 

   Deja vu...

 

   I tried just reinstalling to the existing partitions a couple of times after Windows refused to boot anymore. That never seemed to last very long until it crashed again. After that, I started doing clean installs by going to the Command Prompt window from the installation media and manually deleting all partitions using the "diskpart" subsystem.

 

   For now, the system is stable but then I haven't yet partitioned the second SSD nor the SATA drive. I am almost afraid to do that lest I end up back with the no-boot, non-repairable installation again...

 

 

John

Misery loves company. Bob's story about installing to the wrong disk and follow-on problems sounds exactly like my experience with a new build. I had to wipe/repartition/reformat multiple disks several times. I swear windows never asked me where to install to, it just snatched whatever disk it wanted and started to work. I've since read instructions to ensure only one disk is mounted when installing. Now I know why.

 

So now I'm on my 4th install of Win10 Pro. It now seems stable. But I'm afraid to connect the other disks only to watch windows crash again. I've invested days and days trying to get all my s/w installed only to have Windows report a startup problem and then fail to perform a windows repair.

 

It now seems stable and I'm ready to reconnect everything else to get my data back. But first I'm making a disk clone and a recovery disk.



#19 BobNY

BobNY

    Ranger 4

  • *****
  • topic starter
  • Posts: 322
  • Joined: 30 Nov 2004
  • Loc: Long Island, NY

Posted 27 January 2020 - 01:04 AM

Misery loves company. Bob's story about installing to the wrong disk and follow-on problems sounds exactly like my experience with a new build. I had to wipe/repartition/reformat multiple disks several times. I swear windows never asked me where to install to, it just snatched whatever disk it wanted and started to work. I've since read instructions to ensure only one disk is mounted when installing. Now I know why.

 

So now I'm on my 4th install of Win10 Pro. It now seems stable. But I'm afraid to connect the other disks only to watch windows crash again. I've invested days and days trying to get all my s/w installed only to have Windows report a startup problem and then fail to perform a windows repair.

 

It now seems stable and I'm ready to reconnect everything else to get my data back. But first I'm making a disk clone and a recovery disk.

Geez. I am now feeling pretty lucky that everything went smoothly after I reinstalled. After the OS was installed both other drives were a breeze to set up in disk management. I played a bit in pixisight a bit tonight and this PC is a pleasure to use compared to my laptop. End result is it everything I was hoping it would be. I have a ton to learn about Pixinsight and processing. I could of coarse have done it with my laptop, but the added speed of the PC coupled with the ultrawide monitor made tonights learning session my far the most produce I have had to date. I was able to have Pixinsight up on the screen at the same time as tutorial videos and webpages. In the past I would have to minimize the screen to watch or read before my next step. Or have my imaging laptop on the desk also and use two laptops. One for processing and the other for the tutorials.

 

Here's a couple of pictures of the new set up. I had a bit of fun and went a little overboard on the whole RGB thing. That wasn't a thing when I build my last desktop in the early 90's smile.gif . Sadly its location on the kitchen table is only temporary. 

Attached Thumbnails

  • IMG_2267.jpeg
  • IMG_2270.jpeg


#20 macular hole

macular hole

    Vostok 1

  • -----
  • Posts: 175
  • Joined: 12 Aug 2009

Posted 27 January 2020 - 03:13 PM

Juan recommends the new thread ripper 3990 with max memory to maximize pix insight speed. However, it would appear that the 3950 keeps up with the threadripper 3970 when you look at the benchmarks for windows based systems. In fact, the 3950 posts the fastest times of all when running Linux. Moreover, a 3950 based system is significantly cheaper than a 3970 or 3990 threadripper system.

#21 bobzeq25

bobzeq25

    ISS

  • *****
  • Posts: 21,035
  • Joined: 27 Oct 2014

Posted 27 January 2020 - 04:09 PM

Juan recommends the new thread ripper 3990 with max memory to maximize pix insight speed. However, it would appear that the 3950 keeps up with the threadripper 3970 when you look at the benchmarks for windows based systems. In fact, the 3950 posts the fastest times of all when running Linux. Moreover, a 3950 based system is significantly cheaper than a 3970 or 3990 threadripper system.

For hundreds of subs the 3990x (64 cores/128 threads) will preprocess significantly faster.  128 subs at a time (!).

 

But I wouldn't pay for even 32 cores when I got mine.  <smile>



#22 jdupton

jdupton

    Mercury-Atlas

  • *****
  • Posts: 2,730
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 27 January 2020 - 05:43 PM

BobNY and others,

 

   I still don't have the Windows boot problem solved on my new system but do have a single drive installation running. Even with using only a single NVMe SSD, the system only boots into Windows once every three to five attempts. 

 

   Here is the system build configuration:
https://pcpartpicker.com/list/2ffgzN

 

   However, once booted, the system seems solid and stable. I have Installed the latest PixInsight and have it updated with all patches. Here are the results from last night's benchmark run. (23.45 Seconds; Total = 20,061; CPU = 20,257; and Swap = 19,364)

 

https://pixinsight.com/benchmark/benchmark-report.php?sn=EA6990JPQF51T202J4SA7S0V7M35BEP9

 

   I had one other result about 200 higher but the one above is a high average of what I am getting. The above result is with the one NVMe SSD only. I have not tried using a RAM Disk as I don't think that would help me on a real run with 500+ large subs as the RAM Disk itself might end up being paged out to the SSD anyway as I explained above. I think RAM Disk could possibly hurt the overall performance on large integrations once physical RAM is used up.

 

   I'll be making a post or two on other support forums to see if I can get some help making the system configuration stable for booting. I have all the drives connected right now but the SATA HDD is still un-allocated space and the second of the NVMe SSDs is still un-allocated also. It is essentially a one drive system with two un-partitioned drives attached but unused.

 

 

John



#23 BobNY

BobNY

    Ranger 4

  • *****
  • topic starter
  • Posts: 322
  • Joined: 30 Nov 2004
  • Loc: Long Island, NY

Posted 27 January 2020 - 08:55 PM

John,

 

Sorry to hear you are still having troubles. I hope you are able to get things straightened out soon. Here is the thread I posted when I was having trouble. Our issues were a bit different, but maybe worth a read incase there is anything in there that may help. 

https://pcpartpicker...on-new-pc-build

 

I ran few benchmarks today and they were in the 22's. It's funny how excited I can be by shaving off a quarter second of time.

I noticed the other Windows system in the benchmark list that is above our systems uses ram for it's swap directory. Our systems are very close in performance, with that other systems good amount ahead. I wonder if there is time to be gained moving the directories over to ram.

 

Then there are the Linux systems. As bobzeq25 pointed out.. They crush.


Edited by BobNY, 27 January 2020 - 08:57 PM.


#24 jdupton

jdupton

    Mercury-Atlas

  • *****
  • Posts: 2,730
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 27 January 2020 - 09:34 PM

BobNY,

 

   Thanks for the pointer to your thread. I will also post there to see what others say.

 

   Using RAM Disk for swap directories is great for improving Benchmark performance. I expect it would also work well for small integrations in real life also. So long as the overall size of your frames being integrated are less than your free memory space, RAM Disk should help quite a bit.

 

   However, I would argue that once you fill up all physical ram, the OS will start paging stuff stored in RAM out to disk as part of the virtual memory system. If part of what gets thrown out of RAM to the disk includes your RAM Disk in RAM, then you will actually lose performance rather than gain it. When PI needs to read a file from the temporary swap area, it will issue a read request for the RAM. If that RAM is no longer actually stored in RAM and is on the disk instead (having been paged out), then you have to not only wait for the disk to be read, but you also incur the penalty of waiting on something else in RAM being paged out to disk first to make room for the returning RAM Disk data. It could get ugly.

 

   It would be interesting to set up a RAM Disk and then also open the Task Manager and watch the physical and virtual memory usage during the benchmark. My guess is that for reasonable amounts of RAM, the benchmark code never uses enough RAM to trigger widespread paging. For a real life integration of a moderate number of frames, I know that PI can fill up RAM very quickly and begin allocating from virtual memory via the paging system. That's why I no longer use RAM Disk during real integrations. One of my large integrations allocated multi-hundreds of gigabytes of virtual memory. At one point or another nothing much lasted in in physical RAM during the integration. For bigger Benchmark numbers and smaller integrations, RAM Disk can help quite a bit. I'm skeptical whether it helps any at all when the integration needs 100 GB or more for processing.

 

   If I get some time, I will run such a test on the new system and see what happens.

 

 

John



#25 BobNY

BobNY

    Ranger 4

  • *****
  • topic starter
  • Posts: 322
  • Joined: 30 Nov 2004
  • Loc: Long Island, NY

Posted 28 January 2020 - 09:25 PM

John,

 

That all makes sense. I added another 32gb of Ram this evening and noticed that my ram was only running at 2666mhz. I changed the ram setting in the BIOS to get it running at proper speed. I ran the benchmark again and got a speed of 20:72. I think we may be just about at the max that we can expect to get out of a Windows machine. I hope you are able to get past your boot issues soon.


  • bobzeq25 likes this


CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics