Jump to content

  •  

CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.

Photo

More RAM or bigger SSD, PI hanging up

  • Please log in to reply
31 replies to this topic

#1 mrpizza

mrpizza

    Vostok 1

  • *****
  • topic starter
  • Posts: 128
  • Joined: 30 Apr 2010
  • Loc: Palm Coast, FL

Posted 21 February 2020 - 06:25 PM

I recently bought a QHY 163m and the files are 31MB each.  I was using PI subframe selector on 100 frames and it hung up to the point I had to do a hard shutdown.  I have a dual boot (Win 10 / Unbutu) machine and it did this with each one.  It has  AMD Ryzen 5 1600 with 16G RAM.  I have 8G RAM set up as RamDisk for swap storage directories and have a 120G SSD  for download directory although  that disk only has 67G of free space.  Attached is benchmark for the Linux OS and I think I have everything dialed in.  My question is:  in order to eliminate the system from hanging up  should I add 16G more RAM or get a 500G SSD and replace the 120G one?

 

Chris

Attached Thumbnails

  • PI Benchmark_Linux.JPG


#2 Jim Waters

Jim Waters

    Soyuz

  • *****
  • Posts: 3,806
  • Joined: 21 Oct 2007
  • Loc: Phoenix, AZ USA

Posted 21 February 2020 - 06:37 PM

Try eliminating the RamDIsk and rerun in the full 16GBytes.  Ideally its best if you have more memory - 32 or 64 GBytes.

 

Post this to the PI Forum.


  • jdupton likes this

#3 Xplode

Xplode

    Vostok 1

  • -----
  • Posts: 121
  • Joined: 16 Nov 2011

Posted 21 February 2020 - 06:52 PM

16GB is a little low if you work with a lot of files, 32GB would be perfect for your system and 64GB would be way overkill.
My system has a 3950x and 64GB is ok with 2.5x the threads your 1600 can run, max PI memory isage i've seen is around 40GB when integrating 61mpix/120MB files.
Upgrading the SSD would help you a lot, a NVME SSD would be even better and give you a nice speed boost compared to any SATA SSD.

I suggest upgrading the SSD before the RAM.


  • bobzeq25 likes this

#4 dpastern

dpastern

    Apollo

  • -----
  • Posts: 1,316
  • Joined: 01 Jan 2009
  • Loc: Brisbane, Australia

Posted 22 February 2020 - 03:23 AM

My advice (and I'm no expert, but I have been doing reading on this very subject over the past few days):

 

1.  Run PI in Linux.

 

2.  Upgrade system RAM to 64GB if possible.

 

3.  Get NVME SSD drive m2 v4 if possible, and if your motherboard has the right PCI-E slot to support it.  Set up some tmpfs partitions on said NVME drive for your PI swap.  Larger the better imho.  

 

4.  Profit.

 

:-)

 

Cheers,

 

Dave



#5 vio

vio

    Vostok 1

  • -----
  • Posts: 153
  • Joined: 27 Apr 2011
  • Loc: Fort Lauderdale, FL

Posted 22 February 2020 - 10:27 AM

Seeing the test results, there is a 2s swap time, it is unclear if that was system swapping (of the process) or the PI using the swap partitions (I tend to think of it as cache, it depends on how PI implements this). My opinion is that you can reduce this by making use of the full 16GB instead of allocating some (6G I believe you had) of it as RAM drive. Of course, adding 16GB memory will make up (and give you plenty extra) for what you took away for RAM drive(s). As suggested, NVME type SSD will be a big improvement on disk performance, to the point that taking memory away to create RAM disks may not be worth doing. My ideal setup, based on limited tests on a Mac, would be (for a budget-friendly system): 8-12 core, 32GB memory, 1TB NVME main disk drive for system and data, plus 1 or even 2 small (250GB) NVME drives just as swap/cache in PI. Budget-friendly is a relative term here, see what makes sense for your current setup.


Sent from my iPad using Tapatalk
  • jdupton likes this

#6 Kaydubbed

Kaydubbed

    Mariner 2

  • *****
  • Posts: 250
  • Joined: 29 Sep 2018
  • Loc: Kansas City, Missouri

Posted 22 February 2020 - 10:50 AM

Is your memory clocked at an XMP profile? Or at 2133mhz JDEC spec?

#7 mrpizza

mrpizza

    Vostok 1

  • *****
  • topic starter
  • Posts: 128
  • Joined: 30 Apr 2010
  • Loc: Palm Coast, FL

Posted 22 February 2020 - 11:06 AM

Thanks for all the info.  I should mention that the Win10 OS is loaded on PCIe 3.0x2 NVMe stick with 67Gb free space.  The Linux is loaded on a regular SSD with 70Gb set aside as Temp.  I am going to take the easy road and get 2 more sticks of RAM (on sale at Newegg) to bump it up to 32Gb and see what happens.   I currently have DDR4 2400 clocked at 2933 with CPU overclocked at 3.9.   I'm sure a larger disk will be in the future though.  I do have 2-2Tb Seagate drives to hold the files while not being processed. 



#8 mrpizza

mrpizza

    Vostok 1

  • *****
  • topic starter
  • Posts: 128
  • Joined: 30 Apr 2010
  • Loc: Palm Coast, FL

Posted 04 March 2020 - 10:58 PM

Just a quick follow up. I doubled RAM and now have 32Gb. PI still hangs up even after increasing Ramdisk to 16Gb. I guess I get a bigger Nvme drive and see what happens.

Chris

#9 jdupton

jdupton

    Vanguard

  • *****
  • Posts: 2,262
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 04 March 2020 - 11:26 PM

Chris,

 

   Depending on the number and size of subs you are using in PI, it may be that windows is running out swap space (Virtual Memory). Note that this is completely distinct from the so-called PI Swap directories.

 

   To verify whether this is happening, open the Windows Task manager and click on the Performance tab. Next click on the Memory section and note the values of the Committed entry. (The values there will be larger than the amount of physical RAM you have installed.) From the Task Manager Options menu, select Always On Top. You should also open a Windows File Explorer window and select Computer or This PC (depending on version of Windows) to see the free space on your drives. Leave both of these windows open.

 

   Now, kick off your PI process of interest and switch to the Windows File Explorer window. You should now see the Windows File Explorer and Task manager windows with PI running behind them. Keep watch on both the free file space and the memory utilization values as PI operates. As PI works, you will note that your memory will fill up more or less completely. When that happens, you will then note that the Committed value of virtual memory starts to grow. At the same time, the free drive space will start to fill up. If you run out of drive space, then PI will either error out or possibly lock up.

 

   At the end of this experiment, you will know more about the source of the problem. If PI dies while there is still free drive space, you may need to both free up drive space and / or increase the size of the Windows Virtual Memory Swap file. This can be done using the Windows Performance Dialog of the Control Panel. If PI messes up before even filling up physical RAM, then something else is going on.

 

 

John


Edited by jdupton, 04 March 2020 - 11:40 PM.

  • dpastern likes this

#10 Noah4x4

Noah4x4

    Gemini

  • *****
  • Posts: 3,015
  • Joined: 07 Apr 2016
  • Loc: Colchester UK

Posted 05 March 2020 - 02:40 AM

I had similar symptoms when running my camera/rig at 4K UHD resolutions (but with different software). My seventh generation i5 and 4Gb RAM would crash my camera software. I increased to 8 Gb, then 16 Gb. This stopped the crashes, but my system still ran slowly. I suspect it didn't help that I had a HDD.

 

I upgraded to eighth generation i7 with 16 Gb and SSD drive and my system now works in true real-time. It cured a myriad of other trivial problems too! You may need to uograde, processor, RAM and SSD. The system diagnostic tools mentioned in the previous post should guide you.


Edited by Noah4x4, 05 March 2020 - 02:41 AM.


#11 mrpizza

mrpizza

    Vostok 1

  • *****
  • topic starter
  • Posts: 128
  • Joined: 30 Apr 2010
  • Loc: Palm Coast, FL

Posted 05 March 2020 - 11:06 AM

John,

I took your advise and managed to get a screen shot (on left) before the system froze. Two things stick out to me.  CPU immediately jumps to 99% and stays there and memory is not in danger of running out.  I'm not sure that memory is being utilized like I would have thought.  I'm running 110 subs (63mb each) through Local Normalization.  I suspect something else is going on because I got this error message (on right) before the freeze which I had to use my phone to take.

Attached Thumbnails

  • CaptureA.JPG
  • 20200305_102656.jpg


#12 Noah4x4

Noah4x4

    Gemini

  • *****
  • Posts: 3,015
  • Joined: 07 Apr 2016
  • Loc: Colchester UK

Posted 05 March 2020 - 02:18 PM

CPU hits 99% and then software crashes? Sounds identical to what happened to me until I upgraded RAM (see above).

 

An AMD Ryzen 5 1600 has a CPU benchmark of 12310, which is fast. 99% makes zero sense. However, the specification says “discrete graphics card required”. I wonder what that means? Might it be a clue? It does sound to me that you are running out of system resources. Unlikely to be the CPU, but what about graphics card and RAM? 



#13 mrpizza

mrpizza

    Vostok 1

  • *****
  • topic starter
  • Posts: 128
  • Joined: 30 Apr 2010
  • Loc: Palm Coast, FL

Posted 05 March 2020 - 02:44 PM

From what I understand PI does not require heavy duty GPU. I have a 2mb card.

Chris

#14 Noah4x4

Noah4x4

    Gemini

  • *****
  • Posts: 3,015
  • Joined: 07 Apr 2016
  • Loc: Colchester UK

Posted 05 March 2020 - 02:44 PM

By coincidence, and it could be a blind alley (for which I apologise!). I just read this is a non-astronomy forum. The comment in italics below relates to the capture and streaming of fast frame rate sports video (using PCS Pro, a cricket scoring platform), which is perhaps not unlike the processing load of what you are trying to achieve. 

 

Just note that PCS Pro does not have any specific hardware encoding support for AMD (yet) so you would be forced to use software encoding, unless your specific NVIDIA card covers your encoding needs. If for your example you had a GeForce GTX 1080, it supports a maximum of 2 concurrent sessions which is very important:

 

This might suggest that for AMD processors, you might need specific hardware encoding support and might also be limited as regards concurrent sessions. The earlier comment from the processor specification "....discrete graphics card required" might make sense in this context.

 

Begs the question, can your graphics card handle what you are attempting and if not, does the software concerned offer specific software encoding support for it? It might be worth checking with the vendor if AMD processors are adequately supported by software encoding within its software. Just my two cents. Sorry, if wildly off the mark!

 


Edited by Noah4x4, 05 March 2020 - 02:56 PM.


#15 astrohamp

astrohamp

    Explorer 1

  • -----
  • Posts: 63
  • Joined: 01 Jan 2020
  • Loc: Massachusetts

Posted 05 March 2020 - 04:08 PM

Read/write speed specs on SSDs vary between and within brands.  One place for a possible speed increase.  Doesn't hurt to up the capacity as well and if need be set SSD controller for speed/performance rather than longevity. 

You also may be defeating Dual-channel mode (AMD Ryzen 5 1600 wont do quad) benifits by allocating half RAM to virtual disk.  Are you using the fastest RAM you can?  Install a quad RAM kit if going large to maintain similar spec on all sticks.  Just don't set virtual disk more than half so ensure maximum bandwidth of the RAM memory. 

 

Two different RAM sets may push the second set into single channel mode. 

 

Also try to see what processes are hogging resources.  Have the monitor running before things come on line.  For me even the Resource monitor uses valuable 'space' so adds to my diagnostics dilemma at times.



#16 mrpizza

mrpizza

    Vostok 1

  • *****
  • topic starter
  • Posts: 128
  • Joined: 30 Apr 2010
  • Loc: Palm Coast, FL

Posted 05 March 2020 - 08:10 PM

I'm at a total loss.  I reduced RAMDISK to 1GB and same results. So that's 31Gb of RAM and still freezing at 20 subs (that's how many files get written into folder when it freezes)  I'm heading over to PI forum to see why I'm using 99.6% of CPU during operation.  I will update if there is a solution.

 

Chris



#17 jdupton

jdupton

    Vanguard

  • *****
  • Posts: 2,262
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 05 March 2020 - 10:04 PM

Chris,

 

   It is not unusual to for PixInsight to peg your CPU usage during LocalNormalization. I have a Ryzen 9 3950X 16 core CPU and that process along with ImageIntegration will peg all 32 Windows managed CPUs at 99%. I don't think you are seeing any sort of hardware problem. Was the first screen-shot done just as you let the run start or did it look much the same when the exception was encountered?

 

   To me, it does look like you may need to tweak your Virtual Memory settings but I am not sure that caused your problem although it is possible depending on how your Page Space is being managed.

 

   There are two key things I noticed in your screen shots. The obvious thing is that PI encountered a software exception in its code. I have to wonder if it is the same file that causes it each time you run? It barfed when processing Lum_30sec_1x1_a_0023_c_a_r.xisf as seen in the console shot. I would suggest running the LocalNormalization again after eliminating that sub from your processing list. (You can simply toggle it be inactive in the list -- Xed out.)

 

   The second thing I note is Windows is starting with a virtual memory pool of only 5GB more than your physical RAM. I would expect it be roughly double your RAM space. I would also suggest that you check the virtual memory configuration in Windows. To check how yours is set up and / or make changes, do the following:

  • Open the Windows Settings window.
  • In the search box at the top, type "performance" and press enter
  • Choose the option for "Adjust the appearance and performance of Windows"
  • In the Performance Options dialog that opens, select "Advanced"
  • In the Virtual Memory dialog, let us know what is says for each drive in your system.
    (You generally want to set your Windows Drive [where Windows is installed] to "System Managed". You can optionally set another (preferably fast drive but not RAM_Disk) to "System Managed" also.) Let us see your current settings before making changes for now.
  • Press Cancel for the Virtual Memory dialog. Also exit any other dialogs and programs that are running.

Depending on what you find and report back here, you may want to change the settings for Virtual Memory.

 

   Try rerunning without the Frame_0023 to see what happens first. Then also observe the settings or make a screen capture of the Virtual Memory settings. Maybe we can figure out what is going on.

 

 

John


Edited by jdupton, 05 March 2020 - 11:21 PM.


#18 Noah4x4

Noah4x4

    Gemini

  • *****
  • Posts: 3,015
  • Joined: 07 Apr 2016
  • Loc: Colchester UK

Posted 06 March 2020 - 02:08 AM

I have just Googled  'AMD Ryzen 5 1600 problems".

 

Both Toms Hardware Forum and AMD's own Community Forum report common problems whereby this CPU will inexplicaby be running at 100% and software crashes then occur.

 

The  expert responses  suggest the issue is typically due to using incompatible RAM, and/or running with RAM at the wrong speeds or similar CPU/RAM communication issues. Apparently, AMD Ryzen processors are more fussy than Intel, and the correct pairing CPU, graphics and RAM is sensitive. The fact that your virtual memory is low (RAMdisc?) all point to a RAM issue. Adding more RAM that is incompatible won't assist. I suspect whatever you have done with the RAMdisc might be the culprit.

 

Suggest you explore this challenge in those Forums as there appears to be some knowledge about these symptons there. 


Edited by Noah4x4, 06 March 2020 - 02:18 AM.


#19 mrpizza

mrpizza

    Vostok 1

  • *****
  • topic starter
  • Posts: 128
  • Joined: 30 Apr 2010
  • Loc: Palm Coast, FL

Posted 06 March 2020 - 05:08 PM

John, here is a screenshot of virtual memory allocation.

 

Noah, I'm using GEIL EVO Potenza ram that is designed for Ryzen and has been confirmed compatibable with ASUS B-350 Plus motherboard.  I never had this problem before.  I could work with 300 frames from my QHY8L that were 12mb. The large files (64Mb) are the new variable.

Attached Thumbnails

  • System.JPG

Edited by mrpizza, 06 March 2020 - 05:09 PM.


#20 jdupton

jdupton

    Vanguard

  • *****
  • Posts: 2,262
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 06 March 2020 - 05:50 PM

Chris,

 

   Your Virtual Memory set-up looks good. I would not change anything there. I expect the problem lies elsewhere.

 

   Have you tried to determine if the PI exception happens on the same frame each time? If so, have you looked at disabling that particular frame. I am wondering whether there is something about that frame that causes an error while parsing the data.

 

 

John



#21 mrpizza

mrpizza

    Vostok 1

  • *****
  • topic starter
  • Posts: 128
  • Joined: 30 Apr 2010
  • Loc: Palm Coast, FL

Posted 06 March 2020 - 06:30 PM

John,

 

I actually played with the settings but went back to original so that you could give me your opinion.  Interesting that I doubled the amount of subs that got processed before freezing.  First image is the new setting. I went from automatic to system manage. Second image is just prior to running process. 3rd picture is camera shot of screen after freeze.

 

Chris

 

Attached Thumbnails

  • systemnanagedsize.JPG
  • goeswithmanagessixe.JPG
  • 20200306_181756.jpg


#22 mrpizza

mrpizza

    Vostok 1

  • *****
  • topic starter
  • Posts: 128
  • Joined: 30 Apr 2010
  • Loc: Palm Coast, FL

Posted 06 March 2020 - 06:38 PM

Here is a better image of the frozen screen

Attached Thumbnails

  • 20200306_181756.jpg


#23 jdupton

jdupton

    Vanguard

  • *****
  • Posts: 2,262
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 06 March 2020 - 07:19 PM

Chris,

 

   I am running out of ideas. It appears from the screen shots you posted that the system is not running out of memory (neither physical RAM nor virtual memory). The new screen shots also show that it is not hanging on the same frame each time.

 

   Maybe the next step is to look more closely at RAM reliability as suggested by Noah4x4. Perhaps you can run the Windows Memory Diagnostic for a few cycles to see what it reports. If that seems clean, maybe even a run with MemTest86 or similar software could show something.

 

   (As an aside, I tend to trust the Windows Memory Diagnostic a little more for these sorts of errors. On my older home built system [i7-3930K overclocked, circa 2013], MemTest86 would throw an error about 25% of the time on one test If and Only If I ran the tests in a certain order. The system never failed under normal usage.)

 

   Anyway, if there is a subtle memory timing error that is code sequence dependent (not unheard of), then testing may show the problem. To run the Windows Memory Diagnostic in Windows 10, just use the task bar search function for that name. It opens a windows that allow you start the test. The test will reboot and run the dedicated memory test. After completion, the system will reboot again allow you to view the results.

 

 

John



#24 mrpizza

mrpizza

    Vostok 1

  • *****
  • topic starter
  • Posts: 128
  • Joined: 30 Apr 2010
  • Loc: Palm Coast, FL

Posted 06 March 2020 - 10:21 PM

John,

 

I ran the Windows Memory Diagnostic and there were no problems found. Do you recommend running it multiple times?  It was an hour.  I'm going to give MemTest86 a run.

 

 

Chris



#25 jdupton

jdupton

    Vanguard

  • *****
  • Posts: 2,262
  • Joined: 21 Nov 2010
  • Loc: Central Texas, USA

Posted 06 March 2020 - 10:26 PM

Chris,

 

   Since nothing was found on the single run, giving Memtest86 a try might be the next step. If it runs clean, I am not sure what to try next. 

 

   Have you also run something like CoreTemp to make sure you are not overheating the CPU? Or you can also try HWMonitor to see if some other system component may be getting too hot and causing issues.

 

   Run Memtest86 first to hopefully eliminate memory timing issues as a cause.

 

 

John


Edited by jdupton, 06 March 2020 - 10:27 PM.



CNers have asked about a donation box for Cloudy Nights over the years, so here you go. Donation is not required by any means, so please enjoy your stay.


Recent Topics






Cloudy Nights LLC
Cloudy Nights Sponsor: Astronomics