Last week we shared Part I of III exploring Houdini Software System Requirements, we looked at how different hardware configurations performed rendering Flip, Grains and Pyro scenes with a focus on Houdini’s built-in render engine Mantra. Today we present Part II in the series, the role of cache disk and RAM in a Houdini workflow. Special thanks goes to the meticulous Curvin Huber for leading our testing and Houdini benchmarking.
Houdini Software System Requirements Test Configuration:
- a-X Mediaworkstation
- AMD 2950X 16 Core CPU (4.4 GHz Turbo)
- 32GB DDR4 3000 MHz RAM (and 64GB)
- 2x NVIDIA Geforce GTX 1080 Ti 11GB GPUs
- 512GB Samsung 970 Pro NVMe m.2 SSD (OS Drive)
- Microsoft Windows 10 Pro 64-bit
I’m testing the influence of hard drive speed and RAM capacity on Houdini simulation caches. Hardware acceleration is disabled for these tests thereby focusing the simulation calculations fully on the CPU and caching exclusively on the hard drive or RAM. The disk cache tests are being conducted on the Samsung 970 Pro M.2 drive. Disk caching is written using the File Cache node attached to the DOP I/O node and written to bgeo.sc format. In the RAM cache tests the cache memory is set to 80% of total RAM to avoid system crashes. This is being done in the DOP Network node’s Cache tab. The Compress .sim Files parameter is set to Biosc.
Houdini Disk Caching
Writing to disk (Cache) is being conducted on high resolution simulations to compile the longest data set. These results are for caching to the drive that also contains the OS. Disk caching is written using the File Cache node attached to the DOP I/O or DOP Import nodes and written to bgeo.sc format. Note: No GPU Acceleration is being used.
700K Grains
- 50 frame Disk Cache
- CPU 84-97%
- RAM 18%
- GPU 4%
- OS Drive 1- 4%
Total Cache Time: 6m 42s
31m Point FLIP
- 100 frame Disk Cache
- CPU 12-96%
- RAM 77-90%
- GPU 4%
- OS Drive 1-14%
- NOTE: Cache files nearly 1GB/frame
Total Cache Time: 47m 11s
10m Voxel Pyro
- 400 frame Disk Cache
- CPU 94-98%
- RAM 19%
- GPU 4%
- OS Drive 1-5%
Total Cache Time: 16m 47s
The most interesting observation is that the OS disk was never utilized above 14% for any of the caching tests. The highest utilization occurred with the FLIP simulation. As each frame written to disk was nearly 1GB, the disk performance spiked to 14% every time the sim frame completed its calculation in Houdini then wrote to disk. This corresponds to a peak write rate of approximately 322 MB/s – far below the 2,300 MB/s write capability of the 512GB Samsung 970 Pro, and roughly 65% the write capability of most SATA3 SSDs, though this is an approximately 50% greater data transfer speed than most traditional 7200 RPM Hard Drives are capable of.
Before testing we discussed benchmarking a 2x 970 Pro RAID0 cache disk array for these simulations, but this was unneeded: the amount of time it takes for Houdini to simply calculate the simulation is much longer than the amount of time it takes to write to disk. I also tested playback from the disk cache and there doesn’t seem to be any additional performance hits there either. This is surprising as I thought reading back from disk would tax the system somewhat but it didn’t.
The other interesting observation is that the FLIP simulation utilizes 77-90% of the RAM as it simulates and writes to disk. The Grain and Pyro tests do not seem to impact RAM usage at all.
Houdini Software System Requirements, RAM Caching: Comparing 32GB RAM vs. 64GB
Total Cache Memory is set to 80% of total RAM to avoid system crashes (25GB). This is done in the DOP Network node’s Cache tab. The Compress .sim Files parameter is set to Biosc. No GPU Acceleration will be used. Total frames written to RAM cache is the most important number to consider.
Houdini RAM Caching: 32GB RAM
700K Grains
- CPU 84-97%
- RAM 10% with idle file open
- GPU 1%
- OS Drive 1%
Total RAM Cache Time: 29m 21s (220 frames to max out 25GB of RAM)
31m Point FLIP
- CPU 12-96%
- RAM 45% with idle file open
- GPU 1%
- OS Drive 1%
Total Cache Time: 5m 38s (11 frames to max out 25GB RAM)
NOTE: Cache files nearly 1GB/frame as found in the disk caching tests.
10m Voxel Pyro
- CPU 84-98%
- RAM 10% with idle file open
- GPU 1%
- OS Drive 1%
Total Cache Time: 3m 47s (87 frames to max out 25GB RAM)
Houdini RAM Caching: 64 GB RAM
Total Cache Memory is set to 80% of total RAM to avoid system crashes (50GB). This is done in the DOP Network node’s Cache tab. The Compress .sim Files parameter is set to Biosc. No GPU Acceleration will be used. Total frames written to RAM cache is the most important number.
700K Grains
- CPU 84-97%
- RAM 5% with idle file open
- GPU 1%
- OS Drive 1%
Total RAM Cache Time: 58m 51s (440 frames to use 74% of 50GB of RAM)
Note: Doubling the RAM provides more than double the frame cache. This might be due, in part, to the grains coming to rest after frame 100.
31m Point FLIP
- CPU 12-96%
- RAM 24% with idle file open
- GPU 1%
- OS Drive 1%
Total Cache Time: 10m 04s (22 frames to max out 50GB RAM)
NOTE: Cache files nearly 1GB/frame as found in the disk caching tests.
10m Voxel Pyro
- CPU 84-98%
- RAM 5% with idle file open
- GPU 1%
- OS Drive 1%
Total Cache Time:7m 41s (164 frames to max out 50GB RAM)
Note: Not quite double the frames from 32GB but this is likely due to Pyro continuing to emit particles.
General Notes: Comparing disk cache files which range from 900k – 1GB per frame, these sizes added up determine how many frames can be cached to RAM.
Summary:
In summary, adding a faster hard drive does not provide much performance gain for caching simulations. Unlike 4K, 6K or 8K edit workflows for example where insufficient disk i/o can mean dropped frames, crashes or worse, Houdini simulations take much longer to calculate per frame than writing to disk. Therefore, testing shows that NVME m.2 and SATA SSD drive read/write speeds can easily keep up with the data being passed to it from Houdini on a frame by frame basis. However, Houdini cache files can become very large, especially in the case of FLIP simulations. Cached simulation can be upwards of 1GB or greater per frame, thus a large capacity hard drive would be the important factor to consider when purchasing or upgrading a system for Houdini.
Caching to RAM yields similar results. Adding more RAM (32GB to 64GB) essentially yields twice the caching capabilities. However, caching to RAM on complex simulations where frame by frame simulation times can exceed seconds or minutes in some cases, is not wise. RAM caches are temporary, as soon as a parameter is adjusted, the RAM cache is lost. Having an adequate amount of RAM to account for general processes is beneficial, however,when it comes to caching simulations, a very large hard drive is the most beneficial piece of hardware.
In Part III of our Houdini Software System Requirements series, we’ll dispel the hope (for the moment at least) of Mantra gpu rendering, and explore the best render engines for Houdini.
Houdini Software System Requirements – Houdini Recommended Configurations (Updated May 2020)
For pure Houdini workflows, the price for performance king is our a-X Mediaworkstation.
For peak performance in a pure Houdini workflow, the a-X2 Mediaworkstation is our fastest workstation for Houdini.
Special thanks to Curvin Huber for his thorough testing and Houdini benchmarking.
NOTE: It’s important to consider your top 2-3 applications when considering optimum configurations, as different applications talk to and use hardware differently. This is abundantly clear from a performance standpoint when using GPU render engines for Houdini.
Hey, What about different ram frequency and latency? Is there any “Houdini” difference from 2666 to 3000 or even 3600mhz? I want to upgrade my 3950x with 2x32gb ram but can I save some money buying a lower frequency ram like 2666mhz cl18?
Hi Francesco,
Having tested difference RAM speeds with many different content creation applications over the years, RAM frequency makes little difference in our experience. This is the case even with RAM-hungry applications like After Effects where RAM amount can make a performance difference. There’s almost NO difference whatever using higher frequency RAM with apps like Houdini which are RAM-efficient and happy with 32-64GB system RAM.
In fact, what we’ve seen is that higher frequency RAM often contributes to system instability and increased crashing, which is exactly what you don’t want as on-deadline content creators. Spec RAM for the platform/CPU is really what you want for long-term stability and performance. Re amount of RAM, it’s usually the other applications in a media pro’s Houdini workflow which drive the decision to go to 128GB RAM, or even 256GB RAM for content creation.