How do you set up your editing machine in terms of disks for maximum performance and reliability? (SSD's are left out here.)
This is a question that often arises and all too often one sees that initial settings are really suboptimal. These rules are intended to help you decide how to setup your disks to get the best response times. Of course the only disks in an editing machine must be 7200 RPM types or faster. No GREEN disks at all.
Rule 1: NEVER partition a disk. You may ask why? First of all, it does not increase disk space, it just allocates the space differently. However, the major drawback is that for a partitioned disk the OS must first access a partition table at the beginning of the disk for all accesses to the disk, thus requiring the heads to move to the beginning of the disk, then when it has gotten the partition info move to the designated area on the disk and perform the requested action. This means much more wear-and-tear on the mechanics of the disk, slower speeds and more overhead for the OS, all reducing efficiency.
Rule 2: Avoid using USB drives, since they are the slowest on the market. Do not be tricked by the alleged bandwidth of USB 2.0 advertisements, because is just is not true and remember that the alleged bandwidth is shared by all USB devices, so if you have a USB mouse, keyboard, printer, card reader or whatever, they all share the bandwidth. Stick to SCSI or SATA disks or e-SATA. If needed, you can use Firewire-800 or even Firewire-400 disks, but they are really more suited for backups than for editing.
Rule 3: Use at least 3 different physical disks on an editing machine, one for OS/programs, one for media and one for pagefile/scratch/renders. Even on a notebook with one internal drive it is easy to accomplish this by using a dual e-SATA to Express card connector. That gives you an additional two e-SATA connections for external disks.
Rule 4: Spread disk access across as many disks as you have. If you have OS & programs on disk C:, set your pagefile on another disk. Also set your pagefile to a fixed size, preferably somewhere around 1.5 times your physical memory.
Rule 5: Turn off index search and compression. Both will cause severe performance hits if you leave them on.
Rule 6: If the fill rate on any of your SATA disks goes over 60-70% it is time to get a larger or an additional disk.
Rule 7: Perform regular defrags on all of your disks. For instance, you can schedule this daily during your lunch break.
Rule 8: Keep your disks cool by using adequate airflow by means of additional fans if needed. You can use SMART to monitor disk temperatures, which should be under 35 degrees C at all times and normally hover around 20-24 C, at least in a properly cooled system.
Rule 9: If people want raid, the cheapest way is to use the on-board IHCR or Marvell chip, but it places a relatively high burden on the CPU. The best way is a hardware controller card, preferably based on the IOP348 chip. Areca ARC and ADAPTEC come to mind. 3Ware uses it's own chipset and though not bad, they are not in the same league as the other two. Promise and the like in the budget range are no good and a complete waste of money. Expect to spend around $ 800 plus for a good controller with 12 connectors internally and 4 e-SATA connectors. Important to consider in a purchasing decision is whether the on-board cache memory can be expanded from the regular 256/512 MB to 2 or even 4 GB. Be aware that 2 GB cache can be relatively cheap, but the 4 GB version extremely costly ($ 30 versus $ 300). For safety reasons it is advisable to include a battery backup module (BBM).
Rule 10: If you can easily replace the data in case of disk failure (like rendered files), go ahead and use raid0, but if you want any protection against data loss, use raid 3/5/6/10/30/50. For further protection you can use hot spares, diminishing downtime and performance degradation.
In general when you get a new disk, pay close attention to any rattling noise, do perform regular disk checks, and in case of doubt about reliability, exchange the disk under guarantee. Often a new disk will fail in the first three months. If they survive that period, most of the disks will survive for the next couple of years. If you use a lot of internal disks like I do (17), set staggered spin-up to around 1 second to lessen the burden on the PSU and improve stability.
Hope this helps to answer some basic questions. If not, let me know. Further enhancements and suggestions are welcome.
An excellent set of rules for beginners. Covers all the basics and is well presented.
Thank you for articulating this. Though I usually bend Rule 2, by editing to/from FW-800's on the laptop, you are correct.
Couple dumb questions...
I have CS3 with 32bit XP. I transferred 5 tapes (mini DV ) to my computer via FW and did a work print. Now, Josh ( owner of tape) has a new 64bit vista 7 machine.... There's no way to share an external 1T fw800 hard drive is there ? Where I load the work print from my computer to ext HD, he takes home and loads from ext HD to his machine ?
If it is possible to do despite diff OS, have you heard of a good fast 1T fw800 HD recently for storage ( but fast ) ?
OS version is irrelevant if all versions can R/W NTFS files. FW800 is usually only available with the use of an extra PCI card that supports FW 800. eSATA would be easier and faster. WD has a good range of external disks. LaCie and Iomega have a less than stellar reputation.
Thanks ! That's great news re: NTFS ext HD on different OS.. And thanks re: suggestion on brand value.
Great article on Raid too ! Copied to ms Word ( for the "book" )...spelled your name right this time
Thank you for the thorough exploration of disk issues. I assume that other posts in this forum discuss hardware and software choices, as well, but I was hoping to persuade you to expand your post to include such items as 32-bit vs. 64-bit OS, impact of processor speed, use of expanded cache, RAMdisks, bus speed, memory, etc.
I understand your proposal and see the benefit for a lot of people. But that is also a far more challenging task, because of the large number of variables that come into play. I'll consider it, but make no promises.
I've been so impressed by both of these articles that I have spread some links around. I hope that you do not mind. I feel that your efforts can help so many, even if they only extrapolate some of the info to fit their needs and their budgets.
Can you elaborate on SMART to monitor disk temperatures ?
I'm assuming it's a software tool that can interface to the Controller/Drive and gather info.
Is it specific to a standard, or the drive maker ?
SMART (System for the Mechanical Analysis and Retrieval of Text) is a standard feature on modern day disks. It allows some programs like Speedfan or HWMonitor to report the temperatures of various disks. Maybe even more important is the possibility to determine the relative health of a disk, because the SMART data also contain information about read, write or seek errors, reallocated event and sector count, and stuff like that. This info can show whether the disk is deteriorating, losing its fitness and indicate that a replacement disk may be needed shortly.
There is no user interaction needed. SMART is always available for programs to access.
The screenshot below shows you how HWMonitor reports those temperatures.
"S.M.A.R.T. stands for Self-Monitoring Analysis and Reporting Technology"
Harm, thanx for making me smart, about SMART (pun intended).
Speedfan or HWMonitor seem to be another set of tools to be consider for my toolbag.
I'm running Raid 0 across a pair of 500gig drives, and an i7Core which is not OC'ed.
At some point I may bump up the i7-940 CPU, and these addition tools are good to know about...
I downloaded the Shareware (Free) version of HDMonitor.
The freebee only shows stats on the Core's and Graphics card.
If you want the whole enchilada (Disks, Fan, etc.), looks like you need HDMonitor Pro @ $19.95.
A small price to pay if you need this type of tool and fuctionality...
Does it show temperatures like in the above picture? BTW, this is version 1.14 but for me the results with 1.13 were the same.
Speedfan also gives information on hardware raids.
Harm, I uninstalled the Shareware HDMonitor and rolled back my Sys Image with RollBack-rx, so I can't check the version (and the install file was also rolled back).
I'll find it online and re-check when time permits.
The freebee only diplayed my four (4) Core temps and the Nvidia stats.
Nothing on Drives, Fans, etc.
The app is was pretty straight forward and nothing else was available on the Tree, or with View Options.
It's not a showstopper, unless the Version I used has some issues with my Raid 0 setup, but that would not explain why no Fans in the Tree...
Check in the BIOS whether SMART is maybe turned off.
You must be the patron saint of NLE systems, can you bilocate and help me set up my computer.
If you start your own thread, I'll probably find it and help you out if I can.
Thanks for the suggestion to do a backup guide. I'll keep it in mind and make no promises yet, but one can hope...
Thanks Harm, actually our IT dept is probably going to co-ordinate it and they have a contract through Dell. Up till now I've been spoiled and had my systems built for me by IT staff knowlageable in NLE systems. In my current job the IT staff haven't a clue about video and so here I am.
I have a question reg. storage. Would you pay the extra and get WD RE2 or RE3 serverdisks for your editingrig? I know I should not get "green" ones, but in your oppinion, is the serverdisks worth the extra cash?
This is a purely personal and partly budget driven answer, but with my 16 1TB disks I did not consider it worthwhile to go for the RE disks. There is no performance advantage, there is a price disadvantage and there is a longer guarantee, but with 16 disks the price difference alone would have been enough to buy 8 extra disks in case of failure outside the guarantee just to break even. So my point of view, no, it is not worth the higher price.
Where I now have 16 disks, I could have gotten only 9 RE disks for the same money. That difference was not worth it IMO.
Thanks Harm. 🙂
I'm confused as to why most motherboards (bios) support smart, but it is usally turned off by default. I once read that smart slows down/takes resources?