Highlighted

Storage rules for an editing rig. Some basics.

LEGEND ,
May 03, 2009

Copy link to clipboard

Copied

How do you set up your editing machine in terms of disks for maximum performance and reliability? (SSD's are left out here.)

This is a question that often arises and all too often one sees that initial settings are really suboptimal. These rules are intended to help you decide how to setup your disks to get the best response times. Of course the only disks in an editing machine must be 7200 RPM types or faster. No GREEN disks at all.

Rule 1: NEVER partition a disk. You may ask why? First of all, it does not increase disk space, it just allocates the space differently. However, the major drawback is that for a partitioned disk the OS must first access a partition table at the beginning of the disk for all accesses to the disk, thus requiring the heads to move to the beginning of the disk, then when it has gotten the partition info move to the designated area on the disk and perform the requested action. This means much more wear-and-tear on the mechanics of the disk, slower speeds and more overhead for the OS, all reducing efficiency.

Rule 2: Avoid using USB drives, since they are the slowest on the market. Do not be tricked by the alleged bandwidth of USB 2.0 advertisements, because is just is not true and remember that the alleged bandwidth is shared by all USB devices, so if you have a USB mouse, keyboard, printer, card reader or whatever, they all share the bandwidth. Stick to SCSI or SATA disks or e-SATA. If needed, you can use Firewire-800 or even Firewire-400 disks, but they are really more suited for backups than for editing.

Rule 3: Use at least 3 different physical disks on an editing machine, one for OS/programs, one for media and one for pagefile/scratch/renders. Even on a notebook with one internal drive it is easy to accomplish this by using a dual e-SATA to Express card connector. That gives you an additional two e-SATA connections for external disks.

Rule 4: Spread disk access across as many disks as you have. If you have OS & programs on disk C:, set your pagefile on another disk. Also set your pagefile to a fixed size, preferably somewhere around 1.5 times your physical memory.

Rule 5: Turn off index search and compression. Both will cause severe performance hits if you leave them on.

Rule 6: If the fill rate on any of your SATA disks goes over 60-70% it is time to get a larger or an additional disk.

Rule 7: Perform regular defrags on all of your disks. For instance, you can schedule this daily during your lunch break.

Rule 8: Keep your disks cool by using adequate airflow by means of additional fans if needed. You can use SMART to monitor disk temperatures, which should be under 35 degrees C at all times and normally hover around 20-24 C, at least in a properly cooled system.

Rule 9: If people want raid, the cheapest way is to use the on-board IHCR or Marvell chip, but it places a relatively high burden on the CPU. The best way is a hardware controller card, preferably based on the IOP348 chip. Areca ARC and ADAPTEC come to mind. 3Ware uses it's own chipset and though not bad, they are not in the same league as the other two. Promise and the like in the budget range are no good and a complete waste of money. Expect to spend around $ 800 plus for a good controller with 12 connectors internally and 4 e-SATA connectors. Important to consider in a purchasing decision is whether the on-board cache memory can be expanded from the regular 256/512 MB to 2 or even 4 GB. Be aware that 2 GB cache can be relatively cheap, but the 4 GB version extremely costly ($ 30 versus $ 300). For safety reasons it is advisable to include a battery backup module (BBM).

Rule 10: If you can easily replace the data in case of disk failure (like rendered files), go ahead and use raid0, but if you want any protection against data loss, use raid 3/5/6/10/30/50. For further protection you can use hot spares, diminishing downtime and performance degradation.


In general when you get a new disk, pay close attention to any rattling noise, do perform regular disk checks, and in case of doubt about reliability, exchange the disk under guarantee. Often a new disk will fail in the first three months. If they survive that period, most of the disks will survive for the next couple of years. If you use a lot of internal disks like I do (17), set staggered spin-up to around 1 second to lessen the burden on the PSU and improve stability.

Hope this helps to answer some basic questions. If not, let me know. Further enhancements and suggestions are welcome.

Views

103.9K

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more

Storage rules for an editing rig. Some basics.

LEGEND ,
May 03, 2009

Copy link to clipboard

Copied

How do you set up your editing machine in terms of disks for maximum performance and reliability? (SSD's are left out here.)

This is a question that often arises and all too often one sees that initial settings are really suboptimal. These rules are intended to help you decide how to setup your disks to get the best response times. Of course the only disks in an editing machine must be 7200 RPM types or faster. No GREEN disks at all.

Rule 1: NEVER partition a disk. You may ask why? First of all, it does not increase disk space, it just allocates the space differently. However, the major drawback is that for a partitioned disk the OS must first access a partition table at the beginning of the disk for all accesses to the disk, thus requiring the heads to move to the beginning of the disk, then when it has gotten the partition info move to the designated area on the disk and perform the requested action. This means much more wear-and-tear on the mechanics of the disk, slower speeds and more overhead for the OS, all reducing efficiency.

Rule 2: Avoid using USB drives, since they are the slowest on the market. Do not be tricked by the alleged bandwidth of USB 2.0 advertisements, because is just is not true and remember that the alleged bandwidth is shared by all USB devices, so if you have a USB mouse, keyboard, printer, card reader or whatever, they all share the bandwidth. Stick to SCSI or SATA disks or e-SATA. If needed, you can use Firewire-800 or even Firewire-400 disks, but they are really more suited for backups than for editing.

Rule 3: Use at least 3 different physical disks on an editing machine, one for OS/programs, one for media and one for pagefile/scratch/renders. Even on a notebook with one internal drive it is easy to accomplish this by using a dual e-SATA to Express card connector. That gives you an additional two e-SATA connections for external disks.

Rule 4: Spread disk access across as many disks as you have. If you have OS & programs on disk C:, set your pagefile on another disk. Also set your pagefile to a fixed size, preferably somewhere around 1.5 times your physical memory.

Rule 5: Turn off index search and compression. Both will cause severe performance hits if you leave them on.

Rule 6: If the fill rate on any of your SATA disks goes over 60-70% it is time to get a larger or an additional disk.

Rule 7: Perform regular defrags on all of your disks. For instance, you can schedule this daily during your lunch break.

Rule 8: Keep your disks cool by using adequate airflow by means of additional fans if needed. You can use SMART to monitor disk temperatures, which should be under 35 degrees C at all times and normally hover around 20-24 C, at least in a properly cooled system.

Rule 9: If people want raid, the cheapest way is to use the on-board IHCR or Marvell chip, but it places a relatively high burden on the CPU. The best way is a hardware controller card, preferably based on the IOP348 chip. Areca ARC and ADAPTEC come to mind. 3Ware uses it's own chipset and though not bad, they are not in the same league as the other two. Promise and the like in the budget range are no good and a complete waste of money. Expect to spend around $ 800 plus for a good controller with 12 connectors internally and 4 e-SATA connectors. Important to consider in a purchasing decision is whether the on-board cache memory can be expanded from the regular 256/512 MB to 2 or even 4 GB. Be aware that 2 GB cache can be relatively cheap, but the 4 GB version extremely costly ($ 30 versus $ 300). For safety reasons it is advisable to include a battery backup module (BBM).

Rule 10: If you can easily replace the data in case of disk failure (like rendered files), go ahead and use raid0, but if you want any protection against data loss, use raid 3/5/6/10/30/50. For further protection you can use hot spares, diminishing downtime and performance degradation.


In general when you get a new disk, pay close attention to any rattling noise, do perform regular disk checks, and in case of doubt about reliability, exchange the disk under guarantee. Often a new disk will fail in the first three months. If they survive that period, most of the disks will survive for the next couple of years. If you use a lot of internal disks like I do (17), set staggered spin-up to around 1 second to lessen the burden on the PSU and improve stability.

Hope this helps to answer some basic questions. If not, let me know. Further enhancements and suggestions are welcome.

Views

103.9K

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
May 03, 2009 1
143 Replies 143
LEGEND ,
May 03, 2009

Copy link to clipboard

Copied

An excellent set of rules for beginners.  Covers all the basics and is well presented.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 03, 2009 1
LEGEND ,
May 03, 2009

Copy link to clipboard

Copied

Thank you for articulating this. Though I usually bend Rule 2, by editing to/from FW-800's on the laptop, you are correct.

Appreciated,

Hunt

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 03, 2009 1
Guide ,
Apr 07, 2010

Copy link to clipboard

Copied

Bill,

Couple dumb questions...

I have CS3 with 32bit XP.  I transferred 5 tapes (mini DV ) to my computer via FW and did a work print.  Now, Josh ( owner of tape) has a new 64bit vista 7 machine....  There's no way to share an external 1T fw800 hard drive is there ?  Where I load the work print from my computer to ext HD, he takes home and loads from ext HD to his machine ?

If it is possible to do despite diff OS, have you heard of a good fast 1T fw800 HD recently for storage ( but fast ) ?

Thanks

Rod

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 07, 2010 0
LEGEND ,
Apr 07, 2010

Copy link to clipboard

Copied

Rod,

OS version is irrelevant if all versions can R/W NTFS files. FW800 is usually only available with the use of an extra PCI card that supports FW 800. eSATA would be easier and faster. WD has a good range of external disks. LaCie and Iomega have a less than stellar reputation.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 07, 2010 0
Guide ,
Apr 08, 2010

Copy link to clipboard

Copied

Harm,

Thanks !  That's great news re: NTFS ext HD on different OS..  And thanks re: suggestion on brand value.

Great article on Raid too !   Copied to ms Word ( for the "book" )...spelled your name right this time

Rod

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Apr 08, 2010 0
New Here ,
May 15, 2009

Copy link to clipboard

Copied

Thank you for the thorough exploration of disk issues.  I assume that other posts in this forum discuss hardware and software choices, as well, but I was hoping to persuade you to expand your post to include such items as 32-bit vs. 64-bit OS, impact of processor speed, use of expanded cache, RAMdisks, bus speed, memory, etc.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 15, 2009 0
LEGEND ,
May 15, 2009

Copy link to clipboard

Copied

I understand your proposal and see the benefit for a lot of people. But that is also a far more challenging task, because of the large number of variables that come into play. I'll consider it, but make no promises.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 15, 2009 1
LEGEND ,
May 15, 2009

Copy link to clipboard

Copied

Does this reflect your wishes: http://forums.adobe.com/thread/433549?tstart=0

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 15, 2009 1
LEGEND ,
May 16, 2009

Copy link to clipboard

Copied

Harm,

I've been so impressed by both of these articles that I have spread some links around. I hope that you do not mind. I feel that your efforts can help so many, even if they only extrapolate some of the info to fit their needs and their budgets.

Thank you,

Hunt

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 16, 2009 0
Enthusiast ,
May 16, 2009

Copy link to clipboard

Copied

Can you elaborate on SMART to monitor disk temperatures ?

I'm assuming it's a software tool that can interface to the Controller/Drive and gather info.

Is it specific to a standard, or the drive maker ?

Thanx...

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 16, 2009 0
LEGEND ,
May 17, 2009

Copy link to clipboard

Copied

SMART (System for the Mechanical Analysis and Retrieval of Text) is a standard feature on modern day disks. It allows some programs like Speedfan or HWMonitor to report the temperatures of various disks. Maybe even more important is the possibility to determine the relative health of a disk, because the SMART data also contain information about read, write or seek errors, reallocated event and sector count, and stuff like that. This info can show whether the disk is deteriorating, losing its fitness and indicate that a replacement disk may be needed shortly.

There is no user interaction needed. SMART is always available for programs to access.

The screenshot below shows you how HWMonitor reports those temperatures.

PC-Temps.jpg

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 17, 2009 0
LEGEND ,
May 17, 2009

Copy link to clipboard

Copied

"S.M.A.R.T. stands for Self-Monitoring Analysis and Reporting Technology"

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 17, 2009 0
Enthusiast ,
May 18, 2009

Copy link to clipboard

Copied

Harm, thanx for making me smart, about SMART (pun intended).

Speedfan or HWMonitor seem to be another set of tools to be consider for my toolbag.

I'm running Raid 0 across a pair of 500gig drives, and an i7Core which is not OC'ed.

At some point I may bump up the i7-940 CPU, and these addition tools are good to know about...

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 18, 2009 0
Enthusiast ,
May 19, 2009

Copy link to clipboard

Copied

I downloaded the Shareware (Free) version of HDMonitor.

The freebee only shows stats on the Core's and Graphics card.

If you want the whole enchilada (Disks, Fan, etc.), looks like you need HDMonitor Pro @ $19.95.

A small price to pay if you need this type of tool and fuctionality...

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 19, 2009 0
LEGEND ,
May 19, 2009

Copy link to clipboard

Copied

HWMonitor.jpg

Does it show temperatures like in the above picture? BTW, this is version 1.14 but for me the results with 1.13 were the same.

Speedfan also gives information on hardware raids.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 19, 2009 0
Enthusiast ,
May 19, 2009

Copy link to clipboard

Copied

Harm, I uninstalled the Shareware HDMonitor and rolled back my Sys Image with RollBack-rx, so I can't check the version (and the install file was also rolled back).

I'll find it online and re-check when time permits.

The freebee only diplayed my four (4) Core temps and the Nvidia stats.

Nothing on Drives, Fans, etc.

The app is was pretty straight forward and nothing else was available on the Tree, or with View Options.

It's not a showstopper, unless the Version I used has some issues with my Raid 0 setup, but that would not explain why no Fans in the Tree...

Thanx again...

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 19, 2009 0
LEGEND ,
May 19, 2009

Copy link to clipboard

Copied

Check in the BIOS whether SMART is maybe turned off.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 19, 2009 0
Explorer ,
Jun 01, 2009

Copy link to clipboard

Copied

Harm,

You must be the patron saint of NLE systems, can you bilocate  and help me set up my computer.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Jun 01, 2009 0
LEGEND ,
Jun 01, 2009

Copy link to clipboard

Copied

John,

If you start your own thread, I'll probably find it and help you out if I can.

Curt,

Thanks for the suggestion to do a backup guide. I'll keep it in mind and make no promises yet, but one can hope...

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Jun 01, 2009 0
Explorer ,
Jun 01, 2009

Copy link to clipboard

Copied

Thanks Harm, actually our IT dept is probably going to co-ordinate it and they have a contract through Dell. Up till now I've been spoiled and had my systems built for me by IT staff knowlageable in NLE systems. In my current job the IT staff haven't a clue about video and so here I am.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Jun 01, 2009 0
New Here ,
Jun 06, 2009

Copy link to clipboard

Copied

Hi Harm

I have a question reg. storage. Would you pay the extra and get WD RE2 or RE3 serverdisks for your editingrig? I know I should not get "green" ones, but in your oppinion, is the serverdisks worth the extra cash?

Thanks.

/Ulf

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Jun 06, 2009 0
LEGEND ,
Jun 06, 2009

Copy link to clipboard

Copied

Ulf,

This is a purely personal and partly budget driven answer, but with my 16 1TB disks I did not consider it worthwhile to go for the RE disks. There is no performance advantage, there is a price disadvantage and there is a longer guarantee, but with 16 disks the price difference alone would have been enough to buy 8 extra disks in case of failure outside the guarantee just to break even. So my point of view, no, it is not worth the higher price.

Where I now have 16 disks, I could have gotten only 9 RE disks for the same money. That difference was not worth it IMO.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Jun 06, 2009 0
New Here ,
Jun 06, 2009

Copy link to clipboard

Copied

Thanks Harm. 🙂

/Ulf

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Jun 06, 2009 0
Contributor ,
May 17, 2009

Copy link to clipboard

Copied

I'm confused as to why most motherboards (bios) support smart, but it is usally turned off by default. I once read that smart slows down/takes resources?

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 17, 2009 0
LEGEND ,
May 17, 2009

Copy link to clipboard

Copied

Phil,

You are correct that turning SMART on in the BIOS, takes some resources and for optimal performance it may be turned off. The performance gain by turning it off will be extremely small. For non-OC'ed system with good cooling, you can easily leave it off, but when you turn to overclocking you already have significant performance gains from a faster CPU, faster memory, faster GPU that you will not really notice further performance improvements, but you lose the ability to check temperatures of HD's. An OC'ed system is more critical with regards to cooling and PSU, so I think it better to leave it on. Another thing to consider is that if you use a raid controller, not all of them allow you to turn of SMART monitoring. If you use a raid on a Marvell chip it will be turned off, at least Speedfan and HWMonitor can not report SMART data, so it seems to be turned off.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 17, 2009 0
Contributor ,
May 17, 2009

Copy link to clipboard

Copied

Thanks. I had smart turned on but then turned it off recently after a rebuild. I think I will turn it back on. Especially since a while back I had a drive go bad and smart did alert me to it before it bit the dust.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
May 17, 2009 0