Storage rules for an editing rig. Some basics.

[Harm_Millaard]

How do you set up your editing machine in terms of disks for maximum performance and reliability? (SSD's are left out here.)

This is a question that often arises and all too often one sees that initial settings are really suboptimal. These rules are intended to help you decide how to setup your disks to get the best response times. Of course the only disks in an editing machine must be 7200 RPM types or faster. No GREEN disks at all.

Rule 1: NEVER partition a disk. You may ask why? First of all, it does not increase disk space, it just allocates the space differently. However, the major drawback is that for a partitioned disk the OS must first access a partition table at the beginning of the disk for all accesses to the disk, thus requiring the heads to move to the beginning of the disk, then when it has gotten the partition info move to the designated area on the disk and perform the requested action. This means much more wear-and-tear on the mechanics of the disk, slower speeds and more overhead for the OS, all reducing efficiency.

Rule 2: Avoid using USB drives, since they are the slowest on the market. Do not be tricked by the alleged bandwidth of USB 2.0 advertisements, because is just is not true and remember that the alleged bandwidth is shared by all USB devices, so if you have a USB mouse, keyboard, printer, card reader or whatever, they all share the bandwidth. Stick to SCSI or SATA disks or e-SATA. If needed, you can use Firewire-800 or even Firewire-400 disks, but they are really more suited for backups than for editing.

Rule 3: Use at least 3 different physical disks on an editing machine, one for OS/programs, one for media and one for pagefile/scratch/renders. Even on a notebook with one internal drive it is easy to accomplish this by using a dual e-SATA to Express card connector. That gives you an additional two e-SATA connections for external disks.

Rule 4: Spread disk access across as many disks as you have. If you have OS & programs on disk C:, set your pagefile on another disk. Also set your pagefile to a fixed size, preferably somewhere around 1.5 times your physical memory.

Rule 5: Turn off index search and compression. Both will cause severe performance hits if you leave them on.

Rule 6: If the fill rate on any of your SATA disks goes over 60-70% it is time to get a larger or an additional disk.

Rule 7: Perform regular defrags on all of your disks. For instance, you can schedule this daily during your lunch break.

Rule 8: Keep your disks cool by using adequate airflow by means of additional fans if needed. You can use SMART to monitor disk temperatures, which should be under 35 degrees C at all times and normally hover around 20-24 C, at least in a properly cooled system.

Rule 9: If people want raid, the cheapest way is to use the on-board IHCR or Marvell chip, but it places a relatively high burden on the CPU. The best way is a hardware controller card, preferably based on the IOP348 chip. Areca ARC and ADAPTEC come to mind. 3Ware uses it's own chipset and though not bad, they are not in the same league as the other two. Promise and the like in the budget range are no good and a complete waste of money. Expect to spend around $ 800 plus for a good controller with 12 connectors internally and 4 e-SATA connectors. Important to consider in a purchasing decision is whether the on-board cache memory can be expanded from the regular 256/512 MB to 2 or even 4 GB. Be aware that 2 GB cache can be relatively cheap, but the 4 GB version extremely costly ($ 30 versus $ 300). For safety reasons it is advisable to include a battery backup module (BBM).

Rule 10: If you can easily replace the data in case of disk failure (like rendered files), go ahead and use raid0, but if you want any protection against data loss, use raid 3/5/6/10/30/50. For further protection you can use hot spares, diminishing downtime and performance degradation.

In general when you get a new disk, pay close attention to any rattling noise, do perform regular disk checks, and in case of doubt about reliability, exchange the disk under guarantee. Often a new disk will fail in the first three months. If they survive that period, most of the disks will survive for the next couple of years. If you use a lot of internal disks like I do (17), set staggered spin-up to around 1 second to lessen the burden on the PSU and improve stability.

Hope this helps to answer some basic questions. If not, let me know. Further enhancements and suggestions are welcome.

[dec0n]

Hello guys,

Another FCP switcher here  ^_^

I'm in the middle of building a rig, and have opted for a 6 disk areca controlled RAID3 setup similar to many threads on this forum.

I am now wondering about which drives to put in there, I know the Samsungs are popular, and of course no green disks etc. 
Would it a big enough performance increase to put 6 velociraptors in there for the price?
Thanks for any help or advice! 
^_^

[Frédéric_Segard]

Welcome fellow switcher!

A good drive, tested by Areca with their controlers, very affordable, and a good all around performer, is the Hitachi 7K300 series. Raptor drives in a RAID is kind of useless. Way to expensive for very little gain (at that price, might as well take SSDs). But at roughly $65 for a 1.5TB SATA III drive, you can install drives to your heart's content (or to the maximum number of ports on your card, which ever comes first), and see a tremendous gain in speed.

[RFDPiper]

Rule 5: Turn off index search and compression. Both will cause severe performance hits if you leave them on.

Sorry, I have confusion about this.  What does this refer to and where is this done?  In the OS or in Adobe?

[RjL190365]

RFDPiper wrote:

Rule 5: Turn off index search and compression. Both will cause severe performance hits if you leave them on.

Sorry, I have confusion about this.  What does this refer to and where is this done?  In the OS or in Adobe?

In the OS.

[RFDPiper]

Thanks for the fast reply, but where in the OS?

[RjL190365]

Go to "My Computer", right-click on the disk that you want to change the settings on, select "Properties", then uncheck both of the boxes at the bottom of the "General" tab (labeled "Compress this drive to save disk space" and "Allow files on this drive to have contents indexed in addition to file properties"; by default, the latter check box is checked).

[RFDPiper]

RjL190365 wrote:

Go to "My Computer", right-click on the disk that you want to change the settings on, select "Properties", then uncheck both of the boxes at the bottom of the "General" tab (labeled "Compress this drive to save disk space" and "Allow files on this drive to have contents indexed in addition to file properties"; by default, the latter check box is checked).

I get a promt when I uncheck the "Allow files on this drive to have contents indexed in addition to file properties" box.  Should I "Apply changes to drive C:\ only"  or  "Apply changes to drive C:\, subfolders and files"?

[JEShort01]

Use the later, "Apply changes to drive C:\, subfolders and files".

Jim

[RFDPiper]

Thanks RjL,

Two more questions: 

1) I have separate drives for media, OS/Programs, and render.  I am setting up my RAID 0 render drive right now (don't worry, I regularly back-up to a RAID 1 NAS), and in order for the computer to see my drive obviously I have to initialize it.  It's asking me if I want to initialize it as a MBR or a GPT.  Not sure which to choose, please help.  I'm assuming MBR is OK since I'm well under 2TB. 

The drive is two 450GB Velociraptors RAIDed off of the motherboard controller if that makes a difference (I know, I know, get a hardware controller right?  Well, someday).

2)

Rule 1: NEVER partition a disk.

I set my C: drive (OS/programs) to MBR and it seems to have partitioned itself.  Should get rid of the partition, or does that not make a difference on the C; drive?  I don't mind reinstalling my OS and everything to get the best setup

I'm using Windows 7 Professional 64-bit

[tomglen]

Hi Harm,

Was wondering if you could advise. I was looking at Rule #3 on your guide. I'm not sure I can afford 3 discs so was considering a 2 Disc set up (2x 1TB Seagate Barracuda 7200rpm, 32MB Cache). I'm mainly using CS4 Ppro for HD video (<15mins in length) and some after effects. How crucial is a 3 disc set up? Will I still get good speed on that set up? And is it worth upgrading to 64MB caches?

spec is:

Intel i7 2600K 3.4GHz

Asus P8Z68-V Pro

8GB RAM

Nvidia EVGA GTX 550Ti

2x 1TB Seagate Barracuda 7200rpm, 32MB Cache

Windows 7 Pro

Thanks a lot

Tom

[Harm_Millaard]

Given that you already have two disks, the investment would be one additional disk. You would definitely notice that in performance. Changing disks to 64 MB cache models would only marginally improve performance if at all and would be much more costly.

The half duplex nature of SATA disks means that traffic in one direction (reading or writing) must finish first before traffic in the other direction can begin. That causes delays you would not have with another disk, because one can be used for traffic in one direction and the other disk for traffic in the other direction.

[tomglen]

This is for a new system and I have limited budget so trying to work out what's best to spend my money on. How much will this affect workflow during projects or is it just the opening and saving of projects? Also, would it realistic to add a third external SATA drive at a later date if I need it?

Thanks for the help,

[Harm_Millaard]

Get a third disk for OS & programs, like the Samsung Spinpoint 320 GB F4 at € 70. Makes better use of your intended 2 x 1 TB disks as well, giving you more storage space and dividing the work across more disks.

[tomglen]

What do you reckon to an SSD for OS & Programs drive? Would you recommend? Was looking at 128GB Corsair Performance Pro

[Harm_Millaard]

SSD is in conflict with a tight budget.

[tomglen]

yeah you have a point there. Thanks so much for the advice Harm, it's invaluable having your knowledge on tap!

Okay, here's the setup i'm now looking at and the uses for each drive. Do you think this will work well?. I know it uses a green drive which you weren't a fan of but the company we're ordering through only offer those for a third drive. (unless you get an SSD for drive one) and i'm guessing this will still be superior to a 2 drive setup?

Drive 1: 1TB Seagate Barracuda 7200rpm 32MB Cache.

Drive 2: 1TB Seagate Barracuda 7200rpm 32MB Cache

Drive 3: 1TB Western Digital Caviar Green, 64MB Cache

Drive 1: OS, Programs, Pagefile

Drive 2: Media, Projects

Drive 3: Previews, Media Cache, Exports

My other option is to go for a SSD as drive one but to keep it on budget I think that would have to mean downgrading the motherboard from a Asus P8Z68-V PRO  to a  Gigabyte GA-Z68XP-UD3 and I don't have a clue how much difference that would make to performance. Any help would be much appreciated!

[tomglen]

Okay, here's the setup i'm now looking at and the uses for each drive. Do you think this will work well?. I know it uses a green drive which you weren't a fan of but the company we're ordering through only offer those for a third drive. (unless you get an SSD for drive one) and i'm guessing this will still be superior to a 2 drive setup?

Drive 1: 1TB Seagate Barracuda 7200rpm 32MB Cache.

Drive 2: 1TB Seagate Barracuda 7200rpm 32MB Cache

Drive 3: 1TB Western Digital Caviar Green, 64MB Cache

Drive 1: OS, Programs, Pagefile

Drive 2: Media, Projects

Drive 3: Previews, Media Cache, Exports

My other option is to go for a SSD as drive one but to keep it on budget I think that would have to mean downgrading the motherboard from a Asus P8Z68-V PRO  to a  Gigabyte GA-Z68XP-UD3 and I don't have a clue how much difference that would make to performance. Any help would be much appreciated!

[RjL190365]

In this case, get another 7200 RPM hard drive from another source: Green drives are often too slow in both sequential transfer speed and random access speed to be of much use for anything besides backups. In the case of the Caviar Green, its rotational speed is only 5400 RPM despite Western Digital's "variable-speed" claim.

[tomglen]

Thanks for that. Would an external eSATA connected drive be fast enough do you think?

[RjL190365]

In this case, get an external USB 3.0 or eSATA enclosure and put the Green drive into that. Then get the third internal 7200 RPM hard drive.

[tomglen]

sorry, i meant the external drive as the third drive, instead of the green drive.

[RjL190365]

Nope. Remember, Harm stated "No GREEN disks at all"! This means that the Green drive should not even be inside your PC's case at all - but instead put into an external enclosure with eSATA and/or USB 3.0 capability. Here are three reasons why:

1) Their much slower than minimum spindle speed: The WD Green drives spin at a very slow 5400 RPM (as I noted in a previous post).

2) Most Green drives also have relatively slow random access speed - as much as 22ms instead of the 15ms of the average 7200 RPM hard drive.

3) Many Green drives (and all of the Western Digital Caviar Green drives) spin themselves down after a firmware-set period and cannot be overridden or disabled at all whatsoever! That means that the drives will simply ignore any software- or Windows-set spin-down times unless those times are less than the firmware-set time. In other words, if you have Windows to never power down any drives, the Green drives will power themselves down anyway. And every spin-down and spin-up can (and does) cause crashes in Premiere Pro, especially in the middle of a long encode.

However, points 1) and 2) are not as critical as point 3). The constant spinning down and spinning up of the Green drives will destabilize an editing system that expects every single installed drive to perform at its maximum speed and performance at all times.

This is the basis for my (and Harm's) assessment of Green drives in an editing system.

[claris38]

I was reluctant to ask this as there have been so many similar threads – but as always your own case is slightly different and I cannot find an answer to my specific question or the threads are a couple of years old. I've read so many threads that I'm now more than a bit addled (well, I am blonde).

So – I've just ordered Production Premium and it is for non-critical / non-commercial / non-urgent stuff ie no demands or pressures. I would primarily be working with EOS5d MKII stills and videos and expect Premiere and Encore to be used most to create both DVD and blu ray.

Cannot remember all my PC specs but non HDD key items are i7 @ 2.7gGHz; 6GB RAM; Win 7 64 bit; Radeon HD5700 graphics; Gigabyte GA-EX58-UD3R mobo (sata 2).

Okay; as for hard disks, currently installed I've a

• Corsair CSSD-F120GB3-BK Force Series 3 120GB SSD – fast but not reaching the spec speeds as a sata 2 mobo! Plenty of space (80+GB free) 

• Samsung HD103UJ 1TB; spec says 7200rpm. Pagefile currently on this drive.

• Maxtor 6V300F0 300GB; few years old but Googled spec says 7200rpm.

Available to me are 2 x Western Digital Caviar 2TB SATAII 64MB Cache 3.5-inch Green; 2 more of the Maxtor 6V300F0; a Samsung 64GB MLC SSD (was my C drive – too small so replaced by the Corsair); 2 x Maxtor DiamondMax 10 200GB sata 150 HDD (7200rpm I think).

I've read the 'Guidelines for Disk Usage' and the Adobe spec requirements but would still appreciate any help as there seems to be umpteen possible configurations. My main question is with the WD 2TB drives – this very informative thread states No GREEN disks at all. Considering my 'non-serious' editing - if I could use will their 5400rpm simply impede performance or is it more severe than that and they will effectively not work at all. I do not really want to buy any more HDD's but strong advice otherwise would mean that I will. Would installing any sort of RAID be pointless in my case.

Apologies if this all sounds a bit lame – I've only modest PC experience and wanted to ask even if I do generate some groans amongst the forum regulars. Much of the excellent help here is quite rightly geared towards professional video production and I'm not sure about the relevance for a hobbyist like myself.

Many thanks.

Claire

[RjL190365]

Claire,

I'd recommend going with mostly new hard drives in that system if you're going to be doing anything at all whatsoever with DSLR video. You see, the Maxtor drives that are in your possession are now extremely old - so old that two of them in RAID 0 ("aid0") are still slower than just one of last year's 7200 RPM hard drives. Those old drives could not achieve even 70 MB/s in sequential transfer speed on the outer tracks (because those old Maxtors used a platter layout of only 100GB per platter, resulting in the three-platter design of your old 250GB and 300GB disks) while current hard drives can reach nearly 190 MB/s on the outer tracks (and even the fastest of the two-year-old drives can reach 140 MB/s). In your inventory the fastest drive in your system is currently the 1TB Samsung F1 (note the model number HD103UJ), which uses a three-platter design to achieve its 1TB capacity. That drive can reach about 120 MB/s on the outer tracks. The 2TB WD Green is actually slower than the 1TB Samsung F1 in sequential transfer speed despite the denser platters of the Green because that Green spins at only 5400 RPM (as I noted in my previous post in this thread).

Thus, I am amending "No green drives" to also include "No hard drives more than three years old regardless of spindle speed". In other words, drives that are suitable in an editing system must not only spin at 7200 RPM or higher, but also have a platter layout density of at least 300GB per double-sided platter. Those old Maxtors fall far short of that 300 GB per platter minimum (they employ 100GB per platter at most).

[claris38]

Hi RjL,

Thanks for the reply and your help. I had kind of assumed that many of the drives I listed were not up to the job. I am going to buy either 1 or 2 x 1TB drives although when looking it is not so easy to find info on data transfer rates even on the manufacturers sites. Anyway, it seems the WD Caviar Black is pretty much accepted - albeit with some negative comments on noise and heat - and so I think this is what I'll go for and probably get 2.

So, am I correct in thinking the SSD is okay and kept for OS / progs etc and - as per the 'Guidleines for Disk Usage' chart - use the 2 new WD drives along with my current Samsung ie 4 disks. If so, which of the three suggested use options (D,E or F on the chart) should the slower Samsung be allocated to.

Many thanks for your help.

Claire

ps if your screen name does happen to be 'open date coding', then happy birthday for next Monday ... smile.