Storage rules for an editing rig. Some basics.

LEGEND ,
May 03, 2009 May 03, 2009

Copy link to clipboard

Copied

How do you set up your editing machine in terms of disks for maximum performance and reliability? (SSD's are left out here.)

This is a question that often arises and all too often one sees that initial settings are really suboptimal. These rules are intended to help you decide how to setup your disks to get the best response times. Of course the only disks in an editing machine must be 7200 RPM types or faster. No GREEN disks at all.

Rule 1: NEVER partition a disk. You may ask why? First of all, it does not increase disk space, it just allocates the space differently. However, the major drawback is that for a partitioned disk the OS must first access a partition table at the beginning of the disk for all accesses to the disk, thus requiring the heads to move to the beginning of the disk, then when it has gotten the partition info move to the designated area on the disk and perform the requested action. This means much more wear-and-tear on the mechanics of the disk, slower speeds and more overhead for the OS, all reducing efficiency.

Rule 2: Avoid using USB drives, since they are the slowest on the market. Do not be tricked by the alleged bandwidth of USB 2.0 advertisements, because is just is not true and remember that the alleged bandwidth is shared by all USB devices, so if you have a USB mouse, keyboard, printer, card reader or whatever, they all share the bandwidth. Stick to SCSI or SATA disks or e-SATA. If needed, you can use Firewire-800 or even Firewire-400 disks, but they are really more suited for backups than for editing.

Rule 3: Use at least 3 different physical disks on an editing machine, one for OS/programs, one for media and one for pagefile/scratch/renders. Even on a notebook with one internal drive it is easy to accomplish this by using a dual e-SATA to Express card connector. That gives you an additional two e-SATA connections for external disks.

Rule 4: Spread disk access across as many disks as you have. If you have OS & programs on disk C:, set your pagefile on another disk. Also set your pagefile to a fixed size, preferably somewhere around 1.5 times your physical memory.

Rule 5: Turn off index search and compression. Both will cause severe performance hits if you leave them on.

Rule 6: If the fill rate on any of your SATA disks goes over 60-70% it is time to get a larger or an additional disk.

Rule 7: Perform regular defrags on all of your disks. For instance, you can schedule this daily during your lunch break.

Rule 8: Keep your disks cool by using adequate airflow by means of additional fans if needed. You can use SMART to monitor disk temperatures, which should be under 35 degrees C at all times and normally hover around 20-24 C, at least in a properly cooled system.

Rule 9: If people want raid, the cheapest way is to use the on-board IHCR or Marvell chip, but it places a relatively high burden on the CPU. The best way is a hardware controller card, preferably based on the IOP348 chip. Areca ARC and ADAPTEC come to mind. 3Ware uses it's own chipset and though not bad, they are not in the same league as the other two. Promise and the like in the budget range are no good and a complete waste of money. Expect to spend around $ 800 plus for a good controller with 12 connectors internally and 4 e-SATA connectors. Important to consider in a purchasing decision is whether the on-board cache memory can be expanded from the regular 256/512 MB to 2 or even 4 GB. Be aware that 2 GB cache can be relatively cheap, but the 4 GB version extremely costly ($ 30 versus $ 300). For safety reasons it is advisable to include a battery backup module (BBM).

Rule 10: If you can easily replace the data in case of disk failure (like rendered files), go ahead and use raid0, but if you want any protection against data loss, use raid 3/5/6/10/30/50. For further protection you can use hot spares, diminishing downtime and performance degradation.


In general when you get a new disk, pay close attention to any rattling noise, do perform regular disk checks, and in case of doubt about reliability, exchange the disk under guarantee. Often a new disk will fail in the first three months. If they survive that period, most of the disks will survive for the next couple of years. If you use a lot of internal disks like I do (17), set staggered spin-up to around 1 second to lessen the burden on the PSU and improve stability.

Hope this helps to answer some basic questions. If not, let me know. Further enhancements and suggestions are welcome.

Views

105.5K

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
replies 143 Replies 143
Explorer ,
Sep 26, 2009 Sep 26, 2009

Copy link to clipboard

Copied

Harm Millaard wrote:

How do you set up your editing machine in terms of disks for maximum performance and reliability? (SSD's are left out here.)

This is a question that often arises and all too often one sees that initial settings are really suboptimal. These rules are intended to help you decide how to setup your disks to get the best response times. Of course the only disks in an editing machine must be 7200 RPM types or faster. No GREEN disks at all.


What do you mean by "GREEN disks"?

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Sep 26, 2009 Sep 26, 2009

Copy link to clipboard

Copied

Green disk are in marketing terms ECO friendly disks that use less energy, because their rotation speed is lower. Like a car, if you do not go over 40 MPH you use less fuel than when driving 70 MPH. Slowing down the slowest component in your computer is bad, like a snail hitting the brakes before cornering.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Sep 27, 2009 Sep 27, 2009

Copy link to clipboard

Copied

Thanks for the explanation Harm. Seems like those "Green" drives would be best suited for backup type functions.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Sep 26, 2009 Sep 26, 2009

Copy link to clipboard

Copied

Rule 10: If you can easily replace the data in case of disk failure (like rendered files), go ahead and use raid0, but if you want any protection against data loss, use raid 3/5/6/10/30/50. For further protection you can use hot spares, diminishing downtime and performance degradation.

Is there a reference link you could provide as to what all the different levels of RAID do?

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Engaged ,
Sep 26, 2009 Sep 26, 2009

Copy link to clipboard

Copied

I actually changed a few 'green' discs to black caviar because of your advice, Harm, and I have been really satisfied. Thanks.

/Ulf

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Oct 09, 2009 Oct 09, 2009

Copy link to clipboard

Copied

can you please eleborate on the difference you experienced?

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Sep 26, 2009 Sep 26, 2009

Copy link to clipboard

Copied

Hi Shaluda,

Here you could find some good information about the different RAID-levels (click on the graphics to make the demo play):

http://www.raid.com/04_01_00.html

Nicolaj

(and thanx Harm for you advice "a couple of posts ago"...)

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Sep 27, 2009 Sep 27, 2009

Copy link to clipboard

Copied

Thanks for the link Nicolaj; much appreciate it!!

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Apr 07, 2010 Apr 07, 2010

Copy link to clipboard

Copied

Is there a reference link you could provide as to what all the different levels of RAID do?

THIS will give you the specifics of the levels of RAID.

Good luck,

Hunt

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Apr 08, 2010 Apr 08, 2010

Copy link to clipboard

Copied

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Contributor ,
Jul 22, 2011 Jul 22, 2011

Copy link to clipboard

Copied

removed

Message was edited by: Frederic Segard

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Oct 09, 2009 Oct 09, 2009

Copy link to clipboard

Copied

Thanks for the interesting information. Two questions:

Is a seperate scratch disk for basic editing (AVCHD, little video effects) really needed? Can't you just use a disk for OS/programs and a storage disk for media, project file and preview files?

Scratch disk setup in premiere offers options for captured video and video previews. Assuming captured video means raw source video, this will point to the storage disk. So a seperate scratch disk would only be used for video previews, am I correct?

The WD green 1TB does go to 7200 when needed and therefore seems as fast at other storage disks. Is green really that bad?

@Ulf Larsson: what difference did you notice by going from green to black?

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Oct 09, 2009 Oct 09, 2009

Copy link to clipboard

Copied

Is a seperate scratch disk for basic editing (AVCHD, little video effects) really needed? Can't you just use a disk for OS/programs and a storage disk for media, project file and preview files?

It is about the same as having a car with a manual gearbox. Sure you can use that car and drive wherever you like while using only the first and second gear. However, if you also use 3-rd, 4-th and 5-th gear, it will take less time and entail less risks in traffic. Same with the number of disks.

So a seperate scratch disk would only be used for video previews, am I correct?

No, it will be used for the media cache. the indexed, conformed and peak files and possibly for the pagefile, as well as all preview files.

Mechanical disks are the slowest components in a computer. To get optimal results you need to get the last bit of performance from them. Even if what you say is true that the WD green disk can adjust their speed from 5400 to 7200, consider the enormous amount of time needed to get up to speed. That is an eternity in computer terms. Two or three seconds to stabilize at 7200 is extremely slow when disk access is measured in ms. And why? There is no discernable price difference or it can be measured in peanuts.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Oct 09, 2009 Oct 09, 2009

Copy link to clipboard

Copied

thanks for your reply. What confuses me is that premiere does not offer scratch settings for  these categories you mentioned: media cache. the indexed, conformed and peak file. Only captured video and preview video.

Assuming a 3 disk setup (OS, scratch, storage), and setting captured video to storage disk (raw footage) and preview video to scratch disk, which disk is used by premiere for what?

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Oct 09, 2009 Oct 09, 2009

Copy link to clipboard

Copied

The media cache location can be set in the preferences and contains the indexed, conformed and peak files.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Oct 09, 2009 Oct 09, 2009

Copy link to clipboard

Copied

Hi

I did not have any exact problems with the green ones, I just wanted to be safe based on Harms knowledge. I use the green ones for other purposes like backup and stuff.

/Ulf

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Oct 10, 2009 Oct 10, 2009

Copy link to clipboard

Copied

you say you can set media cache in preferences, but I looked and it only mentions captured audio/video and preview audio/video. but I think I understand what you mean: when you set captured video to scratch disk in those preferences, media cache is one of the temporary files that will be stored on the scratch disk, correct?

so for example, I have this setup:

disk 1: OS etc

disk 2: scratch

disk 3: storage with raw footage

I start a new project, I set captured audio/video and preview audio/video to scratch disk 2. And I save the project file itself to disk 3.

After I import video into the project from disk 3, and start editing, all scratch files, including media cache, confirmed audio, preview files, etc will now be stored on scratch disk 2. Is this how it works?

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Oct 10, 2009 Oct 10, 2009

Copy link to clipboard

Copied

Correct.

10-10-2009 17-04-33.jpg

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Oct 10, 2009 Oct 10, 2009

Copy link to clipboard

Copied

thanks for pointing out this media cache setting. But I am confused again.

I opened a project on my current PC (just 2 disks, one OS and one data). Scratch disk settings under project/project settings/scratch disks is ''same as project'' for captured video/preview

Than I looked under edit/preferences/media and the location for media cach files is a folder on my C (OS) drive: user/.../AppData/etc.

Than I tried to locate that media cache folder on my PC to see what's in it. There is no AppData/Roaming/Adobe/etc folder on my PC

Than I looked in the project folder on my data disk and next to project file it has these  folders:

auto-save

preview files

encoded files

media cache files

this makes my wonder: how did I get a media cache file in my project folder on data disk while the setting in premiere points to my OS disk? And where is that folder mentioned under media cache in premiere?  And why didn't they just add media cache to scratch disk settings?

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Oct 10, 2009 Oct 10, 2009

Copy link to clipboard

Copied

You ticked the box in the upper right part of my screenshot.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Oct 10, 2009 Oct 10, 2009

Copy link to clipboard

Copied

no, it's unticked (I never touched it). I am starting to believe this edit/preferences/media is simply not working. The media cache setting is overruled by the scratch disk settings.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Oct 11, 2009 Oct 11, 2009

Copy link to clipboard

Copied

Hello Harm,

if you look at scratch disk settings in premiere, what is the ''captured video/audio''' setting actually used for? There is no folder on my harddisk called captured video, while there are folders called media cache and preview files made by Adobe. Maybe captured video = media cache, overruling the media cache setting in preferences, explaining why I do have a media cache folder within my project folder but not on the path mentioned under media cache setting?

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guest
Oct 27, 2009 Oct 27, 2009

Copy link to clipboard

Copied

.... i was just wondering.. all these talk about HDD performance.... why did you leave out SSD again? as per my other posts i am planning to build the BEST machine money can buy (whilst still being a somewhat practical) and SSD was my first choice dedicated for OS Drive and Scratch/temp files.. was just wondering why you left it out.. is it a bad idea?         

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Oct 27, 2009 Oct 27, 2009

Copy link to clipboard

Copied

At the moment I wrote this article, SSD's were too expensive and did not offer a significant speed/performance gain over a good raid setup.

One can still wonder at the current time (october 2009) whether they will offer any significant performance gain with the new controllers and trim function in Win7 over old fashioned hard disks.

For OS, SSD's are approaching the level where you can consider them, a single 160 GB Intel SSD for around € 520 here in comparison to a single 150 GB Velociraptor for around € 140.

For editing the capacity is still a major bottleneck. I prefer a 12 TB raid30 with 800+ MB/s transfer rate for around € 1600 (including a top notch raid controller) over a 640 GB SSD raid0 with around 750 MB/s transfer rate for around € 2080.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Oct 29, 2009 Oct 29, 2009

Copy link to clipboard

Copied

Harm,

First of all, I'd like to congratulate you on your excellent articles about your hardware configurations.  I've just ordered a new pc and I'd have a question for you.  I would like to set up a system with one HDD for my OS and 4 HHD's in raid 0 config for my main storage with a hardware raid card.  But mostly on the net when you reed about raid, it's always a "total" raid config (with all the HDD in raid & OS also on the raid) and mosstly a software raid.

Also some scary story's about suddenly having a pc that's not bootable anymore...

So my question is how about am I to go to work ? I tought of the following scenario :

1) set up pc with 1 HDD and install win7 ultimate 64bit on it.

2) open up pc, put hardware raid controller card & 4 HDD in it.  Also connect the 4 HDD to the hardware raid card.

then I'm not so sure, do I have to go into the BIOS at next startup to set up the 4 HDD in raid (or is this only needed for software raid ?)

Or do I only have to config the 4 HDD with the setup software that comes with the raid card ?

Sincerely,

Frank from Belgium

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines