Storage rules for an editing rig. Some basics.

LEGEND ,
May 03, 2009 May 03, 2009

Copy link to clipboard

Copied

How do you set up your editing machine in terms of disks for maximum performance and reliability? (SSD's are left out here.)

This is a question that often arises and all too often one sees that initial settings are really suboptimal. These rules are intended to help you decide how to setup your disks to get the best response times. Of course the only disks in an editing machine must be 7200 RPM types or faster. No GREEN disks at all.

Rule 1: NEVER partition a disk. You may ask why? First of all, it does not increase disk space, it just allocates the space differently. However, the major drawback is that for a partitioned disk the OS must first access a partition table at the beginning of the disk for all accesses to the disk, thus requiring the heads to move to the beginning of the disk, then when it has gotten the partition info move to the designated area on the disk and perform the requested action. This means much more wear-and-tear on the mechanics of the disk, slower speeds and more overhead for the OS, all reducing efficiency.

Rule 2: Avoid using USB drives, since they are the slowest on the market. Do not be tricked by the alleged bandwidth of USB 2.0 advertisements, because is just is not true and remember that the alleged bandwidth is shared by all USB devices, so if you have a USB mouse, keyboard, printer, card reader or whatever, they all share the bandwidth. Stick to SCSI or SATA disks or e-SATA. If needed, you can use Firewire-800 or even Firewire-400 disks, but they are really more suited for backups than for editing.

Rule 3: Use at least 3 different physical disks on an editing machine, one for OS/programs, one for media and one for pagefile/scratch/renders. Even on a notebook with one internal drive it is easy to accomplish this by using a dual e-SATA to Express card connector. That gives you an additional two e-SATA connections for external disks.

Rule 4: Spread disk access across as many disks as you have. If you have OS & programs on disk C:, set your pagefile on another disk. Also set your pagefile to a fixed size, preferably somewhere around 1.5 times your physical memory.

Rule 5: Turn off index search and compression. Both will cause severe performance hits if you leave them on.

Rule 6: If the fill rate on any of your SATA disks goes over 60-70% it is time to get a larger or an additional disk.

Rule 7: Perform regular defrags on all of your disks. For instance, you can schedule this daily during your lunch break.

Rule 8: Keep your disks cool by using adequate airflow by means of additional fans if needed. You can use SMART to monitor disk temperatures, which should be under 35 degrees C at all times and normally hover around 20-24 C, at least in a properly cooled system.

Rule 9: If people want raid, the cheapest way is to use the on-board IHCR or Marvell chip, but it places a relatively high burden on the CPU. The best way is a hardware controller card, preferably based on the IOP348 chip. Areca ARC and ADAPTEC come to mind. 3Ware uses it's own chipset and though not bad, they are not in the same league as the other two. Promise and the like in the budget range are no good and a complete waste of money. Expect to spend around $ 800 plus for a good controller with 12 connectors internally and 4 e-SATA connectors. Important to consider in a purchasing decision is whether the on-board cache memory can be expanded from the regular 256/512 MB to 2 or even 4 GB. Be aware that 2 GB cache can be relatively cheap, but the 4 GB version extremely costly ($ 30 versus $ 300). For safety reasons it is advisable to include a battery backup module (BBM).

Rule 10: If you can easily replace the data in case of disk failure (like rendered files), go ahead and use raid0, but if you want any protection against data loss, use raid 3/5/6/10/30/50. For further protection you can use hot spares, diminishing downtime and performance degradation.


In general when you get a new disk, pay close attention to any rattling noise, do perform regular disk checks, and in case of doubt about reliability, exchange the disk under guarantee. Often a new disk will fail in the first three months. If they survive that period, most of the disks will survive for the next couple of years. If you use a lot of internal disks like I do (17), set staggered spin-up to around 1 second to lessen the burden on the PSU and improve stability.

Hope this helps to answer some basic questions. If not, let me know. Further enhancements and suggestions are welcome.

Views

105.5K

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
replies 143 Replies 143
LEGEND ,
May 17, 2009 May 17, 2009

Copy link to clipboard

Copied

Phil,

You are correct that turning SMART on in the BIOS, takes some resources and for optimal performance it may be turned off. The performance gain by turning it off will be extremely small. For non-OC'ed system with good cooling, you can easily leave it off, but when you turn to overclocking you already have significant performance gains from a faster CPU, faster memory, faster GPU that you will not really notice further performance improvements, but you lose the ability to check temperatures of HD's. An OC'ed system is more critical with regards to cooling and PSU, so I think it better to leave it on. Another thing to consider is that if you use a raid controller, not all of them allow you to turn of SMART monitoring. If you use a raid on a Marvell chip it will be turned off, at least Speedfan and HWMonitor can not report SMART data, so it seems to be turned off.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Contributor ,
May 17, 2009 May 17, 2009

Copy link to clipboard

Copied

Thanks. I had smart turned on but then turned it off recently after a rebuild. I think I will turn it back on. Especially since a while back I had a drive go bad and smart did alert me to it before it bit the dust.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Advisor ,
Jun 01, 2009 Jun 01, 2009

Copy link to clipboard

Copied

I somehow missed this thread till now.

ITS EXCELLENT!!!

Nice job Harm.

If you ever get bored; a sister thread on backups would be nice.  I vascilate on backup methods myself.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jun 08, 2009 Jun 08, 2009

Copy link to clipboard

Copied

How would you conduct the sizing of the RAID's for a professional editing system - let's say for the material of a RED ONE. Here are some thoughts - Can you comment on it if they make sense or if I miss something?

RAID for mayor project storage

I consider a 1.5TB NAS storage (RAID10), just to save the whole project environment and to access some files by other applications. So the NAS storage is on a high performance network and serves as my key storage, from which I create my regular backup. If I need to work on more projects in the same time I probably will add more NAS units within the 19" rack.

RAID for video editing

I should probably use at least 6 discs in a RAID10 environment, each 300GB, which gives me about 600 GB to work with, if I don't fill it up completely - let's say about 70%.

RAID for software installation

Here I would just use 2 discs (RAID1), 32GB each - just for ADOBE CS4 and operating system.

RAID for page-file/scratch/renders

Redundancy isn't that critical here- so I would use 4 discs in RAID0 - I assume 120GB each should be fine. Which criteria would you use for the sizing of this RAID?

I would really appreciate your thoughts and feedback!

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Jun 08, 2009 Jun 08, 2009

Copy link to clipboard

Copied

The NAS, 1.5 TB capacity in a NAS in a 19" rack does not seem very large. How many disks are you talking about? Raid10 makes sense, but what are you intending to use for the connection, iSCSI with teamed NIC's? (Sorry for the geek talk here, by teaming your 1 Gb network connectors, you can double the network capacity to 2 Gb, which is accessable by SCSI over IP). You mention high performance network connections. Personally I use a Thecus N7700 with 7 x 1.5 TB Seagates on a single iSCSI NIC, not teamed, but only for backup and offline storage, so speed is not critical for me. I have this in raid6, but the main advantage is the hot-swappable bays.

Your main storage, 6 x 300 GB in raid10. Why only 300 GB per disk? Why not get some 640 GB disks, that have only 2 platters. The extra expense is negligable, they run very cool and quiet and you will not quickly run into the problem of filling them up soon.

For OS/programs I have the feeling you are looking into SSD's in raid1. 32 GB is possibly not enough, no likely not enough. I have a pretty clean system, but with Vista64, Master Collection CS4, some plug-ins and utilities I currently have around 60 GB in use on my boot disk. From that perspective alone I would go for at least 128 GB SSD or 150 GB Velociraptor and you need two of those disks for a raid1. I am not yet convinced of the benefit of SSD's over conventional hard disks, at least at the current price point. Their access times are fabulous, their read times are magnificent, the power consumption is great, but from some early adaptors I have heard (no personal experience here) that the write times suffer quite dramatically after a few months of use, dropping to a mere 35 MB/s.

For scratch/render/pagefile 4 disks in raid0 seems quite good, but again the question, why only 120 GB per disk? If these come from an older system, quite handy and economical, but if you need to buy them, why not use the 300 GB disks you intended for your main storage?

Now for the critical question. The NAS will have it's own raid controller (not much you can do about that), but how do you plan on connecting all the other disks and configuring the three raids? You are talking about 12 disks in total (apart from the NAS) in three different arrays. Your choice of raid controller, cache memory and BBM may be critical (and as you can derive from my posts I'm a great fan of Areca).

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jun 08, 2009 Jun 08, 2009

Copy link to clipboard

Copied

...thanks a lot for your quick reply!

A quick answer related to the connection of discs - for the NAS I intend to use SATA discs, for the other (3) three RAIDS I have foreseen SCSI or SAS discs. Here are my thoughts and please don't hesitate to criticize me because the editing system is budgeted for about 20000 Euro's - and I love to have a discussion with an expert in the field. This is really a great forum. Here are the details:

Offline storrage, NAS RAID

A RED ONE delivers about a hour 4k material on a 128 GB disc (the RED ONE discs are striped). For documentary film making I rarely use 20 hours of material - ok, some people shoot 100hours - but I don't. For corporate I use less. So in total I would need to use about 3 TB (4 discs x 1.5 TB) in a RAID10 environment. There are a couple of companies out there, which supply these 1 HE NAS storage solutions, which fit nicely in these 19" racks (I don't want to make any commercial for any brand here.) They usually hold 4 discs - I mean these SATA discs. Ok, to have more reserve I could get an enclosure, which holds 8 discs - doesn't cost much more - I got you here. I intend to use this NAS quite normally over a 1 Gb network.

Main storage, VIDEO RAID

I intend to use an older IBM server since (i) I know IBM quite well and I can maintain it and (ii) I know they work quite well - and are purchased in a used manner very affordable. However, the system I have chosen supports SCSI discs - I could go with extra money for the SAS discs - but SCSI discs are just available up to 300 GB (and they aren't cheap either). Redundancy I would like to have on a video storage since discs tend to fail and the cost to repeat a day's work exceeds the cost of extra drives. If I work on a sequence, which is usually about 5 minutes long, I don't need to have that much storage here. So I basically upload the material from the NAS to the VIDEO RAID, do the editing, and store it again on the NAS. My enclosure for the video RAID can hold 6 discs, which mean I can stripe 3 discs in a RAID10 environment - ok, maximum performance is achieved by striping up to 5 discs, but 3 striped discs, isn't this enough?

RAID for software installation

I intend to use Windows XP (SP3) as operating system. Unfortunately, I haven't found VISTA stable enough in a variety of setups and mayor software companies abounded VISTA installations from their environment, replacing them again with Windows XP. If I would install the whole CS4 Master Collection on my server - would I exceed 20 GB? (With regard to the utilization of tools in any video setup, probably even in the same installation, I am quite conservative - I prefer using a different machine, which accesses the NAS, as mentioned above.)

RAID for page-file/scratch/renders

Here I am looking for a formula or criteria to use - do you have some? Obviously speed is more critical here. Therefore, I thought I go for a RAID0. I have 4 bays left - so I thought I stripe them all. I thought a scratch/render RAID shouldn't be bigger then my VIDEO RAID - therefore I came up with my 4 discs, each 120 GB.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Jun 08, 2009 Jun 08, 2009

Copy link to clipboard

Copied

I'm going to reply tomorrow, I want to contemplate your setup before making any hasty remarks.

One question (because of your display name): A, CH or D?

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jun 09, 2009 Jun 09, 2009

Copy link to clipboard

Copied

...well, it is a northern D - they call us often "Fischköpfe" because we love to eat fish here in Hamburg!

I just have summarized a bit the storage configuration I am thinking of

RAID TypeObjectiveSystem requirementsRAID level
Offline Storage
  • store a whole video project (1h of 4k material requires about 128 GB)
  • needs to be highly reliable (redundancy is a must);
  • doesn't need to be extremely fast;
  • discs can be cheap because they don't have a high burden (just upload & download) to the video RAID.
10
Video RAID
  • store material for a day work
  • fast and reliable
10
Installation RAID
  • just to install Windows XP with CS4 Master Collection
  • redundant but speed isn't critical here
1
Working RAID
  • for page-file/scratch/renders
  • as fast as possible
  • disc failure isn't a big problem
0

In order to realize this, I am thinking of the following configuration


RAID Typenumber of discsTypeGB/disctot. storage [GB]usable storage [GB]cost [€]
Offline Storage8SATA150060004800900
Video RAID6SCSI/SAS3009007202100
Installation RAID2SCSI/SAS363630200
Working RAID4SCSI/SAS1475804701000


Here are my assumptions and constraints:

  • I only have 6 bays for the Installation - and working RAID;
  • For the video RAID I also would like to reuse an enclosure, which just has 6 bays;
  • I would need to buy a NAS enclosure - so here I am open minded and just assumed 8 bays;
  • the usable storrage I estimated as 80% of the total storage;
  • discs, which are used heavily should be SCSI or SAS - I am thinking of the Cheetah 15K

Looking into the cost associated, I hit 4000€ easily just for discs. Ok, I can reuse some discs and enclosures, which I have here - but since I need to purchase the NAS enclosure (with 8 bays), which will also cost 1000€ additional,  I will use 25% of my foreseen budget for storage.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Jun 09, 2009 Jun 09, 2009

Copy link to clipboard

Copied

Von der Weser bis zur Elbe,
Von dem Harz bis an das Meer
Stehen Niedersachsens Söhne,
Eine feste Burg und Wehr
Fest wie unsre Eichen
Halten allezeit wir stand,
Wenn Stürme brausen
Übers deutsche Vaterland.

Refrain:
|: Wir sind die Niedersachsen,
   Sturmfest und erdverwachsen,
   Heil, Herzog Wittekinds Stamm. :


Ich hab mal einen ersinnigen Abend mitgemacht in Kelheim mit Leuten von Hannover die dieses Lied gesungen haben....und ich hab mitgemacht mit einem Dunkles dazu. Es war damals.....

Bin aber nicht ganz sicher ob Hamburg auch als Niedersachsen angedeutet wird. Im Fall ich mich irre, bitte meine Entschuldigungen.

Back to topic:

Your setup looks very good, there is no weakness I can discern. Yes, the cost is high, but OTOH with your budget you can (relatively) easily afford that and still get a dual W5580 with 24 GB RAM and make a fabulous high perfomance machine, that makes my PC absolutely outdated, slow, and ready for replacement.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Jun 09, 2009 Jun 09, 2009

Copy link to clipboard

Copied

Offline storrage, NAS RAID

I would go for the 8 disk enclosure. You may not need to populate them all right away, but at least the backplane is 8 disk ready. Another thing to consider why a relatively large storage capacity may be beneficial is that you may need to keep one project available, while awaiting final approval from your client, but need to start your next project. And SATA storage is cheap.

Main storage, VIDEO RAID

From what you told me, the data rate of Red One 4K material is about ten times HDV. Luckily you are talking about SCSI disks and I assume about 15K RPM disks. Yes, they are expensive, but also very fast. With your relatively short timelines my guess is (this is a guess, since I have never worked with Red 4K material) that you should be OK with 6 disks in raid10, but pay attention to the cache memory on the raid controller. A 256 MB cache or a 2 or even 4 GB cache makes quite a difference. Another advantage you have is that SCSI disks do not show performance degradation with increasing fill rates as much as SATA disks.

RAID for software installation

First of all, in my case just the Master Collection uses over 16 GB. But add the OS and some additional tools and you may well end up with about double that amount. I use Vista 64 Business and of course Vista has a much larger footprint than XP, but my Windows directory is 20 GB.

RAID for page-file/scratch/renders

Using your existing 120 GB disks makes a lot of sense. Striped you will have 480 GB and that is more than enough space. Will the speed be enough? I guess so, even with older disks that may achieve a transfer rate of 60 MB/s per disk, you will increase that with a factor of somewhere around 3.2 to 3.6 by striping 4 disks. It depends on the raid controller whether you can achieve only 3.2 or even 3.6 as a speed increase over single disks.

All in all this setup makes a lot of sense for your purposes. Let me know what raid controller you are contemplating.

Be aware of the limitation of Win XP of not supporting LBA64, so you have to use MBR structures, limiting you to max 2 TB per volume. That does not apply to Vista or Server.

Hope this helps in your decision making.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jun 09, 2009 Jun 09, 2009

Copy link to clipboard

Copied

Thanks a lot for your advise!

Offline storage, NAS RAID

I think here we are in sync. I looked up your THECUS recommendation - and indeed you can get already here in Germany the N8800 for about 3600€ incl. VAT configured with 16TB (8 discs with 2TB each). They are other brands out there, such as xtivate, QNAP..., probably even faster but also more expensive. I wouldn't use the iSCSI functionality because of Windows XP 2TB limitation - as you stated. I would use it as simple assigned network drive so that I can see more than 2TB and I can live with the fact that a movement of data to the 900GB SCSI Video RAID probably takes 2 hours. Indeed, in such an environment a RAID level 6 (probably 5 is better) with hot spare(s) is more economical. Conservative people would probably still use RAID level 10 - however, the key arguments are that

(i) SATA storage is relatively cheap and suitable in this "buffer environment" and

(ii) Should be probably oversized in order to store a couple of recent customer projects - in case changes occur.

I think point (ii) becomes very relevant since I don't have a solution for the backup in mind. If a project takes 1-2 TB I probably need to deliver to my clients the discs, telling them to take care of the data security and after "time X" I will have erased their project from my RAID - not sure what they say.

Other 3 RAIDS

I think with the sizing we are in sync - ok, VISTA versus XP is probably a different blog. In Germany even the big production houses still use XP because of its stability. And it gets even more tricky if it comes to Microsoft's new proposed operating system. I just remember the time a couple of years ago where I got lost with my TARGA cards, where we changed from NT to Windows 2000 and from Premiere 5.1 to 6.0 but I was left without drivers for my cards.

Connection of SCSI drives

Here you made a good point - my old IBM server has an onboard 1 Channel RAID controller and I have (2) two large/full-size and (1-2) additional half-size PCI-X slots. Now, the PCI-X 133 bus is 64-bit and operated at 100 MHz and makes 800 MB/s. Ok, the bus is quite old and 20% slower than a PCIe 4x bus. One PCI-X slot is going to be occupied by the AJA card and I will probably add an additional dual PCI-X graphic card. So I have one to two PCI-X slots left and consider using a dual channel SCSI card.

Here are my thoughts: I would choose an

Adaptec 2230SLP. IT is a dual channel and it has a low-profile and physically easily fits into my server. IBM also uses these Adaptec boards - so I don't expect problems with compatibility. For the configuration I see (2) two options:

OPTION 1:

  • Main storage, VIDEO RAID on channel (1) one on the Adaptec 2230SLP;
  • RAID for page-file/scratch/renders on channel (2) two on the Adaptec 2230SLP;
  • RAID for software installation on onboard 1 Channel RAID controller;

OPTION 2:

  • Main storage, VIDEO RAID on channel (1) one on the Adaptec 2230SLP;
  • RAID for page-file/scratch/renders together with RAID for software installation on channel (2) two on the Adaptec 2230SLP;
  • I will disable onboard 1 Channel RAID controller.
  • Which option do you think is better? I could also go for the SAS discs or a mixed environment SAS and SCSI environment. However, I like to reuse my enclosure and some of the SCSI discs I have.

    Likes

    Translate

    Translate

    Report

    Report
    Community guidelines
    Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
    community guidelines
    LEGEND ,
    Jun 10, 2009 Jun 10, 2009

    Copy link to clipboard

    Copied

    I would go for option 1. The 2230 seems like an excellent choice. The only drawback is the fixed 128 MB cache, but I have not found any alternative.

    Likes

    Translate

    Translate

    Report

    Report
    Community guidelines
    Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
    community guidelines
    New Here ,
    Jun 11, 2009 Jun 11, 2009

    Copy link to clipboard

    Copied

    Thanks again for your feedback -

    Well, I think the good aspect on a forum is (despite the marketing aspect) to discuss ideas, concepts before puting them into reality, which unfortunately cost a lot of money. The value of a video editing system unfortunately decays away quite rapidly and therefore, I think probably three times before I put the coins on the table and I want to see the ROI. 20 Thousand Euro for a video editing system aimed to make some money sounds like a great budget - but looking into the details it isn't (- small duplication facility for 100 DVD's or so, class 2 monitor with propper card, datasafety and storage management...).

    ... I had a couple of conversations with hardware vendors during the last days related to storrage management and I mentioned your idea iSCSI as well. We also discussed solutions with SATA discs only, probably used by 4x 1GB lans. As you mentioned to pair 4x 1 GB lan cables. However, the initial data transfere would be very small, probably 40 MB/sec before it catches up to about 200 MByte/sec. This latency effect is also present with iSCSI. (... your iSCSI is probably one reason to use VISTA, even though I can get tools for XP64 as well.)

    I think I will go for the "3 RAID solution" as initially porposed and an additional NAS with dual link as network storage to store the ,main project and probably the once, which are a bit older in case changes ocure. In two weeks or so I will let you know how it looks like.

    P.S.:

    ...zur Mentalität in Hamburg und Niedersachsen. Obwohl die beiden Städte nur 90 Minuten entfernt sind, für Amerikaner etwa einmal Chicago's Lakeshore Drive von South Side nach Evanston, betrachten die Hamburger die Menschen südlich der Elbe als Bayern. Die Bayern "Lederhosen" sind für die "Fischköpfe" zwar liebenswert, aber doch sehr anders - was sich auch im Bier bemerkbar macht. Übrigens sind bei uns im Norden die Fischbrötchen und der Matjes sehr beliebt - in Bayern wohl eher die Würstchen - wenn ich mal so in die Cliche-Kiste

    greifen darf. Teile von Norddeutschland, das heutige Schleswig Holstein, gehörten ja auch mal zu Dänemark (...sorry) und das Alte Land "Dat Ole Land" westlich von Hamburg wurde ursprünglich von den Hollander bewirtschaftet, d.h. die habe hier z.B. die vielen kleine Entwässerungsgräben angelegt (...also ein dickes Dankeschön an die Käsköpfe). Der Mentalitätsunterschied ist wohl das Ergebnis der historischen Wurzeln zu Holland und Dänemark und, das macht Hamburg erst zu einem Hamburg, der Hafen mit dem Handel mit vielen Ländern der Welt. Ihr zitiertes Lied ist schon sehr alt und wohl heute nicht mehr "scenegerecht" um nicht zu sagen "out". Naja, unter dem Einfluß von einigen Litern Bier wird Ihnen hier in Hamburg alles vergeben - auf der Reperbahn in Hamburg wird der Begriff "Sächsischer Stamm" aber wohl eher anders verstanden

    Likes

    Translate

    Translate

    Report

    Report
    Community guidelines
    Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
    community guidelines
    New Here ,
    Jul 15, 2009 Jul 15, 2009

    Copy link to clipboard

    Copied

    Nice article but I have to nitpick a few points:

    Rule 1: NEVER partition a disk.

    This is incorrect. The partition table it read at boot and never again, unless you make changes to it or force it to (e.g. to rescan disks).

    You can even destroy your partition table and everything will run just fine - until you reboot.

    Rule 5: Turn off index search and compression. Both will cause severe performance hits if you leave them on.
    Correct, but I would just like to point out that the compression feature can be left on - because files aren't compressed by default, only when you ask it to. I know I'm splitting hairs but someone might want to use compression on other (non-relevant) folders/files on their disk and think they can't/shouldn't.

    Rule 7: Perform regular defrags on all of your disks.

    Defragging is in general overrated but nevertheless yes, it should be done. Personally I recommend using a 3rd party product like Diskeeper which can defrag automatically when needed and/or when usage level is low. Much easier than trying to schedule it and use the built in defragger which is slow and unreliable.

    Rule 10: <snip> if you want any protection against data loss, use raid 3/5/6/10/30/50.
    I just want to point out that RAID-5 or any other RAID where parity disks are used is likely going to cost you way more performance than all the other mentioned performance points here combined. The read performance is fine but writing is on a different planet, compared to mirroring RAIDS (1, 10, etc). Of course in most cases, RAID-5 is "good enough" but it is very relevant if we're talking about setting up a high-end large storage system, which seems to be the scope here. Also, a good fast RAID-10 makes the points about spreading your data on multiple disks, using separate disks for paging, cache, temp space, etc, redundant - because everything is spread automagically. RAID-5 does not, because the parity disk(s) becomes a bottleneck.


    Likes

    Translate

    Translate

    Report

    Report
    Community guidelines
    Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
    community guidelines
    LEGEND ,
    Jul 15, 2009 Jul 15, 2009

    Copy link to clipboard

    Copied

    I just want to point out that RAID-5 or any other RAID where parity disks are used is likely going to cost you way more performance than all the other mentioned performance points here combined. The read performance is fine but writing is on a different planet, compared to mirroring RAIDS (1, 10, etc). Of course in most cases, RAID-5 is "good enough" but it is very relevant if we're talking about setting up a high-end large storage system, which seems to be the scope here. Also, a good fast RAID-10 makes the points about spreading your data on multiple disks, using separate disks for paging, cache, temp space, etc, redundant - because everything is spread automagically. RAID-5 does not, because the parity disk(s) becomes a bottleneck.


    In fact, raid5 with it's distributed parity also spreads everything across all disks. Even with raid3 with it's dedicated parity drive, the data is spread over the data disks automatically. There is no discernable difference in performance between raid configurations that are mutiples of 10. Raid10, 30 and 50 perform about equal with the same number of disks. The difference is cost. Raid10 is extremely expensive and for most overkill. Example, you need 10 TB of storage. With raid30 or 50 you need 12 disks, with raid10 you need 20 disks plus additional housing and power supply and a larger (more ports, 24 ports instead of 12) controller. Usually when using a somewhat larger array, the bottleneck is the bus. Adding more disks does not give noticeable performance gains after a certain limit.

    Likes

    Translate

    Translate

    Report

    Report
    Community guidelines
    Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
    community guidelines
    New Here ,
    Jul 19, 2009 Jul 19, 2009

    Copy link to clipboard

    Copied

    Harm Millaard wrote:


    In fact, raid5 with it's distributed parity also spreads everything across all disks. Even with raid3 with it's dedicated parity drive, the data is spread over the data disks automatically. There is no discernable difference in performance between raid configurations that are mutiples of 10. Raid10, 30 and 50 perform about equal with the same number of disks. The difference is cost. Raid10 is extremely expensive and for most overkill. Example, you need 10 TB of storage. With raid30 or 50 you need 12 disks, with raid10 you need 20 disks plus additional housing and power supply and a larger (more ports, 24 ports instead of 12) controller. Usually when using a somewhat larger array, the bottleneck is the bus. Adding more disks does not give noticeable performance gains after a certain limit.

    Yes, the data and parity are striped but this isn't the issue. With raid-5 any write operation means you must read the parity and recalculate it. In raid-10 or any other mirroring raid, you just write it - done. This may not sound like a big deal but is in fact a very real issue in the storage industry. Any redundancy based on parity instead of simply mirroring has this problem. How big of a problem it is depends entirely on your needs and specific setup. Something like raid-50 (which is just two raid-5's striped) will of course be better, but hardly "equal". If you do need good performance (in writes as well), it's much better to just avoid the whole issue by going for raid-10. Yes, it will be more expensive but really, not massively so, and the users we are talking about here likely have photo equipment running into the tens of thousands, and make a living off this data. If you're using SATA, there is no reason to use anything other than raid-10. With SCSI/SAS, the cost difference is much more tangible (e.g. something like 10k vs 15k) but still considering the scope and audience here I consider it a small investment.

    The whole other issue is data security. Raid-5 will lose all your data if two disks fail, and even after losing one the chance of having lost or corrupted data is very real. With the size of disks today, you're almost guaranteed to encounter read errors even if a single disk fails.

    If you're a single home user or amateur, raid-5 is fine in terms of performance, but I'd personally still avoid it for this reason, plus the cost difference at this low end is very small. For a pro, with a heavy need for large IO operations and multiple simulatenous users, I would never recommend raid-5.

    A few links with more details:

    http://www.yonahruss.com/2008/11/raid-10-vs-raid-5-performance-cost.html

    http://www.miracleas.com/BAARF/RAID5_versus_RAID10.txt

    http://www.cyberciti.biz/tips/raid5-vs-raid-10-safety-performance.html

    Likes

    Translate

    Translate

    Report

    Report
    Community guidelines
    Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
    community guidelines
    LEGEND ,
    Jul 19, 2009 Jul 19, 2009

    Copy link to clipboard

    Copied

    For a pro, with a heavy need for large IO operations and multiple simulatenous users, I would never recommend raid-5.

    I quite agree for a real pro, but also a real Pro would not use anything less than a HP EVA Storageworks solution. See here: http://h18000.www1.hp.com/products/storageworks/eva8400/index.html

    But then, this Pro would never use PR in the first place. First of all because Adobe is not suitable for multiple simultaneous users. Secondly because a single user does not have large IO operations. He would turn to Autodesk Smoke, Flame, Flint, Lustre, Inferno and the like under Linux.

    BTW, have you noticed how many people around here ask questions with less than $ 30K of equipment and how the majority of questions come from people with a camera below the $ 1K boundary? Maybe your perception of PR being "pro" is a bit overrated.

    I wonder how many here have an EVA Storageworks 8400 with 324 disks (300 GB/15 K RPM SCSI) for storage?

    If you're a single home user or amateur, raid-5 is fine in terms of performance, but I'd personally still avoid it for this reason

    You are worried about your 'pro' image when using raid5 instead of raid10? Makes sense if you have an EVA solution. Not so much if you only have 3 or 4 disks, but then a real 'pro' would not limit himself to only 3 or 4 or 20 disks, when he would have to accomodate multiple simultaneous users with heavy IO, he would have at least 96 disks or a multiple of that, of course in a 19" rack.

    I don't know whether you realized this was written as some basics for an editing rig, as the title shows, intended for a single user since PR is not intended for network use, so while your remarks have some value in a corporate Linux/Unix based environment, they are completely irrelevant here.

    Likes

    Translate

    Translate

    Report

    Report
    Community guidelines
    Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
    community guidelines
    LEGEND ,
    Jul 19, 2009 Jul 19, 2009

    Copy link to clipboard

    Copied

    If you do need good performance (in writes as well), it's much better to just avoid the whole issue by going for raid-10.

    Not sure I agree.  In a benchmark test, the Areca ARC-1680 card turned in an average write of 602 MB/s with RAID 3, and 615 MB/s with RAID 10.  That's not a significant enough "improvement" in my book to warrant the extra cost.

    RAID 30 and 50 were even slower (around 570 MB/s average write).

    Further, you'll be reading from a disk as you edit far more often than you'll be writing to it.  And here the numbers definitely favor RAID 3, with an average of almost 400 MB/s for RAID 3, and only 257 MB/s for RAID 10.

    All this supports a point which I've long expressed.  RAID 3 is simply the best choice for media security and speed in a video editing environment.

    Likes

    Translate

    Translate

    Report

    Report
    Community guidelines
    Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
    community guidelines
    Community Beginner ,
    Aug 01, 2009 Aug 01, 2009

    Copy link to clipboard

    Copied

    I have a troubling situation.  My rig once was a gaming rig with an h20 custom setup.  After I configured the loop, the remaining space allowed for 1 HDD, my OS/game/ drive.  Now, this machine is loaded with Premiere Pro CS4 and I only have the one drive to work with and I'd like to get a storage drive.

    my rig for references; Q9650 8g 4870 Ultra 320 SCSI 300g drive

    I know i'm going to need a storage drive for holding my AVCHD files (MTS) so from the get go it looks like thinking outside the case is the case i'm dealing with.   Is it a good idea to use NAS for and editing machine & how will it effect my workflow?

    Thanks  rollem!

    EDIT: tried creating paragraphs - this forum program sucks

    Likes

    Translate

    Translate

    Report

    Report
    Community guidelines
    Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
    community guidelines
    LEGEND ,
    Aug 03, 2009 Aug 03, 2009

    Copy link to clipboard

    Copied

    Is it a good idea to use NAS for and editing machine & how will it effect my workflow?

    That depends entirely on your NAS configuration. If you have an iSCSI NAS and use teamed NIC's both on the NAS and on your workstation, you may be able to edit effectively, but I have never tried it, so I have no real life experience here. I only use the iSCSI NAS with a single NIC and use it only for backup, not for editing. With a single NIC it is just too slow. Maybe you should look at fibre channel configurations.

    Likes

    Translate

    Translate

    Report

    Report
    Community guidelines
    Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
    community guidelines
    New Here ,
    Aug 03, 2009 Aug 03, 2009

    Copy link to clipboard

    Copied

    Hi Harm,

    I'm looking to build a system very similar to yours, minus the large raid array.

    For storage, I'm looking at:

    System/App drive: 300GB WD VelociRaptor drive

    Editing Drive: Two 1TB Samsung Spinpoint drives in Raid 0 using onboard Marvell

    Pagefile/Scratch/Render: 1TB Samsung Spinpoint drive

    (all drives would be backed up to a homeserver)

    Is this a decent setup? I'm looking for something faster than 3 single disks, but cheaper than a full RAID solution.

    Rest of the hardware would be P6T Workstation, i7 920, good case and ps and cooling, 12GB corsair, gtx260 or 285.

    Let me know your thoughts. I need to order tonight.

    Thanks!

    Mike

    Likes

    Translate

    Translate

    Report

    Report
    Community guidelines
    Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
    community guidelines
    LEGEND ,
    Aug 03, 2009 Aug 03, 2009

    Copy link to clipboard

    Copied

    Looks like a good disk setup. Your media drives on the Marvell will be nearly twice as fast as a single disk with very limited CPU overhead. The video cards are good as well, although the 285 is a bit pricey in comparison to an ATI 4870/4890.

    Likes

    Translate

    Translate

    Report

    Report
    Community guidelines
    Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
    community guidelines
    New Here ,
    Aug 04, 2009 Aug 04, 2009

    Copy link to clipboard

    Copied

    Thanks Harm - went ahead and pulled the trigger. I need all the horsepower I can get to handle 5D mkII files!

    Thanks again!

    Mike

    Likes

    Translate

    Translate

    Report

    Report
    Community guidelines
    Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
    community guidelines
    New Here ,
    Sep 11, 2009 Sep 11, 2009

    Copy link to clipboard

    Copied

    Thank you Harm for very informative guides! One question about storage:

    I've just got a new Canon Legira HF200 camera, and I need to build a new PC to edit the AVCHD files. I'm only a hobbyist, so I I'm not going for the high end systems, but I need a system that is reliable and "fast enough" for making editing AVCHD a "positive experience" ...

    On my present editing machine (3 yrs old by now) I've got 3x320 GB SATA2 (7200) drives - which I guess are the only components that might be used in my new system. I've also got a WD 1TB e-SATA for backup.

    My plans for storage on the new PC is:

    1.- 150 GB Velicoraptor for OS/Premiere

    2.- Media drive (AVCHD-files, music, pictures etc)

    3.- Scrath/pagefiles/???/-drive

    4.- A fourth array???

    (And it will be a i7 920, 12GB+ RAM, ... system)

    The basic question is:

    As a hobbyist, how much will I profit of making no. 2 and/or no. 3 (see list above) a RAID-array (0 and/or maybe 1)? I do not think I want to pay a lot to get an expensive RAID-controller. BUT disks are not expensive, so if the performance will increase significantly by using RAID1 (2 or 3 disks), and that can be efficient managed by the MBO - I'll use such a system. I'll use my eSATA drive for backup (and maybe an internal drive for backup as well?).

    Of course - I do already have the 3x320 GB SATA2 disks - where would this be most useful, or should I rather invest in som newer and faster (7200) disks?

    I guess I'll wait a couple of months before I buy, as I still have some DV-material I have to edit, which work just fine on my previous system. No need to upgrade before I need to -as prices are decreasing and components improve fast! And I guess it is worth waiting for Windows 7 64 bit - am I right?

    Thanx

    Nicolaj

    Likes

    Translate

    Translate

    Report

    Report
    Community guidelines
    Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
    community guidelines
    LEGEND ,
    Sep 12, 2009 Sep 12, 2009

    Copy link to clipboard

    Copied

    Nicolaj,

    I would initially keep it simple, since the burden of AVCHD editing is on the processor and memory, not on the disk.

    1. Velociraptor for OS & Programs

    2. 320 G for Pagefile, scratch and final results

    3. 320 G for media

    4. 320 G for projects

    5. 1 TB eSATA for backups or additional media.

    Raid1 has no speed advantage over a single disk, it just duplicates the single disk. If you run out of space in the future, you could replace a 320 G disk with a 1 TB or lager disk for you media.

    Get a very good cooler for your system, since you want to overclock the i7 for editing AVCHD and get DDR3-1333 memory, not DDR3-1066.

    Likes

    Translate

    Translate

    Report

    Report
    Community guidelines
    Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
    community guidelines