Copy link to clipboard
Copied
There have been many questions about how to set up your disks.
Where do I put my media cache files, where the page file, and what about my preview files?
All these allocations can be set in PR, so I made this overview to help you find some settings that may be beneficial. It is not a law to do it like this, it is a generic approach that would suit many users, but depending on source material, workflow and backup possibilities, it is not unthinkable you need to deviate from this approach in your individual case.
The reasoning behind this overview is that you want to distibute disk access across as many disks as possible and get the best performance.
Look for yourself:
I hope this helps to remove doubts you may have had about your setup or to find a setup that improves performance.
Copy link to clipboard
Copied
Do you know what USB stands for?
Universally Serious Backups = USB. That is all USB drives can be used for. Even with the much touted theoretical bandwidth of USB3, the USB3 disks are still around 100+ % slower than SATA and there are no raid controllers for USB drives in existence. It is all marketing hype. USB2 could not achieve more than 20-25 MB/s and it looks like USB3 will be hard pressed to achieve more than three times that performance, still lagging way behind SATA/SAS connections.
From early benchmark results I have read, the speed of USB3 drives was comparable to FW-800 drives. Things may have improved somewhat since those early tests, but still no match for SATA/eSATA.
Copy link to clipboard
Copied
I realize that USB 3.0 is slower than RAID, but is a 7200RPM drive, which is limited to say 125 MB/sec, going to be slower on USB 3 than SATA?
Copy link to clipboard
Copied
So, to handle the uncompressed format I should set them up with one as the primary boot drive and the other three in a RAID0?
Copy link to clipboard
Copied
Thanks Harm.
You said "Get two new 7200 RPM SATA disks. Your current drive C and the new disks as D and E and use your current Green disk as F for backups. "
It looks like I can squeeze only a single new drive out of my budgeting department. It will be a Caviar Black.
I'm thinking the Caviar Black (D:) should have Media and Projects, and Caviar Green (E:) should have Exports. C: will be for OS, Programs, and Pagefile. Which drive should Previews and Media Cache be on?
Copy link to clipboard
Copied
New to all this, but I've been reading for days now. Thanks for all your help.......... One source of confusion.
Yesterday I found this here (somewhere) C: OS & Programs
D: Projects & Scratch
E: Media
Now, at the begining of this thread I see your table. Does the difference matter??
Thanks, Tom B.
Copy link to clipboard
Copied
Thank you, Harm for an excellent guide!
I have the following setup:
OS: Win 7 64 - Ultimate
MB: Gigabyte GA-EX58-Extreme
CPU: Intel Core i7 975X
RAM: 24GB, (6x4 GB Mushkin DDR3 1333)
GFX: Nvidia GTX285
RAID: 3ware 9750-4i
Single Drives:
OS Drive: 7,200 SATA 500GB
Prog Drive: 7,200 SATA 500GB
Page file and Misc.: 7,200 e-SATA 1TB
RAIDS:
Projects, Media, Render etc.: 4 x 1TB SAS2 RAID-5
Backups: 4 x 1TB SATA RAID-5 (e-SATA)
I have one question; My read speed from the SAS raid averages about 400MB/s which I think is OK, but the write speed is about 80-100 MB/s. Is that normal, is that a result of the nature of RAID-5 over 4 disks?
Thanks,
Magnus
Copy link to clipboard
Copied
Magnus,
That figure seems very low. With a 4 disk raid5, even on a 3Ware controller, I would have expected that to be in about or in excess of 200 MB/s.
What processes do you have running? Please post a screen shot from Process Explorer, not Task Manager. And post this in a new thread to avoid contamination.
Copy link to clipboard
Copied
Thanks Harm, will do once I get home!
Copy link to clipboard
Copied
Set your write policy for the raid volume properties to "Always Write Back"
Eric
ADK
Copy link to clipboard
Copied
Thanks, Eric!
I currently have:
Write Cache: Enabled
Read Cache: Intelligent
Auto Verify: x (checked)
Overwrite ECC: - (unchecked)
Queuing: x (checked)
I was thinking that maybe the queuing should be unchecked for SAS drives?
Copy link to clipboard
Copied
Are you using the Raid console management in windows or the bios raid management?
Use the one in Windows and look for a write policy setting under your raid volume properties. If you can't find it contact the manufacturer and ask them where that setting is at.
Windows 7 will continue to throttle the drive performance during write operations for data integrity until you change that setting.
Eric
ADK
Copy link to clipboard
Copied
For my controller (9750-4i) Write Back is called Write Cache, I believe. If I uncheck that I assume it would act as Write Through because write speed drops to 10-12 MB/s!
I am using the Web Interface through Firefox.
Copy link to clipboard
Copied
I would contact the manufacturer. Because the write speed your getting still sounds like the write policy is incorrect. The Cache settings are normally strictly tied to controller and drive cache and not policy settings which are separate. Disabling the cache for writing would still enable Controller cache for reading. Most controllers just give you an option to enable or disable controller cache. It looks like they split those up so you can decide which instructions get cached. That is not the same thing as write policy. It's up to you but I would contact them.
Eric
ADK
Copy link to clipboard
Copied
Thank you, Eric, for clarifying!
I will certainly contact 3ware/LSI and see if they can direct me in the right direction!
Copy link to clipboard
Copied
I contacted 3ware support and they had me update the firmware for my Seagate Constellation drives and then test the throughput using IOmeter.
These are the results I get for write throughput now: 399MB/s.
Copy link to clipboard
Copied
This is a great thread. Thanks for all the info. I am hoping that I can get some answers here. I just got my new Mac Pro last night and had it set up into a RAID 5 configuration mostly because I was not sure what I needed and the Apple guy insisted this is what I needed. I have a Mac Pro running Snow Leopard 10.6.5 three internal 1 TB hard drives with one external 2TB iOmega Hard Drive Firewire 800. I will be using this for Adobe Premiere Pro CS5 and Logic Studio.
I do primarily wedding video and some other small videos (documentaries) all on uncompressed media from Canon 5D and Canon 7D as well as video captures from other sources. I want to optimize the system. I am thinking that the RAID 5 I set up from the install disc to get all three drives in the RAID set is not the best use of this set up. I am thinking I need to put the OS and apps on one drive and then have the other two in a RAID 0 and then use my external for storage of video files and music files. But, don't know where to keep the project files and where I should do the scratch, etc. Or should I get a fourth drive, and re-do my RAID 5 to include that drive, if I do, how do I dedicate the specific drives to house the specific tasks. For example how would I get just one drive to do the OS? I understand I can dedicate the rest of the functions of the drives in the RAID set via Pr Pro after the RAID is put together.
Please provide some help if you can. It makes no sense for me to start work in this configuration since if I need to change it later, there will be much reworking of data. Figured I better get this sorted out asap so I can get up and running in the right way.
Thanks
Copy link to clipboard
Copied
Hi Billy,
Well, you certainly have the horsepower you need, but it needs a little configuring. I shoot 5D2 footage and use Premiere Pro as well, and there are a few things to know to make this easy so it is done right the first time.
We'll discuss your Mac in a second, but first you should know that you are going to need to pull off every trick in the book to get the footage to edit smoothly. You may know a lot of this, I'm just including it so it's all in one basket (and to benefit another reader possibly). The H.264 QT format that comes out of the 5D Mark II and 7D is a good compressed format for finished video, but when we attempt to edit it, frustration can occur. You aren't editing uncompressed video by any means, if it was it would be 4K footage being that the 5D2 sensor is 5616x3744. Wouldn't that be awesome if we COULD get that out of it??
To be clearer on this, the 5D2 compresses the image down to 1920x1200 in H.264 (H.264 for video is the same thing as shooting a still image in JPEG format... we have RAW for stills but not for video YET in a 5D2). The compression used by Canon stuffs the video down into an 8-bit 4:2:0 color space which is more lossy than I would prefer, but I imagine Canon did that to make the footage as manageable as can be. In fact, with Premiere Pro CS4, it was uneditable, it just froze Premiere. To fix this, we all bought Cineform's NeoScene which converts the 5D2 footage up to a 4:2:2 10-bit file (still 8-bit but now with headroom for color grading, etc) and converts it into an .AVI file so it'll run smoothly in Windows. They do have a Mac version also, and you can buy it at VideoGuys.com for $99. ANYWAY, that's how we were able to edit in CS4. Now that CS5 is out, Adobe went to great lengths to accomodate the DSLR world by adding the right presets, etc, but most importantly by adding the MPE for all editors to leverage for all types of formats. not just DSLR vids. Now, the Mercury Playback Engine helps greatly, but as soon as you stack some effects onto the footage, it can still get jumpy. To this day I still convert everything with NeoScene because in conjuction with the MPE it's even that much better. Premiere Pro CS5 does leverage the CUDA technology in the video cards to help power that MPE, but that does mean having a compatible Nvidia graphics card, not sure if you have that in your Mac. I know there's a GTX 285 for the Mac that would work, you would have to research other models such as the uber-expensive Quadro series. I don't think there's a card in the 400 or 500 Nvidia series for Mac yet. But that may be a good reason to get NeoScene anyway.
OK, what on earth did all that have to do with your Mac's hard drives? Glad you asked! Once you ensure you have the best converted file available to use for editing (by doing all that above), you then want to place that file on your fastest drive which would be your RAID5 config. You need read speed like crazy which RAID5 will give you. You also get redundancy with that. I use RAID0 but I always back up my footage first so if RAID0 drops, I can quickly recover. I realize that doesn't help me halfway through a 6 hour encoding session, lol. ANYWAY, there's one issue with your setup: The OS and programs are also on that RAID5, correct? That's what you don't want. It'd be great if you could install a 10K RPM 300GB WD Raptor and put your OS and the Adobe Suite on that drive and save your RAID5 for your assets. I would go one step further and add one more drive to write your output video to and to also use for the Adobe Scratch Disks. Don't use partitions, and by all means don't use the external drives, FireWire is way too slow. The only external drive you would want is eSata, but really you want your footage on that RAID5 setup. I know, this all sounds kookie, you probably just wanted to get the powerhouse Mac with RAID and be done with it. That'd be fine for gaming, but for video editing you need the right setup, it will make your life much much easier.
BTW, do you have at least 12GB of RAM? We love Adobe software, but it eats ram like a cheetah eats gazelles...
OK, that's all I have. Whatever you choose to do, just make sure your footage is on that RAID5 rig, even if it shares it with the OS. Worst case, you could always buy just one additional internal drive and put all the Adobe software on it, and leave the OS on the RAID5 with your assets, because the Mac OS loads mostly into memory anyway.
Recap: Your best config if it was possible:
This is pretty much the setup I use on the PC I built and it works well.
There it is! If I'm off track, somebody pipe in, let's get Billy up and running the right way the first time!
Message was edited by: PaulieDC... fixing typos!
Copy link to clipboard
Copied
Thanks for the great info!
In a three-drive set-up as described in the original post, which would drive would need to be the fastest? Which could be your slowest? (Assuming all three are 7200RPM SATA drvies of varying speeds/ages)?
Copy link to clipboard
Copied
Another Photographer, your assets need to be on the fastest drive, you need lots of READ speed when editing. Your slowest can be the scratch disk drive. So in the most minimal setup, a Raptor for OS & Adobe Suite (don't need to RAID it really), then RAID0 or RAID5 for assets, and a good 7200rpm drive for scratch. If you do RAID0 for assets, you'd better have backup copies of everything on that drive stuck out on some external drive. Really, adding the third drive to make RAID5 is smartest so you get redundancy. That one extra drive is peanuts compared to what we spend on these rigs, lol!
Copy link to clipboard
Copied
Another Photographer, If you do RAID 5 with only 3 drives, it will distribute the OS across the set, which is my problem currently. As far as the best set up, according to Paulie and some others, is that one of the drives in the set needs to have the OS. In a three drive RAID 5 all drives are in the RAID set with the opporating system. That is how I understand it, I may be wrong as I too am still trying to figure this out. I am getting a fourth drive to do what is being suggested here, but don't know how to do that. Waiting to hear how this should be done.
Thanks
Copy link to clipboard
Copied
Thanks guys! I still have a lot to learn. Two questions/observations:
(1) I need to check whether my RAID controller allows me to attach both RAID and non-RAID devices at the same time.
(2) When I edit AVCHD, the files are only 1 or 2 Meg/sec. I'm not seeing why I need RAID for such low-bandwidth files. Please advise.
Thanks again!
Copy link to clipboard
Copied
(1) I need to check whether my RAID controller allows me to attach both RAID and non-RAID devices at the same time. I have no idea if RAID cards do that, but I would say for optimal performance, you probably want ONLY your RAID set on that card.
(2) When I edit AVCHD, the files are only 1 or 2 Meg/sec. I'm not seeing why I need RAID for such low-bandwidth files. Please advise. I don't work with AVCHD all that much but I've heard it's a fiesty format to edit. It may use a more complex codec algorithm that takes up CPU, but don't quote me on that! Others will need to pipe in and answer that one. But I would say that your interests and formats may widen in the near future so it's better to have your RAIDed editing rig already configured and running, you never know what may happen...
Copy link to clipboard
Copied
I've just ordered components to build a video editing rig and I've enjoyed reading all the good suggestions on this forum.
My question concerns where the fastest drives shoud go. Paulie mentioned that the scratch disk could go on your slowest drive, but this seems counter to what Adobe suggests.
Here is what I will be assembling:
ASUS P6X58D Premium MB
Corsair 850HX Power Supply
Intel Core i7 950 w/ CoolerMaster CPU cooler
EVGA GTX-470
G.Skill 24GB RAM Memory
Win 7 Pro, Adobe Production Premium and Lightroom
CoolerMaster Case
For hard drives I'll be primarily using these:
Crucial 64GB SSD SATA III
OCZ 60GB SSD
WD 1TB 7200RPM SATA III
WD 2TB Green SATA
Various older SATA drives for program files, storage or possibly a RAID array
External USB Drive(s) for storage
I am not into overclocking as I think it puts too much of a burden on the system heat-wise with not much real-world performance gains.
I just want a stable system that will last a long time.
My initial plans are to use the Crucial SSD as my system drive (C:) and the OCZ SSD as a cache or scratch drive.
The WD SATA III drive will hold my videos to be edited.
I figure my hundreds of thousands of photos can go on any of the slower drives.
Any suggestions as to how to set things up will be appreciated.
My main question is what to put where? How best to use my faster drives, which I assume will be the SSDs and the 6GB/s SATA III drives.
Harm suggested a static cache file equal to 1.5x the size of your memory unless you have lots of memory. What size would work best with 24GB of RAM?
Being new to Windows 7 I'll also appreciate advice on exactly how to direct things to other drives rather than everything getting piled up on the C: drive as Windows XP likes to do.
Thanks in advance.
-Bret
Copy link to clipboard
Copied
1. Crucial: OS & programs
2. OCZ: pagefile, around 12 GB static + media cache database
3. WD SATAIII: media, projects and media cache
4. Green disk: exports, auto save
5. USB: backups
Copy link to clipboard
Copied
Please let me know if there is a better way to configure these drives (especially the first three):
(1) 750GB Seagate SATAIII 7200RPM: OS, Programs, Pagefile, Media Cache.
(2) 2TB Caviar Black SATAIII 7200RPM: Media, Projects, Previews.
(3) 2TB Caviar Green: Exports
--------------------------------
(4) USB: backups
Thanks in advance.