Generic Guideline for Disk Setup

[Harm_Millaard]

There have been many questions about how to set up your disks.

Where do I put my media cache files, where the page file, and what about my preview files?

All these allocations can be set in PR, so I made this overview to help you find some settings that may be beneficial. It is not a law to do it like this, it is a generic approach that would suit many users, but depending on source material, workflow and backup possibilities, it is not unthinkable you need to deviate from this approach in your individual case.

The reasoning behind this overview is that you want to distibute disk access across as many disks as possible and get the best performance.

Look for yourself:

I hope this helps to remove doubts you may have had about your setup or to find a setup that improves performance.

[ECBowen]

Question with regards to the internal raid 5. If you lose 1 to 3 days worth of work from drive/data failure, will the cost be prohibitive to you or your company?

Eric

ADK

[Switch7]

Not at this point no.

ECBowen wrote:

Question with regards to the internal raid 5. If you lose 1 to 3 days worth of work from drive/data failure, will the cost be prohibitive to you or your company?

Eric

ADK

[ECBowen]

I would still not recommend a 4 drive raid 0 because of the possible failure rate, but since the cost wont be prohibitive then it would not be catastrophic if a drive failed.

Eric

ADK

[Switch7]

I was thinking that so I could have the maximum throughput while editing. I would only make a copy of the files I am using onto the raid 0. The Project files are on the SSD Drive with a back up of those on the raid 5. So in theory if the raid 0 goes down I can still continue working from the raid 5 array. It would just be a matter of relinking to the footage on the raid 5. Correct? or am I wrong in this line of thinking.

ECBowen wrote:

I would still not recommend a 4 drive raid 0 because of the possible failure rate, but since the cost wont be prohibitive then it would not be catastrophic if a drive failed.

Eric

ADK

Thanks.

[ECBowen]

That will minimize your loss yes as long as you keep source and exported files elsewhere as well.

Eric

ADK

[Switch7]

Yes, but the real question is...it this the best and most efficient use of my

ECBowen wrote:

That will minimize your loss yes as long as you keep source and exported files elsewhere as well.

Eric

ADK

drives. Or would something like a raid 10 be better? I just want to find the most effective way for speed and redundancy as possible. With all these drives.

[ECBowen]

Raid 5 is the best combination of speed and redundancy. I would keep the current config and not make the change to a 4 drive raid 0 unless you needed to handle uncompressed HD files or more than 5 layers R3D material at 4K.

Eric

ADK

[Switch7]

Basically right now I run a core i7 920 OC to 4ghz, with 12ghz of

ddr3 1600 ram and a GTX 480. I built it for CS5 and DSLR Editing, but I am still getting stutter on playback even at 1/2 resolution when I apply a simple disolve transition. It is very frustrating I am hoping the through put of raid 0 would help that. As my system often seems to get severly bogged down once the timeline starts really getting built. I have currently been trying keep my timelines minimal and tidy to work around this. But It's not always possible. I find it is the worst if I am transitioning with alot of bright or whites in the footage. Dark footage is not as bad.

Basically the adobe TV MPE articles are a pipe dream to me so far.  And my windows 7 experience index rating is maxed at 7.9 across the board all except the Disk data transfer rate which is 5.9. So that seems to by my biggest bottleneck. Now whether this mean anythign to CS5 remains to be seen but If raid 0 gets me smooth playback then I am a happy lad.

[ECBowen]

Well your software raid could be slowing the process down so it doesn't hurt to try the raid 0. That material should be able to play 1/2 or full scale on a 920 @ 4GHz without issues. Something else is going on. BTW that HDD readout for Win 7 is just for the OS drive. Unless you have a SSD or Raptor drive it's going to read in the 5.8 or 5.9 area. Unless you have files that should not be going to your OS drive going there, then you playback should not be effected by that. I would be interested in remoting into your system and seeing how you have things setup and the Adobe configuration. Something is odd.

Eric

ADK

[Switch7]

ECBowen wrote:

Well your software raid could be slowing the process down so it doesn't hurt to try the raid 0. That material should be able to play 1/2 or full scale on a 920 @ 4GHz without issues. Something else is going on. BTW that HDD readout for Win 7 is just for the OS drive. Unless you have a SSD or Raptor drive it's going to read in the 5.8 or 5.9 area. Unless you have files that should not be going to your OS drive going there, then you playback should not be effected by that. I would be interested in remoting into your system and seeing how you have things setup and the Adobe configuration. Something is odd.

Eric

ADK

Hmmm, I have a Vertex 60gb SSD as my OS and Project File Drive.

I have no idea how to have you remote into my system and forgive me for declining the request as you are an internet stranger and my mom always said to not open the door to strangers.

Is there anyway I can take screen shots or something for you?

Thanks for the offer to help.

[poppabear69]

Did you have stutter when you were at full resolution?

I had a problem with stutter when I first installed my EVGA GTX 275 in MPE.  It was discussed in the CS5 forum that MPE works better at full resolution.  Sure enough, I haven't had a problem with stutter since I have been using full resolution.  I am using the four HD configuration from Harm's recommendation and an i7-920.

[Switch7]

poppabear69 wrote:

Did you have stutter when you were at full resolution?

I had a problem with stutter when I first installed my EVGA GTX 275 in MPE.  It was discussed in the CS5 forum that MPE works better at full resolution.  Sure enough, I haven't had a problem with stutter since I have been using full resolution.  I am using the four HD configuration from Harm's recommendation and an i7-920.

Yes I get the stutter at all resolutions. Unfortunately I started with the GTX 275 got frustrated so bought the Gtx 480 with minimal improvements on playback. Rendering timelines out is much faster but timeline playback stutters.

I think I might go with a 4 disk raid 10 and work from there. But the question I have regarding the guidelines is: would having all working files on a raid 10 array in one place be slower than spanning them across multiple drives like the 4 drive set up? Should I keep the footage on the raid 5, run all previews and exports and media catch on the raid 10? or have another for exports as well?

[ECBowen]

We have licensing with Teamviewer for remote access and that is how we remote in. You just need to download the client which is a stand alone file from www.teamviewer.com. We use Team viewer because it is easy for the client and safe. It requires a person on the other end to give us a user id and password to remote into your system. That was simply an offer because something was odd. I would definitely try what the previous poster said regarding full scale. The MPE and video card does handle scaling so it makes sense. That system you have should easily handle that material at full scale.

Eric ECB

[Switch7]

ECBowen wrote:

We have licensing with Teamviewer for remote access and that is how we remote in. You just need to download the client which is a stand alone file from www.teamviewer.com. We use Team viewer because it is easy for the client and safe. It requires a person on the other end to give us a user id and password to remote into your system. That was simply an offer because something was odd. I would definitely try what the previous poster said regarding full scale. The MPE and video card does handle scaling so it makes sense. That system you have should easily handle that material at full scale.

Eric ECB

My SSD is the OCZ Vertex 60gb. I thought the 5.9 was odd, because when I originally ran it new it was 7.4 I think. So I did some digging and looked up Trim, I supports trim and from my understanding windows 7 automatically takes care of TRIM. Is that correct?

[ECBowen]

Yes Win 7 supports Trim by default for any SSD drive that has trim support in the firmware. I was looking up that drive and reviews to see what controller is has and trim support. I could not find the controller yet but it's suppose to have trim support based on the review. Now they did not mention if they had to update the firmware for the trim support like you have to on other drives including Intel SSD. You might want to call OCZ and see if they have a firmware update for that drive. One last note was the Sequential performance results.  They were in the 160MB/s to 180MB/s range which is not bad at all. I still can't find the controller though to see if it's one that has the latency issues for asynchronous commands. I will check on OCZ's website and see if they have more info. Try downloading HDTune Pro trial and see what the SSD is reading out for Read performance currently.

Eric

ADK

[Switch7]

ECBowen wrote:

Yes Win 7 supports Trim by default for any SSD drive that has trim support in the firmware. I was looking up that drive and reviews to see what controller is has and trim support. I could not find the controller yet but it's suppose to have trim support based on the review. Now they did not mention if they had to update the firmware for the trim support like you have to on other drives including Intel SSD. You might want to call OCZ and see if they have a firmware update for that drive. One last note was the Sequential performance results.  They were in the 160MB/s to 180MB/s range which is not bad at all. I still can't find the controller though to see if it's one that has the latency issues for asynchronous commands. I will check on OCZ's website and see if they have more info. Try downloading HDTune Pro trial and see what the SSD is reading out for Read performance currently.

Eric

ADK

It's the vertex 64gb, it does have the firmeware 1.5 that supports trim. It was new enough to come shipped with that. As for the Read performance it is average of 110 mb/s Min 71 and Max 188.

But when I looked at the Health tab of HD TUNE PRO it is showing a warning in the (C7) Ultra DMA CRC Error Count. The discription says there were communication errors this may be caused by a damaged Cable.

Intereting.

[ECBowen]

Ya that read speed is low. It sounds like you have an SSD with bad NAND Flash or a bad stick of ram causing the controller to send corrupted data to the drive. Also the overclock could be causing the ram errors. You should test the ram in current config and then test it at stock if you are getting errors with the overclock.

Eric

ADK

[Switch7]

ECBowen wrote:

Ya that read speed is low. It sounds like you have an SSD with bad NAND Flash or a bad stick of ram causing the controller to send corrupted data to the drive. Also the overclock could be causing the ram errors. You should test the ram in current config and then test it at stock if you are getting errors with the overclock.

Eric

ADK

Well I am happy to report that after much digging and one question leading to another I found that Trim needs to be left on the login screen for a couple of hours each week for it to do it's job. If you never give your ssd a break Trim can't clean, I then found  on the OCZ site the "Wiper" clean up utility. OCZ said to use it once a month on the Vertex and other drives. I didn't think this was neccesary because "Hey I got TRIM" but thought what the heck. Lets give it a try. It took all night to run the clean but results afterwards show things back to new. The new HD Tune test is now 144mb/s up from 110mb/s and my Window experience index is up from my last test of 5.5 to 7.2. I guess that is what happens when you don't do any ssd maintanence for 4 months.

Note to everyone. TRIM still needs the "Wiper"

As for New Disk Set up. After testing Raid 10 and Raid 0. The Raid 10 with 4 drives giving me only 1.2TB gave me the same read numbers as the Raid 5 which was around 135mb/s that was there before it. I dont' know what the writes were as the write test didn't work. The Raid 0 gave me 208mb/s with 2.2TB of space, still don't know the write speed for that either. But when I transfer large files from the Raid 5 to the Raid 0 I am getting 115mb/s which seems to be consistent with the HD Tune test of 108mb/s I beleive this all has to do with the tranfer raid of the External Raid unit but strangely enough a tranfer test from the SSD to the Raid 0 is actually worse at around 75mb/s.

How this will all effect my editing playback remains to be seen. I will have a better idea by the end of the day.

So now I have all Media Footage on External Raid 5 Drive, I transfer the project I am currently working on to the Raid 0 and have Footage, Media Cache and Previews all running from the Raid 0. I have Project files on the SSD with a back up of those on the Raid 5 so there is all Project Files and Raw Footage and exported movies safe on Raid 5. I created a 20GB Partition at the begining of the Raid 0 for my Pagefile and then use my last 500Gb loner drive for all Windows related Back ups.

Lets see how it all works out.

[ECBowen]

Excellent news.

"Note to everyone. TRIM still needs the "Wiper"

This is absolutely not correct. Wiper is a manual Trim and is meant for certain drives and is manual. Intel drives seem to have their auto trim feature functioning better so I would not use it for those. Some report issues with Wiper when the ICH is on raid or AHCI mode. You have to be careful with this and see if the drive manufacturer supports this utility. They will have it as a download if so. Some drives support the "garbage collection" instead so you don't want to use the wiper for those drive it seems. If you are not sure call the manufacturer support line and ask them what if any should you use for this.

BTW those CRC errors are still at play and trim would not have an effect on those. You might want to test your ram.

Eric

ADK

[Switch7]

Ok I stand Corrected. If your running a OCZ SSD, go to their site and see if they recomend you run the "Wiper" They did for my Vertex and they do for the others in that line.

As for the error. I tested the ram with no OC and the error is still there. So I did more digging on the OCZ Forums and apparantely HD Tune Pro does not read SSD Drives properly. There are many people getting this error with the Vertex while using HD Tune PRo and moderators are saying it's an HD Tune Pro Error. Not the SSD.

So as for now I will see how it all runs, if it all runs smoothly I will beleive them. If not I will have to find a utility that tests the ram like you say Eric.

Thanks.

[ECBowen]

Thanks for the heads up on SSD and HDTune. I will have to check that here as well.

Eric

ADK

[Switch7]

Well I am happy to report after a solid day of editing that my new configuration works amazingly well comparted to my old raid 5 setup. Everything was working as advertised. I now run all my footage and scratch disks from the 4 disk raid 0 array with project files on my ssd OS Drive and the back up footage and Project files backups on the External Raid 5 Array.

The only problems I encountered was when I was trying to create a funky slideshow with JPGs I took with my 7D, I didn't scale them down so they were full 18mp resolution. After the project began to become pretty complicated it has become very unstable and keeps crashing with an Nvidia Error, saying something like my system does not meet minimum requirments. Something like that. I dug around and found others were having the same issue when trying to edit picture in their timelines. Looks like I will have to build it in AE.

The other problem was stutter on 2 random clips with a disolve added. After digging more I finally found that CS5 has a know issue with AVCHD Footage that has a dissolve between 2 parts of the same clip. So I tried the work around of making a copy of the footage in explorer and importing the copy. Then using that for the second clip in the timeline. Bingo! It worked.

Multiple layers are so far playing like butter. Lets hope it holds up as the sequences get more complicated as they get nearer to completion.

Thanks for all your help.

[UlfLaursen]

Harm, I have one question for this guide - just to be sure.

I have 4 discs all SATA >= 7200 rpm. For disk 4 (disc F) you write "Previews and Export" - do you mean "Preview Renders"?

Thanks.

/Ulf

[Harm_Millaard]

Yes Ulf, both the rendered files and the final destination of your exports. The thinking behind it is that on export you need access to the original media, to the media cache and media cache database files, possibly the pagefile, depending on your memory, but not often to the rendered files, because if you don't have MPE, you will use MRQ and not the preview files, and if you have MPE, you do not really care since MPE is so fast. So what you try to achieve is maximum spread across your different disks.

[UlfLaursen]

Thanks for clarifying it Harm.

/Ulf