Using External HDD's for Video Editing

External HDD’s (Hard Disk Drives) are everywhere, and quite popular. These allow one to increase the storage capacity of a system greatly, quickly and easily, for not that much money. We now have many sizes, and many connection types. Some connection types are better than others, if one is doing more than off-loading data to archive it, and perhaps free up internal HDD space. With the right connection type, one can also edit to/from the external HDD’s. Again, some connection types are better than others. Note: for editing Video, the physical HDD’s need to be at least 7200 RPM.

For simple archiving, any connection type will work well. In that process, files are just written to the external HDD, and when needed, read from the external HDD. The problem comes in, when one tries to edit to/from the external HDD. Most current computers are faster than the slower connection types can manage. The OS is firing commands for reads/writes too fast. The OS expects rapid response, and can overwhelm the external HDD. When one adds in an NLE (Non Linear Editor) program, there is a great deal of HDD activity, and this requires many reads and writes, and requires them quickly. The files are very large too. Suddenly, the I/O (Input/Output) sub-system is overwhelmed, but the requests keep coming.

Here is a real-world look at the data transfer of some common connection types:

Average transfer rates in MB/s for different interfaces:

• USB2:    20 - 25, depending on other USB devices sharing the same bandwidth
• FW400:  30 - 35
• FW800:  50 - 60
• USB3:    65 - 80, depending on other USB devices sharing the same bandwidth
• eSATA: 100 -140
• SATA:   100 - 140

Raid0:    0.9 x N disks over a single disk

Raid3/5: 0.8 x (N-1) disks over a single disk for read, 0.6 x for write. ICHR10 figures are a bit lower than hardware controllers.

Note: “FW” refers to FireWire, which might also be referred to as IEEE-1394a (FW-400) or iLink, and as IEEE-1394b (FW-800). The connectors are different, as are the cables. Also, one needs the appropriate controllers. Most computers have FW-400/IEEE-1394a/iLink controllers on the MoBo and connectors available for the cables. For FW-800/IEEE-1394b, one usually needs to add a controller, either on a PCI card, or via PCMCIA, or ExpressCard. More on controllers later.

As one can see, the USB 2.0 connection is slower than the rest. Problems can arise, when trying to edit to/from such an external HDD. They can be plagued with read/write errors, as they cannot maintain the data transfer, required by the OS and by the NLE program. When the allowed time has expired, based on the OS’s expectation of data transfer, it stops the process, and issues a read/write error. In a worst-case scenario, the OS will have routed the write data into a buffer, waiting for the external HDD to catch up. These buffers are finite in their capacity to store the data, while the OS is waiting. It does not take too long, before the dreaded “Delayed write failure” rears up. This can lead to a catastrophic failure of the external HDD, as the data is only partially written, and the File Allocation Tables are not completely written, causing the loss of ALL data on the external HDD. At that point, all data on the external HDD is lost. It might be possible to do a recovery with a program, like Stellar-Phoenix, but it is not cheap, not is it fail-safe. Depending on the value of the files lost, one might hire a data recovery service, but these are anything but cheap. Probably best to just do a complete low-level Format of the external HDD and then do an OS Format.

Also, while speaking of Format, most external HDD’s come from the factory, Formatted to FAT-32, an older Format type. Besides being slower, FAT-32 has a max. file size of about 4GB, and AV files are often much larger. One needs to CONVERT the Format to NTFS for the PC, or to the Mac OS’s Format, to allow for the larger files, and to increase the speed a bit. On the Mac/PC front, a PC can read/write to FAT-32, or NTSF. However, a Mac can ONLY read/write to a PC external HDD, Formatted to FAT-32. If one is migrating the HDD’s between Mac’s and PC’s, the external needs to remain in FAT-32 with its limitations. Just one consideration, when setting up external HDD’s for use with an NLE program.

For editing Video, I have found that the connection types should be considered this way:

• USB 2.0 - archiving for storage only
• FW-400 - archiving for storage, and light editing to/from (just very slow)
• FW-800 - archiving for storage, and regular editing to/from (fairly fast)
• eSATA - archiving for storage, and regular editing to/from (about as fast as an internal SATA)
• I have not tested USB 3.0 in editing situations, so cannot comment on their acceptability.

Obviously, a SATA internal is ideal for all tasks.

Now, back to controllers. The real-world throughput of an external can be diminished due to the controller, and the number of chips available. Many MoBo’s have 1 - 2 USB 2.0 chips, and 1 - 2 FW-400 chips. One needs to check the number of chips provided. One might have multiple connectors, either on the computer’s case, or direct from the MoBo. These connectors might go to multiple controller chips, or might all feed into one. The same can be said for add-on cards, PCMCIA cards, or ExpressCards. The number of connectors does not guarantee the number of chips.

The more externals, and peripherals that are connected to the controller chip, the lower the data transfer rates. This holds for hubs especially. One might have various externals and peripherals plugged into various connectors, but everything being fed through a single controller chip. Add to that, the ability to daisy-chain multiple externals through the In/Out connectors on most external HDD’s, and overall performance falls even more. USB 2.0 and FW-400 suffer most from these multiple externals limitations, but FW-800 can too, just not to the same degree.

When one is using an external to edit to/from, there is another consideration. When peripherals with mass storage capabilities are plugged into a computer, a drive letter is assigned by the OS. Most NLE programs, like PrE/PrPro rely on the drive letter to link to the Assets. They do this via an absolute path, including that drive letter. Depending on how many other peripherals are plugged in, and the order that they are plugged in, that drive letter assignment might differ, session to session. When using external HDD’s, one should go into the OS’s Disk Management console and assign a fixed drive letter for that individual external HDD. This would hold for however many one is likely to use, and each should have a unique drive letter. I suggest marking each external with the drive letter that the user assigns to it, say Z:\. Then, whenever Z:\ is plugged in, it will always be seen as Z:\. This way, the NLE can keep up with where the Assets are located, starting with the drive letter. If one is migrating Projects between computers, they will repeat this exact process in the OS of each computer.  Note: when doing the migration, ALL Assets, Scratch Disks, and the Project file, MUST be included on that external, or one will spend a good deal of time tracking down the Assets.

External HDD’s are great devices, but some considerations need to taken into account, before one just plugs them in, and begins editing.

Hunt