Storing Large Amounts of Video Data Efficiently, Redundantly and Quickly Accessible

I am writing this down because I read about 50 blog articles and how tos on the web regarding this topic. Yet, I couldn’t find any good advise that fit my video data storage issue.

Like I mentioned before, I am also involved in Slate Takes Limited – a boutique video production company based out of Hong Kong. Slate Takes is currently finishing up the last touches of a 42 webisode travel series for The Vagabond Project (more info and the official trailer at their Vagabond Blog), working with clients like Red Bull at events, assisting in shooting movies, and helping international brands to turn their vision onto moving pictures. As all of our work is done at least in Full HD, you can imagine that there is a lot of data that needs to be handled, stored and backed up.

In matter of cost efficiency, nothing beats magnetic drives — still. The difference is not as big as it was some years ago, but it is still there. However, such “tapes” don’t really perform that well. Working video projects is not just about saving the final video somewhere. It is all about storing the “raw data” that comes from the camera consistently and easily accessible. If you need one day to find a tape, copy data from that tape to your computer, and then find out that the tape was damaged towards the end of it, you are f***ed. Yes, tapes do definitely provide a longer durability than hard drives or CDs, but in order to achieve redundancy, you would need to have at least two copies of each tape. We don’t even want to think about the speed if you need to access data quickly and get things done.

I have always been a fan of hard drives and RAID systems in general (about 8 years ago I had a server with 1 TB storage at home — I thought I’d never get that filled up!). We have been using a 4-bay Synology NAS loaded with 4x 2TB hard drives for about a year now. Running in RAID 5 they provide some data protection in case one hard drive has an hardware defect. On top of that, the system comes with a 1 GB/s LAN connection which allows fairly quick access. In fact, it is possible to edit videos using Final Cut Pro 7 on Mac connected to the NAS via a GigaBit switch — of course not as fast as accessing the data on a local hard drive nor on a local SSD drive; but it works. If that’s not fast enough, the GB/s connection allows quick data transfer to your local machine.

Now, one year after its incarnation, we had to look into expanding our storage again. 8 TB were filled up (from which about 6 TB were usable due to RAID5) and data kept flying in. All options were on the table, but I decided to stick with Synology and only upgrade to a 12-bay NAS which can easily be extended by another 12 bays to store up to 72 TB. Synology’s “DiskStation” software is easy to use, yet quite feature rich and low in maintenance. Furthermore, the same partitions that had been created on the first 4 TB could be kept; making the transition easier. The new setup looks like this:

The great thing is: this unit even has 2x 1 GB/s LAN connections which allows more than one user to access files with maximum data transfer rates. Two additional 2 GB hard drives were installed to increase our storage by 50%. The transition to the new system was pretty straight forward if you follow Synology’s Migration How To. This setup allows us now to expand the NAS by up to 6 hard drives at any time without worrying about needing to expand it to fit more drives. Now the “expansion process” is running and from the progress it shows so far, it might run for a while.

In case you are looking for a similar setup, here a quick wrap-up of the costs:

• Synology DS2411+ NAS: USD 1,540
• 6x 2 TB Western Digital Green (each): USD 672 (USD 112)

We will see how long this setup will last. At least for now, it comforts me that there is gonna be at least a year or two until we have to think about an alternative storage system. For now, this should be sufficient and price-wise it was well, required.

Please follow and like us: