How I upgraded To a 144 TB Thunderbolt 3 Direct Attached Storage
As a photographer storing, backing up, and working with image files is almost important as capturing them in the first place. Having a reliable and efficient workspace is critical to producing quality work in the most efficient manner possible. This topic is also as personal as deciding which camera to buy or vehicle to drive; ask three different people their approach to this and you’ll likely get three different passionate answers. While I had a solid backup solution already in place, I was in need of upgrading my image editing workspace. The following is a short synopsis of my thought process and experience updating my image editing workspace to a Thunderbolt 3 144 TB 12-Bay RAID Direct Attached Storage (DAS) solution. In this post I will cover the current leading Thunderbolt 3 DAS solutions on the market from Solid State RAID devices, to expansion chassis options, to traditional rotational platter hard drive solutions. I’ll also touch on the performance of the DAS solution I chose, its initial RAID5 configuration and basic administration tasks as well as the importance of file systems utilized on the volume. Finally, I’ll close with what my approach to redundant file backup from DAS to NAS to offsite cloud backup is. This isn’t meant to be an all-inclusive tutorial on things such as RAID and it certainly won’t include every model on the market, only the top leading options I was considering.
Thunderbolt 3 DAS Solutions
After upgrading to the iMac Pro and having the option of utilizing Thunderbolt 3 speeds (40Gbps) I was really excited to see some impressive Thunderbolt 3 products hit the market. Not only are there currently great Thunderbolt 3 DAS solutions on the market right now, but there are also impressive Thunderbolt 3 10GbE network solutions, eGPU, and a plethora of expansion chassis that really make for exciting performance increase options for a variety of high-end user requirements from demanding video editing (4K, 5K, 6k, and even 8K) and increased Adobe and Lightroom and Premiere Pro optimizations to 3D modeling and CAD performance. The iMac Pro comes with a 10GbE port, the ability to add another 10GbE port via Thunderbolt 3 would enable the option of configuring a 10GbE link aggregated solution for high-speed networking. You can even direct connect to a 10GbE NAS. Why is this exciting? A 10GbE network link can transfer at 1250MB/s, so configure those with link aggregation (LA) and double the speed, you will see 2,000MB/s+ speeds over Ethernet! This is starting to challenge Enterprise level iSCSI offerings at a much lower price point! Perhaps Thunderbolt 4 will challenge InfiniBand Awesome. LA 10GbE links are still nowhere near the 5,000 MB/s potential of Thunderbolt 3 though, which is one of the reasons I’m so excited about Thunderbolt 3.
I contemplated a plethora of Thunderbolt 3 DAS solutions produced by a variety of manufacturers before deciding on one. I will go through my thought process and how I finally decided on a solution that is right for me. Again, the purpose for my DAS was nothing more than a “workspace” for me to edit and process work on, it’s not meant to be a backup solution (see NAS section for that). The first question you have to decide upon is a budget for your DAS as that will obviously be a major factor in what you can include as a candidate. That said, I didn’t initially let a budget limit my research because I wanted to know what the market had to offer from a technology/performance perspective and what those cost points were. The goal is to get the most for your money obviously. So my first step was to look at four main Thunderbolt 3 DAS solutions, 1) a Solid State Drive (SSD) solution, a 2) PCIe Expansion chassis solution, and 3) a traditional RAID hard drive (rotational/platter) solution, and 4) a performance NAS with Thunderbolt 3 DAS connectivity. Each option has advantages and disadvantages, and while some are practically not obtainable from a cost perspective I think it’s important to know the market and what you are actually getting for your money and where the performance bottle-necks are for your setup.
Solid State RAID DAS Solutions
Even though SSD solutions have come a long way in both performance and affordability the reality is that they are still cost prohibitive at larger storage capacities (16 TB and above). SSDs offer very good performance, but to truly get top performance reads/writes you still need to run them in a RAID configuration such as RAID 0. Once you look into running SSDs in a RAID 0 configuration you very quickly hit the performance limits of the hardware RAID chip the manufacturer has engineered into the chassis. It is this chip that can quickly drive up costs and determine market segments from power-users to Enterprise leaders such as NetApp. One work-around is Software RAID, which in some cases will outperform hardware RAID depending upon the processing power of the system running it. One of the newer SSD hardware RAID arrays on the market as of April 2018 is the G-Technology 16TB G-SPEED Shuttle 8-Bay Thunderbolt 3 SSD RAID Array (8 x 2TB) which retails for $7,499.95. At this price-point you get 16TB of SSD storage that can transfer data at up to 2,800 MB/s. One cheaper alternative is to go with Software RAID as in the Akitio Thunder3 Quad X 4-Bay Thunderbolt 3 RAID Array Enclosure which retails for $369.00. While it can only hold a maximum of 4 drives it has a maximum transfer speed of up to 1,480 MB/s which isn’t quite as fast as the former; however, if filled with 4 Samsung 4TB 860 EVO SATA III 2.5″ Internal SSD which retail for $1,376.31 you could have a 16TB SSD setup for $5,874.24 which is $1,625.71 less than the G-Tech. It’s just a matter of how much you need that extra transfer performance of 1,320 MB/s the G-tech SSD unit would give you. There are other 4-bay alternatives like the OWC / Other World Computing ThunderBay 4 Four-Bay Thunderbolt 3 RAID 0 Enclosure, that offer SoftRAID Lite solutions for a slight speed increase to 1,527 MB/s. I used to run a 2-bay Akitio SSD solution I mentioned earlier, the Akitio Thunder3 Duo Pro 2-Bay RAID Enclosure has a great build quality with the down-side being that the hardware RAID chip is limited to a max transfer speed of 785 MB/s, much too slow for my needs. While there are a few other manufacturers like LaCie and Promise, they tend to only sell their RAID units with their own pre-populated hard drives as does G-Technology. As you can imagine, purchasing an Enterprise level drive separately can be more cost effective.
DAS PCIe Expansion Chassis Solution
If money were no object, absolutely insane speeds can be achieved utilizing Thunderbolt 3 and PCIe 3.0 x16 host interface cards such as the HighPoint 8TB rSSD7101B Professional NVME RAID SSD which retails for $7,099.00, but offer sequential reads up to 13,500 MB/s and sequential writes up to 8,000 MB/s. There is certainly an immense advantage to having your high-speed memory wired directly on the bus opposed to an SSD drive. A cheaper alternative to this card would be something like the Intel 480GB Optane 900P PCIe 3.0 x4 Internal SSD which retails for $565.99 and offers sequential reads of up to 2,500 MB/s, but are much, much smaller in capacity. A PCIe expansion card over Thunderbolt 3 requires an expansion chassis such as the Akitio Node Pro Single-Slot PCIe to Thunderbolt 3 Expansion Chassis for $349.99. My dream setup would be comprised of several DAS units consisting of dual HighPoint cards each in a RAID0 stripe.
DAS RAID Traditional Hard Drive Solution (rotational/platter)
One of the first traditional RAID units I considered was building my own FreeNAS Linux based solution, but I’m not one of those people that like Technology just to like Tech. For me, technology is a tool to solve problems and most importantly to create, nothing more. I did not want to spend my free time moonlighting as a Linux Sysadmin. That said, I’m still interested in how well an add-in PCIe Thunderbolt 3 expansion card would be supported in FreeNAS, because a solution like this is really turning a NAS into a DAS which isn’t a problem if the FreeNAS OS supports the needed Thunderbolt 3 card kernel modules. This could be amazing with a lot of time and work. My requirements for a DAS were 1) it had to be low-maintenance and completely hands-off after initial configuration, 2) it needed to allow for customization, 3) while allowing for as much performance as possible in RAID5, with 4) as large of a capacity as possible, 5) had to provide Thunderbolt 3 (40 Gbps) connectivity interfaces, and 6) support both SAS 12 Gb/s backplane interfaces in addition to the slower SATA III 6 Gb/s. When I first started looking, the G-Technology G-SPEED Shuttle XL 48TB 8-Bay Thunderbolt 3 RAID Array (8 x 6TB) was a major contender as it was brand new to the market place offering up to 2,000 MB/s transfer speeds. The G-Technology units are extremely popular in the video and photo editing worlds, but I wanted to know, was it worth $4,999.95 to me in both performance and capacity? Using the Thunderbolt 3 Shuttle XL as a baseline I started looking at other units like the HighPoint RocketStor 8-Bay Thunderbolt 3 RAID Enclosure which has a very attractive price point of $1,299.00 and transfer speeds of up to 2,400 MB/s with SSDs so I knew the RAID chip it was using was decent, but after coming across the Areca ARC-8050T3 12-Bay Thunderbolt 3 RAID Storage everything I read kept pulling me towards the Areca. Areca makes a 24-bay 4U Rackmount server in addition to smaller 12, 8 and 4-bay units. I liked the fact Areca was making higher-end rackmount servers; albeit a bit odd because having a 24-bay rackmount server and using it as a DAS via Thunderbolt 3 would really require a sound-deadening enclosure otherwise the noise from this 24-bay would drive anyone insane! You can’t just run a 100 ft Thunderbolt 3 cable because they don’t exist, at least not yet. Corning does makes a 100 ft optical Thunderbolt 2 cable for $529.00 at B&H, but as of this writing I haven’t seen a Thunderbolt 3 optical cable. Needless to say I was drawn to the desktop 12-bay Areca ARC-8050T3 DAS with a Dual-Core 1.2 GHz LSI RAID-On-Chip and 2 GB of ECC RAM. Not only would 12-bays offer more performance and expandability than the smaller 8-bay units, but the Areca units are known for having an absolutely awesome tank-like build quality with performance to match at a price point of $2,299.00 (I got mine for $2,099.00 from B&H during the NAB show). So what the Areca enabled me to do was to get a base DAS 12-bay quality unit with stellar performance utilizing an industry standard LSI RAID-On-Chip, which in turn enabled me to put the money into the drives themselves. I filled my 12-bays with Seagate 12TB Enterprise Capacity 7200 rpm SAS III 3.5″ Internal HDD drives for a total capacity of 144 TB (132 TB usable in RAID5 after parity). So for $7,354.88 I have a 144 TB Thunderbolt 3 DAS solution that has performance faster than some SSD units. I will go more in-depth into performance of my Areca in a bit, but I was able to achieve 144TB at a lower price point with superior performance than any of the pre-configured DAS Thunderbolt 3 units! For example, the LaCie 12big 120TB 12-Bay Thunderbolt 3 RAID Array (12 x 10TB) DAS costs $7,699.00 which is $344.12 more and has 24 TB less storage capacity as my Areca 12-bay setup! Likewise and even more telling is the G-Technology G-SPEED Shuttle XL 96TB 8-Bay Thunderbolt 3 RAID Array (8 x 12TB), which costs $9,999.95 for only 96 TB! That’s $2,645.07 more with 48 TB less than my Areca 12-bay!
Performance NAS with Thunderbolt 3 DAS connectivity
One of the other solutions I had considered was to utilize a high-performance NAS that had Thunderbolt 3 connectivity in order to utilize it as a DAS. I first considered the QNAP TVS-1282T3 12-Bay NAS Enclosure because of its 4 Thunderbolt 3 ports and it also has two 10GbE ports out of the box with no need to add in a PCIe card! In addition, the unit comes with a 3.6 GHz Intel Core i7 and 64GB RAM. All that will cost you $3,999.00 with no drives. One of the things I did not like about the QNAP units though were the mixed drive bay sizes. For example, this 12-bay unit only has eight 3.5”/2.5” bays and the other four bays will only accommodate 2.5” SATA drives. I also didn’t like that the unit only supported SATA III and not SAS drives, but the unit does support two bays of M.2 SATA III slots which can be useful. The performance was limited up to 1,600 MB/s, which isn’t quite as fast as I was looking for either. QNAP also makes a 16-bay (mixed drive size) unit QNAP TS-1685 16-Bay NAS Enclosure which is currently one of their top of the line models that offers impressive features such as a 2.2 GHz Xeon 6-core processor and 128 GB ECC RAM. This unit claims transfer speeds of 2,339 MB/s reads and 1,974 MB/s writes all for $5,299.00 with no drives. The unit does not come with any Thunderbolt 3 connectivity so adding a Thunderbolt 3 PCIe card for DAS connectivity would be required. I do like the QNAP units and may consider one for a future NAS upgrade, depending on what Synology has available at that time. As much as I love my Synology NAS units, Synology does not have any great offerings for using their top-tier performance units as DAS devices. Synology are made for NAS only and the only Thunderbolt options they offer are consumer-grade Thunderbolt 2 options and nothing for the professional photographer or videographer in a DAS setting. That said their NAS units are killer. As of this writing there is speculation Synology will release an 8-Bay Thunderbolt 3 solution, which is definitely a step in the right direction, but it will still not have the performance of the Areca 12-bay. Before I dive into the real-world performance of my Areca 12-bay I need to discuss file systems.
The environment you will be using your DAS in will play a key role in determining the file system you format it as. Normally this is transparent to the consumer as they simply use whatever the manufacturer formats their Costco or Best Buy purchased hard drive as, but when you are dealing with sizeable 100 TB+ sized arrays the file system becomes a major area of consideration. It’s not black and white and each user will have different needs especially depending on what type of data you most deal with, the type of hard drive you will be using (SSD, rotational, etc…), and especially if you are using your system as a NAS or DAS. The file systems I considered for my DAS were the B-tree File System (Btrfs), Z File System (ZFS), Apple File System (APFS), Mac OS Extended (HFS+), and Extended File Allocation Table (exFAT). I quickly ruled out a few. Since I’m an MacOS environment and do not use Windows I especially didn’t want to format my array as exFAT, but if you use both Windows and MacOS or will be alternating between those environments exFAT will ensure cross-system compatibility. The Btrfs file system is now gaining support for use in Synology NAS units and offers some awesome features. Btrfs works great in a NAS environment serving files over the network, because the Synology Linux based OS has Btrfs support built into the Linux kernel. MacOS Sierra does not natively support the Btrfs file system unfortunately and I did not want to hack together an unsupported solution. ZFS offers some of the same benefits as Btrfs and has an MacOS High Sierra installer. The feature-set of both Copy-On-Write file systems like Btrfs and ZFS is quite impressive and one of the major draws that makes me want to use ZFS is checksums and self-healing from bit-rot. At a high level bit rot is the slow deterioration in the performance and integrity of data stored on storage media. It is also known by the names bit decay, data rot, data decay and silent corruption, but the principle is that basically for a variety of possible reasons some of the 0’s or 1’s flip and create corruption in a file on the file system. Both ZFS and Btrfs offer a much more elaborate approach to file system data integrity through a variety of mechanisms like checksums and snapshots more so than any other file system mentioned here. As a new file system I’m curious why APFS does not utilize checksum integrity? Like the other file systems there are advantages and disadvantages with each, ZFS has its own drawbacks as well. Moving on to APFS, Apple recently rolled out APFS in MacOS High Sierra and it is a completely updated file system rebuild. APFS is Apple’s update to their HFS+ file system optimized for SSD devices, but can be used on external DAS units as well just not fusion drives. I did some of my testing in APFS and I so wanted to like this file system, but there were two issues that gave me concern with using APFS. 1) There is a known current bug in APFS discovered by the developer of Carbon Copy Cloner with APFS formatted sparse disk images, and 2) the tendency for APFS to fragment files through duplication and negatively impact performance. I was less likely to be concerned with the former issue, but APFS performance on traditional hard drives especially as they become more saturated with data ultimately became a deciding factor for me not to use APFS on my DAS. ZFS was definitely the top contender, but there was no native support for it in MacOS and having to administrate updates for OpenZFS as they are released was certainly a detriment. So what file system did I ultimately decide to go with? I chose HFS+ because it was a tried and true fully supported file system that offered excellent performance. While HFS+ does not have nearly the intelligence that ZFS and Btrfs or even APFS has, it’s still currently the best option for my needs. However, ZFS is still calling my name!
Performance: Areca ARC-8050T3 12-Bay Thunderbolt 3 RAID Storage
Before I go into the results from my testing (hint: look at the image above and below), Areca has published their own results from their performance testing. I used a combination of Blackmagicdesign’s Disk Speed Test and AJA System Test in conjunction with timed copies from 2TB SSDs to the Areca 12-bay unit. To test the performance of each RAID stripe size the Areca 12-bay supported which is 64K, 128K, 256K, 512K, and 1024K I used the test applications above to standardize file reads/writes at preconfigured file sizes from 256MB up to 4GB. I also added in a series of Lightroom Classic CC tasks like navigating thousands of RAW files and time it takes to fully load a RAW image in the Develop Settings module. As a photographer who has terabytes upon terabytes of images and video, these test applications offered a fairly accurate methodology. These tests were extremely time consuming and took about 2 weeks to complete because each time I needed to test a different stripe size I had to blow away and recreate the RAID volume with a new stripe sizing. Areca does offer the option to migrate a stripe sizing to a different size, but it was quicker and easier to create a new volume. Each time I did this though it took about 13 hours to recreate the 144 TB RAID volume! I also tested using standard 512 byte hard drive sectors, more on that below.
RAID strip sizing is also referred to as “chunk size”, but basically it’s the size of the pieces of the stripe spread across each drive. Another factor that is starting to evolve in the storage industry as rotational hard drives grow to massive sizes is the logical block format size known as a sector. The International Disk Drive Equipment and Materials Association (IDEMA) has established what is called the “Advanced Format 4K”, which increases the hard drive’s sector size from 512 bytes up to 4096 bytes known as 4K or 4Kn sectors. This provides for more efficient use of large disk capacities as well as increased performance. This is something that I actually did not have the opportunity to test as the largest SAS drives B&H had in stock were the Seagate 12TB Enterprise Capacity 7200 rpm SAS III 3.5″ Internal HDD and since I was going for capacity and performance I compromised with the traditional 512 emulation. B&H did have 8 TB 4Kn SAS drives in stock, but I needed more space. 14 TB drives are now entering the market as well. The performance I’m seeing from the Seagate 12 TB Exos Enterprise helium filled drives is impressive with up to 261 MB/s Sustained Data Transfers.
After weeks of testing each stripe size with preset file sizes I took three readings for each test after which I averaged the three tests. The overall top performing stripe size for my needs turned out to be a stripe size of 256K, which resulted in an average read of 2,242.55 MB/s and an average write of 1,834.80 MB/s write! This is something I’m extremely pleased with considering the massive capacity of this array size at 144 TB.
Configuration: Areca ARC-8050T3 12-Bay Thunderbolt 3 RAID Storage
The configuration and administration of the Areca 12-bay DAS is straightforward. First, here are links to the Areca installation quick start guide, the user manual, ArcSAP quick manager, the command line interface (CLI), and MRAID MacOS installer. There are several ways to accomplish configuration and administration of the Areca 12-bay. You first have the MRAID software option followed by the web interface, but also an application called ArcSAP, and finally the Command Line Interface (CLI). I prefer using ArcSAP because it integrates commonly accessed information with the addition of including a terminal shell for CLI administration. I’ve also used the CLI straight from MacOS terminal which is nice if you want to nerd it up with automating scripts, but why would anyone want to do that when you could be out in the wilderness photographing wildlife or beautiful landscapes? To create a usable RAID5 array with a stripe size of 256K you execute the following from the CLI:
Create the RAID Set
- rsf create drv=all name=RaidSet1
Create the Volume
- vsf create raid=1 level=5 name=Vol1 stripe=256
There are a plethora of other options depending on what you want to do, like add a hot spare or initiate an integrity check on the volume, configure email notification alerts, etc…, but again, everything can also be configured in the web interface if you don’t want to use the CLI.
After years of faithfully using a fully loaded MacBook Pro since 2012, a few months ago I upgraded my photo and video editing solution to an iMac Pro. The MacBook Pro is still going strong, but was showing limitations with processing 5.2K CinemaDNG RAW footage from my DJi Inspire 2 as well as exhibiting all-around sluggishness with the Adobe Creative Cloud suite. When B&H Photo put the iMac Pro on sale during the WPPI conference I knew it was time to pull the upgrade trigger.
While upgrading to the iMac Pro solved the performance limitations I was experiencing with my aging MacBook Pro I was still hitting a wall with my Direct Attached Storage (DAS) workspace. I had previously been using a variety of Thunderbolt 2 (20Gbps) RAID DAS devices to include the Akitio Thunder2 Duo Pro running two Samsung 2TB 850 Evo 2.5″ SATA III SSD in a RAID 0 configuration, the 10,000RPM WD 2TB My Book VelociRaptor Duo Thunderbolt Desktop Hard Drive in a RAID0 configuration, the WD My Book Thunderbolt Duo 6TB in a RAID 0 configuration and a few Seagate Backup Plus Thunderbolt 2 drives. Ever since its introduction I have been a huge proponent of Thunderbolt due to its superior performance when compared to slower interfaces such as USB/3 devices; Thunderbolt is the brand name of a hardware interface standard developed by Intel in collaboration with Apple. So now that I was using an iMac Pro with Thunderbolt 3 (40Gbps) I found myself in the situation of being limited by both the performance and array size of my previous DAS units. So along with the iMac Pro upgrade inevitably came the upgrade of my DAS solution as well.
My Approach To Redundant File Backup
My focus is definitely on performance when it comes to DAS systems, but I will clarify that my DAS unit(s) are not my backup solutions. A DAS for me is nothing more than a workspace to process and edit in and therefore I want performance and space. My backup solution started back in April of 2013 when I purchased my first Synology Network Attached Storage (NAS) DS1812+ and then upgraded it again with a Synology DiskStation DX513 5-Bay Expansion Unit a few years later and I still have one eSATA expansion port available for future upgrades. There are many nice things about the Synology units especially the user interface, but also the ability to configure Link Aggregation (LA) on the 1GbE ports for increased network performance. I’d highly recommend configuring LA for redundancy and performance increases, think of LA as similar to a RAID for your network ports. The beauty of using the Synology solution is to couple it with one of the integrated applications for offsite cloud backup and the beautiful Synology interface makes usability a pleasure. Since I’m an Amazon Prime user I use the Amazon Cloud for offsite backup and have been for years, ever since the Amazon Cloud Drive app was in beta on the Synology NAS and Amazon started offering unlimited storage for Prime users (this has since changed and Amazon pretty much angered its customers by changing its policies). The way my backup solution works is that I periodically Secure FTP (SFTP) anything I want to backup from my DAS workspace onto my NAS. My NAS is configured to sync anything I transfer to it up to the Amazon cloud. This solution gives me two local copies of what I’m working on as well as an offsite cloud backup in a semi-automated solution as I prefer to manually initiate uploads instead of automating an “rsync” process. My NAS solution currently consists of the Synology 8-bay with a 5-bay expansion unit attached for a total of 108TB configured as Synology Hybrid RAID-2 (SHR-2) which allows for the failure of up to 2 drives while also allowing for the use of mismatched drives sizes for an impressive amount of flexibility and redundancy. I also have the normal hard drive in a fire-safe lock box as well. I never thought I’d be in the situation in which my DAS (144 TB) is larger than my current NAS (108 TB), but it’s a good problem to have and one that’s easy to resolve by adding another eSATA 5-bay expansion to my Synology. It’s also worth noting that the better storage media you buy for your camera is the first step in the right direction to this process. I use the PCI based XQD cards which have reads of 440 MB/s and writes as high as 400 MB/s, while not cheap they offer longevity, performance, and resilience in adverse environments. Good storage starts in camera.
I hope this little article leaves you a little more informed with things you may not have previously considered as it pertains to Thunderbolt 3 RAID solutions, file systems, current storage market options, and price points for different performance levels. Storage methodologies are as different as people are, but what I’ve included here are the process and technologies that I employ to manage my photo and video editing requirements. As you can see I’ve attempted to “future-proof” myself as much as possible (which really isn’t possible) moving forward. As it is in technology, everything has a life-cycle so try and make yours as long as possible. I’m sure I’ll continue to tweak and optimize this setup moving forward, but I can promise I’ll be outside more than sitting behind a desk talking about technology.
Just came across this new to be released Thunderbolt 3 desktop enclosure. Looks interesting for another DAS option that has Thunderbolt 3 support, hardware RAID, etc… Here is the B&H Photo link and here is the mLogic manufacturer’s site.