Benchmarking high-capacity NAS drives

At some point I decided to build a NAS with a maximum capacity at minimum cost. After some research I found Seagate 16 TB HDDs that had the least price per TB.

So I bought:

• 4x used Ironwolf 16TB (ST16000NE000-2WX103) from a friend, with 2 to 17 thousand hours each;
• 2x new Exos 16 TB (ST16000NM000J-2TW103) from authorized reseller in EU;
• 1x used Exos 16 TB (ST16000NM001G-2KK103) from Aliexpress.

All the purchased disks have exactly the same usable capacity of 16,000,900,661,248 Bytes, same SATA v3 interface and rotation speed of 7200 RPM so there shouldn’t be major issues to create RAID array on them. File storage software will be described in another article, and now the goal is to test their performance.

For benchmarking I will use FIO, the same tool used for enterprise-grade storage systems. You can learn more about it in great article by Nikolay. All disks are connected to same controller on same computer and tested one at the time.

There would be 4 drives for test:

• Brand new Exos
• Ironwolf with 17000 h
• Ironwolf with 2000 h
• Exos from china

To compare key metrics among different drives I use a profile with multiple access patterns and block sizes:

Random read: 4K, 8K, 32K, 64K, 128K;
Random write: 4K, 8K, 32K, 64K, 128K;
Sequential read: 4K, 8K, 32K, 64K, 128K, 512K, 1M;
Sequential write: 4K, 8K, 32K, 64K, 128K, 512K, 1M;

Latency test – queue depth is 1 with 1 job.
Random read 4K;
Random rw 4K;
Random write 4K;

And some mixed workloads.
Random rw 70/30 4K;
Random rw 70/30 64K;

Full results with a lot of details are attached here and below is only a most interesting part.

For NAS systems storing large files, the most important thing is a sequential operations performance with a large block. Why the large block? Because it much faster while providing just a little capacity overhead on large files. For example, Synology uses 64K block in their systems and I will use 128K.

Here is IOPS performance overview. Values in table are 128K block based, test performed with queue depth of 32 in 4 jobs.

	New Exos	IW 17000	IW 2000	China Exos
seqread_128K, IOPS	2064	2061	2009	2148
seqread_128K, BW, MiB/s	258	258	251	269
seqwrite_128K, IOPS	2109	1905	1921	2593
seqwrite_128K, BW, MiB/s	264	238	240	324

I noticed that peak performance on these disks will be acheived on 512K block with bandwidth over 500MiB/s

Here are some pictures to visualize latency with randrw jobs:

The main point of performing this test was to see performance capabilities of these HDDs, and later compare with NAS disk groups performance.