After one of my hard drives failed in my 4 bay eSATA SAN, it sparked some thinking with the purchase of a new drive. I priced out a replacement and saw drives that were bigger and better for the same price. The four drives I'm using are outside their warranty, so it might not be much longer for the remaining drives.
I definitely don't want to purchase a replacement drive that's identical to what I have when I could upgrade to a better and bigger drive. With my 4 bay SAN, I think I'm limited on the drive size (it might be 3 TB) that currently has 2.5 TB drives (and even if I did buy a bigger drive and it worked, I would have unused space that I would never be able to reclaim until I replaced all the drives and started from scratch). I recently read that FreeNAS has the support to increase the size of the zpool / volume automatically by replacing individual drives with larger drives, so I'm going to take advantage of this (I already tested it in the lab, and it works great!). So, I definitely want to take this into consideration as I could easily expand my storage in 3 or 4 years (once my warranty expires and my hard drives start to fail) without having to build a new machine and having to migrate data.
So, the plan is to build a new machine (going to go with some of the hardware recommendations from some of the FreeNAS forums) with 4 WD Red 3 TB drives that will be in a RAID 10 configuration. This will hopefully give me optimal performance with the slow 5400 rpm drives. From my use of FreeNAS as a VM supporting all my VMs, I don't see the 1 GB link being saturated (it's mainly disk IO intensive) so I should be OK.
**** Hardware Purchased ****
I think the total price will come to around $1300. I'm hoping it'll last at least 5 years, so it'll be about $22/month for just over 5TB usable RAID 10 storage that's local and very versatile. And hopefully all I'll need to do is invest another $400 into the server for new HDDs and get another 5 years out of it.
- SuperMicro motherboard X10SLM-f (http://www.supermicro.com/products/motherboard/Xeon/C220/X10SLM-F.cfm)
- Intel Xenon E3-1220v3 Haswell 3.1 GHz CPU (http://ark.intel.com/products/75052/Intel-Xeon-Processor-E3-1220-v3-8M-Cache-3_10-GHz#@specifications)
- 4 8GB sticks of Crucial DDR3 SDRAM ECC unbuffered 1600 server memory (CT2KIT102472BA160B)
- 4 3TB WD Red NAS drives (6 GB/s 5400 RPM)
- Case - purchased a used one from a friend
- Power Supply - pending
- USB drive for install of FreeNAS - pending
**** Update - March 20, 2015 ****
Overall I would say that I'm satisfied with the new FreeNAS server. It's opened a lot of options, and I was able to learn a lot about FreeNAS. There were a few set backs along the way, but overall, things are looking good now.
The new FreeNAS server needed three network cables. One for the IPMI port (this is really nice to have), one that will be dedicated to NFS traffic hosting the VMs, and one for the remainder traffic. The interface supporting the VMs is using a cross over Ethernet cable to the ESXi box as I only have one ESXi host and I don't want to waste two ports on my already limited gigabit switch.
My first problems started when two of my new hard drives failed within FreeNAS. After two RMA's, I was back on track. I replicated my datasets from the VM FreeNAS to the new FreeNAS, re-added the new VMs to the inventory, and was onto using the new setup.
With the new setup, I've only run into issues just recently, but that came after the latest updates for ESXi 5.5 (I was on the patch from ~April 2014 and I patched it with the latest Feb 2015 patch). I started seeing issues where the datastores would become unavailable. From either the ESXi host or FreeNAS, I couldn't ping the other. From the ESXi logs, it showed the vmnic with a watchdog alert. After some searches, it appeared that the issue was related to the Realtec driver/card I was using. I updated the driver without any success. Then, I switched the interfaces on the vSwitches, moving the NFS traffic to the Broadcom NIC and the user traffic to the Realtec. I only saw the issue when the datastore was under high load, so I should be ok. Since I made the change, I haven't had any issues. I do not recommend using the Realtec cards in an ESXi install.
Overall, the performance is great for what I'm doing. However, I had to disable the NFS sync writes. When it was enabled (and it is highly recommended that it is), everything was very slow. If I had some extra money, I would probably invest in some SSDs for the ZIL. But, since I have good backups, I'm not too worried about it right now. I can now reboot several VMs at the same time without noticing any delay. And when starting a SpaceWalk verify scan, I saw esxtop show about 2200 IOPS for the RAID 10 array (I'm assuming most of this speed is from the read cache). And with the RAID 10 array, my writes a much faster than what they would be with RAID 5 or 6.
I also finished configuring a remote FreeNAS for off site replication (yeah!). I installed the PHPVirtualBox jail, attached a 5 TB USB NTFS formatted drive, granted the jail access to the USB drive, created a FreeNAS VM on the USB drive, and replicated all the datasets I configured. A friend is hosting it now on his Windows machine that is running VirtualBox. I created two volumes: the first uses a passphrase with encryption, and the other just has encryption. My data is on the volume with the passphrase, so every time the FreeNAS VM reboots, I have to connect and unlock. The second drive holds the OpenVPN configuration files. FreeNAS already had OpenVPN installed, but I had to replace the binary with a read-password-from-file enabled OpenVPN binary that was copied from a compiled version running on another FreeBSD jail. I then have a cron job to monitor the connection and restart the service if the connection goes down.
I'm hoping the above protects me from someone that tries to break into the FreeNAS OS during a reboot, as they still wouldn't be able to access my data. The worst case is that they would get my configs for the OpenVPN client, however, the client is limited on what it can see and I'll be monitoring the connection closely. To recover my data, I would only need to attach the USB drive to the FreeNAS server, setup VirtualBox, power on the VM, and then begin replicating data.
