I decided to finally split my File-server from my Hypervisor. Mainly because there were some issues with my the System not liking one of my RAID Controllers resulting in all kinds of funky error Messages.
I found some nice used Hardware at a good enough price but i still had to decide on the Software. After asking around in the forums and reading up on it a little, it was clear that at this point the only real option was something with zfs.But first lets look what my old Server looked like:
Intel Xeon E3 1235
Supermicro X9SCM-F
16GB ECC
Some old 300GB Disk as system disk
2x 1TB WD Green for my Windows Home Server VM
Supermicro USAS2-L8E
8x Samsung HD204UI 2TB (software RAID 6 with madm)
Highpoint Rocketraid 3520
8x Seagate Barracuda 11 1,5TB (hardware RAID 5 on the Highpoint)

Since the Highpoint Controller messed with VT-d i couldn’t just give the controllers through to the Fileserver KVM. Because of that i just used the LVM Volumes of both raids as virtio disks in KVM. This actually worked decently, but once in a while the Hardware Raid would have a problem with it.
And i had to use my Hypervisor to check the disk and RAID health, which was not the best solution.

All in all it was time to change that and a dedicated Fileserver seemed a like a good idea. But if i was spending money on it, i wanted quite a few features in scalability, comfort and error handeling:
– Web-interface for all the common tasks
– fast setup (i wanted to be able to get it running in under one day)
– needs to be able to manage large amounts of disk space (20TB+)
– needs to serve NFS and SMB/CIFS
– E-Mail alerts if anything goes bad
– needs to saturate Gigabit Ethernet with the Hardware i got for it
– needs to be reliable and protect against data corruption
– an easy and fast way to check disk health
– disk replacement should be no problem

In the end i opted for OpenIndiana with Napp-It for easy Web-management.
The install would have been done in under one hour normally.Unfortunately I discovered a problem with it though: the Gui of OpenIndiana doesn’t like the Supermicro IPMI KVM. After i figured that one out, everything else went smooth.

My new ZFS server consists out of the following Hardware:
Intel Xeon L3426
Supermicro X8SIE-F
12GB ECC
Seasonic X750 (from my old Server)
WD Scorpio Black 500GB(System)
6x Samsung HD204UI 2TB
2x Seagate ST2000DL004 HD20 2TB(relabeled Samsung HD204UI)

After i moved the 1st datapool off my old Server following Hardware will go into the File server:
Supermicro USAS2-L8E
8x Samsung HD204UI 2TB

The basic Steps are:
1. Boot OpenIndiana Live CD
2. When it is on the desktop start the installer.
3. When the installer is done boot to your new OpenIndiana and Login.
4. Download and start the Napp-it installer.

There is a good and easy tutorial for it on the official Homepage

The Web-interface is pretty much self-explanatory and my new Zpool with NFS and SMB share was running in under 5 Minutes.
After that i migrated my 1st data pool from my old Server to its new home, which took a while.
Once that was done i took the disks and controller from that pool and put it in the new Storage Server as well.
Expanding the Pool went without any issues, so i migrated the data off my 2nd data pool on the old Server.
Some when during the Transfer the System faulted one of my drives, that didn’t impress the transfer all that much and it completed without any problems.

So it turns out that one of the drives i have been using in my old Server wasn’t working properly. A failure of that drive wouldn’t have been all that bad even in the old Server since it was in a software RAID 6, but that fact that it didn’t fail might have led to data corruption and ext 4 isn’t equipped to detect that.
Quick SMART checks reported the drive as OK. But the self-tests failed it, so reading out just the SMART Status isn’t a very reliable way of detecting drive failure.

Update: the replacement drive arrived safely and is in the Server now. Everything is fine again and there a no more errors reported by zfs. Resilvering the drive took around 24 hours which is pretty OK for my zpool.

What i learned from all this: If you care about your data zfs is the way to go. This might change in the future when BTRFS gets to stable and i heard that ext 4 now has a checksum function as well.