Have you ever wanted to watch one of the shows that Netflix has available in another country?

I bet you have, especially if you live outside the United States of America. The solution to this issue is easy enough: you can use a proxy server or VPN service. But there are two issues with that approach:

• All your internet traffic is going through the VPN. If this can result in very notable delays when surfing your normal websites.
• You usually have no insight into what kind of logging you VPN provider does. So you really shouldn’t do any sensitive stuff over that connection.

Ideally you would want all of your normal surfing to go out through your normal internet connection and all the region specific stuff through a VPN or some other proxy.
And you can actually build something to do this with Squid. Squid is an Open Source proxy server.

A proxy Server sits between your browser and the websites you want to surf to. It accepts all your requests to surf to certain websites and processes them according to its configuration. Once it has determined that the request is valid, it will contact the web server for you and fetch the content you want. It will then forward it to your browser.
Since it sits in the middle of your traffic, it is the perfect place to redirect some traffic through another connection.

This diagram visualizes the difference between the two options for you:

And this is really just the start of your capabilities of Squid. While this tutorial will only show you a few basics of squid and how you can redirect some content over a VPN or another Squid server, there is so much more that can be done with Squid:

• Are you on a connection with a fairly low Volume available (like some mobile contracts)?
  No problem! Just crank up the caching in squid and repeated visits of the same website won’t be as demanding on your volume.
• Have kids that that visit bad websites?
  No problem! You can use squid to filter the internet by pretty much any criteria your want. And you can do it on a per user or computer basis if you need to.
• You hate ads on websites, but maintaining you Ad-Blockers across all devices is annoying?
  No worries! You can use squid as your Ad-Filter.

If you want to quick jump to the meaty parts here is an index:

1. Get started with Squid
2. Basic principles of Squid configuration
3. Write your personal Squid proxy configuration from scratch
   1. Need granular user access? Use Active Directory Authentication and Authorization.
   2. Are websites always loading to slow?
4. Two ways to avoid geoblocking with Squid
   1. How to mask your origin with the help of a Squid cache peer
   2. How to mask your origin by changing the TCP outgoing address
5. Why stop here?

Get started with Squid

This tutorial is based on Debian Linux, but Squid is available for many other operating systems and the Squid configuration will be largely the same on those.
The important thing is that you already have a computer or a virtual machine on which you can run Squid.

So let’s get started!

To install Squid on Debian Jessie, simply run the following command:

apt-get install squid3

Once Squid is installed, you need to configure Squid.  The default Squid configuration on Debian is very well documented.
This is great, because you can read up on a lot of things and pretty much any option in there explained in detail.

It’s also annoying because its extremely long and it’s hard to get an overview of the running configuration by just looking at the squid.conf.

In this tutorial you will learn to write you squid config from scratch, but is a good idea to keep the original configuration around for reference:

cp /etc/squid3(squid.conf /etc/squid3/squid.conf.bk
touch echo '' > /etc/squid3/squid.conf

Basic principles of Squid configuration

Before you start writing your own squid configuration, you might want to learn some of the basics. This will enable you to write your own policies for squid later on.

A comprehensive introduction can be found in the official Squid documentation. But if you just need an overview for now, read on.

In Squid you write rules with Access Control Lists(acl). There are two parts to acls:

• acl elements: define who or what is affected.
• access lists: define how an element is affected. They consist of an allow or deny and one or more acl element(s).

If you combine those two you create acl rules. For example to allow all traffic from your local network to access the internet you could write an src acl element and allow it to access the http_access access list:

acl local_network src 192.168.1.0/24
http_access allow local_network

The first line contains an element. Elements always follow this syntax:

1. acl: to let Squid know you are defining an acl element:
2. unique name: You can define this name freely.
3. acl type: The available types are predefined and dependent on the squid version you are using. A list of acl types can be found in the official documentation.
4. value list: a list of values that you want this acl element to match. The required syntax for a correct value depends on the acl type used. If you list more than one value here, the list will be evaluated with a logical OR (If one condition is met the entire element will be evaluated as true).

That’s it already. If you have a long list of values for one element you can also split them across multiple lines. Multiple definitions of an acl element are cumulative.

This means that this configuration:

acl local_network src 192.168.1.0/24 192.168.2.0/24

And this configuration:

acl local_network src 192.168.1.0/24 
acl local_network src 192.168.2.0/24

will create a local_network acl element with exactly the same behavior in the end.
It is however not permitted to combine different acl types in this way.
This will lead to an syntax error:

acl local_network src 192.168.1.0/24 
acl local_network dst 10.0.0.0/24

Back to the original example:

acl local_network src 192.168.1.0/24
http_access allow local_network

The second line the example is the actual access list. In this case the “http_access” access list. Access lists have the following Syntax:

1. Access List: The available access lists are predefined, but depend on the Squid version. A list of available access lists can be found in the official squid wiki.
2. allow / deny: You can either allow or deny the access to the resource represented by the access list.
3. acl element(s): You can list one or more acl element here that has to be matched by the traffic in order to activate this access list. If you list more than one acl element they are linked by a logical “AND”.
   This means all acl elements listed here have to match at the same time for the.

You can also list an access list multiple times in one configuration file. In that case the all access list statements will be check in order and once one matched the traffic will be allowed.
This means that you can use multiple occurrences in a similar fashion to a logical or. But you have to consider the order the statements are made.

A few examples will help explain:

http_access allow from_local_network from_remote_network
http_access deny all

This rule will match when traffic if both the listed acl elements are true. Since the name implies that both elements are src type elements, this configuration might stop any traffic through squid since traffic can’t have more than one source.

http_access allow from_local_network
http_access allow from_remote_network
http_access deny all

This rule would allow traffic through squid if either from_local_network or from_remote_network is evaluated as true.

http_access deny all
http_access allow from_local_network
http_access allow from_remote_network

This configuration would prohibit all traffic through squid since the rules are checked in order. While it is a good idea to have a deny all rule for http_acces in your configuration, make sure that it is the last http_access rule.

Write your personal Squid proxy configuration from scratch

Now that you know the basics, it is time that you get your hands dirty.

First you have to tell squid what interface and port it should listen on. I usually let it listen on the internal network adapter for my network:

http_port 192.168.1.10:3128

There are also some basic rules in the default configuration of Squid that you should employ:

acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
acl localnet src 192.168.80.0/24
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl Safe_ports port 901         # SWAT
acl CONNECT method CONNECT
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports

This configuration prevents users from connecting to unusual ports. Especially if this proxy is for more than you and your family, this will help you limit traffic to sane destinations.

Since you want to pretend to be in another country, you should disable the x-forwarded for header.
Otherwise Squid will always tell the servers you are contacting that it is acting as a proxy:

forwarded_for delete

Next you should probably tell Squid who is allowed to access the proxy. The simplest form of doing that is by using an src type acl element:

acl local_network src 192.168.1.0/24
http_access allow local_network

Need granular user access? Use Active Directory Authentication and Authorization.

If Network based authentication is not enough for you, you can use external authentication helpers with Squid. This part of the configuration might differ slightly based on the operating system and the way your system is joined to the Active Directory.

This configuration works on Debian Jessie joined to the AD through Samba.:

auth_param negotiate program /usr/lib/squid3/negotiate_kerberos_auth
auth_param negotiate children 10
auth_param negotiate keep_alive off

auth_param basic program /usr/bin/ntlm_auth --helper-protocol=squid-2.5-basic
auth_param basic children 10
auth_param basic realm please login to the squid server
auth_param basic credentialsttl 2 hours

As you can see there is no acl element configured yet. You can use following configuration to allow any authenticated user access to your proxy:

acl auth_users proxy_auth REQUIRED
http_access allow auth_users
http_access deny all

But since you have Active directory, you probably want to authorize access based on group membership. For that you will need to define external acls and use helper scripts or programs. But don’t worry everything you need ships with the default Squid install. Authorizing the Active Directory group proxy_users would look something like this:

external_acl_type testForGroup %LOGIN /usr/lib/squid3/ext_wbinfo_group_acl -K
acl trusted_users external testForGroup proxy_users
http_access allow trusted_users
http_access deny all 

Basically you have to create an external acl that calls the helper program with the correct parameters. Once that is done you can call this external acl as normal acl of the type external. In the acl statement you pass the name of the external acl and the value which it should check as parameters.

Are websites always loading to slow?

Squid is a very good web cache if you are on a high latency or limited Volume internet connection you might want to take advantage of squids capabilities in this regard.

Caching will speed up the loading of websites, because resources such as pictures or static websites will be delivered by your squid server itself instead of getting them again through the internet.

Depending on how large your cache is going to be it might increase the hardware requirements of squid quite a bit.

This is really just the absolute basic configuration that you need to get squid working as web cache. So if you have time do some research and make it even better:

cache_mem 512 MB
maximum_object_size_in_memory 1 MB
cache_dir aufs /var/spool/squid3 2048 16 256
maximum_object_size 10 MB

Let’s see what each option does:

• cache_mem: defines how large the RAM cache of squid should be.
• maximum_object_size_in_memory: defines how large an object can be to be cached in RAM. The default is 512KB
• cache_dir: used to configure a cache directory on the disk. The default is not to have a disk cache.
• maximum_object_size: defines the maximum size of any object in the disk cache. The default is 4MB

The cache_dir directive has the following syntax:

1. cache_dir: calls the directive
2. cache directory type: I chose aufs because it starts a separate thread for the cache access. If you would use ufs which does not start the separate thread, your disk i/o from the cache operation could block the main Squid thread.
3. /path/to/cache/directory: point it to where ever the cache should be saved on the disk. Make sure that the squid process has read and write access to that folder
4. options: in this case there are 3 options:
   
   • Size of the disk cache in mega bytes
   • number of sub folders on the 1st level of the cache directory. The default is 16.
   • number of sub folders on the 2nd level of the cache directory. The default is 256.

If you followed the tutorial so far you should have a Squid configuration that covers the basics. So it’s time to get to the important part.

Two ways to avoid geoblocking with Squid

You have options to route traffic selectively through other public interfaces with Squid:

• Use a Squid cache peer
• Use a different interface on the Squid host. This may also require you to set up policy based routing.

I personally use combination of both options. I have a squid server running in a virtual machine that is connected to my network via VPN. This is also the implementation I recommend to you.
The first option is easier to troubleshoot, since cache hierarchies are a fairly common and well documented use of Squid. While the VPN and source based routing of the 2nd option add another layer layer of complexity that you may not think of immediately when troubleshooting.
In addition you can use the Virtual Machine Squid is running on for other purposes as well.
But if you have an existing contract with a VPN provider, that you can not easily get out of, or if you need multiple destinations then the second option might be the one for you.

How to mask your origin with the help of a Squid cache peer

First you need a Server in the country your traffic should be routed to. Personally I use vultr (affiliate link). Their Virtual Servers are quite fast for the price. Another very reputable option is Digital Ocean. Both of these services have a pay as you go model and bill the server by the minute, so they are also great for just testing some stuff and then killing the machine again.
If you want a possibly cheaper option you can scour sites like lowendbox.com and lowendtalk.com for good deals. I recommend at least taking a KVM virtual machine as they are more flexible then their OpenVZ counterparts.

Once you have your virtual server running, install Squid on it. Then set up a minimal configuration for squid:

http_port 192.168.2.11:3128
forwarded_for delete
acl peer_host src 192.168.1.10/32
http_access allow peer_host

The interface used in the example is an internal interface that is only accessible via site to site VPN. You could also run Squid on the public interface, but I advise strongly against it.
If you do however run it on a public interface, you need to take more measures to secure Squid and you should encrypt the traffic between both squid servers.
Once you have the configuration in place you should start Squid on your remote server:

service squid3 restart

Once squid is running on the remote machine it’s time to turn your attention back to your local Squid server.
On your local server: Add the new Squid proxy as cache peer:

cache_peer 192.168.2.11 parent 3128 0 default

This is a basic version of the cache peer directive, but for our purpose it is enough. The following options are used:

1. cache_peer: calls the directive
2. remote host: IP address or fully qualified domain name of the host
3. relation: this is either parent or sibling. Squid will always try to send traffic through a parent host (as long as acls permit it). Siblings on the other hand are only checked if they already have a copy of the required object and if not squid will go through a parent or direct connection.
4. http port: the port you set the other squid to listen on
5. icp port: the Internet Cache Protocol is a way in which Dquid servers can request and exchange cache objects. It is necessary in more complex hierarchies, but for this setup it is not needed.
6. options: You can use this to influence failover and load balancing in larger caches. The default is just fine for this use case.

Since you don’t want to route all your traffic through the cache peer, you need to set up two acls to tell squid which traffic should go through which cache peer. For each domain your should set up a pair of acl directives just like this example:

acl domain_to_remote_proxy dstdomain .whatismyip.com
acl ref_to_remote_proxy referer_regex [^.]*\.whatismyip\.com.*
# redirect cbs
acl domain_to_remote_proxy dstdomain .cbs.com
acl ref_to_remote_proxy referer_regex [^.]*\.cbs\.com.*

The first acl entry is needed to reroute traffic that is going directly to the target domain. The second entry reroutes all traffic that has the target domain as its referer. This is needed because some content providers might host their actual content on different domains and in that case the referer would be the domain you want to reroute.
As you can see that each subsequent entry to the list has the same name again. This makes writing the actual access lists easier:

cache_peer_access 192.168.2.11 allow domain_to_remote_proxy
cache_peer_access 192.168.2.11 allow ref_to_remote_proxy
cache_peer_access deny all
never_direct allow domain_to_remote_proxy
never_direct allow ref_to_remote_proxy

The cache peer directives allow all traffic matching the acls to access the cache parents, but deny any other traffic access to it. In effect they reroute only the desired traffic.
The never_direct setting is a matter of taste. The setting in the example prevents Squid to directly connect to these domains. This means if the cache peer is unreachable squid won’t be able to connect to those domains.
I recommend that you leave those in at least while you are testing the system. This way it is at least be immediately apparent when something doesn’t work.

Now just restart squid and enjoy your unblocked content:

service squid3 restart

How to mask your origin by changing the TCP outgoing address

In order to make use of this you will have to connect either the squid host or your firewall to your remote VPN location.

If you connect the squid host directly to the VPN, then you will need to use policy based routing to route all traffic, that gets out through the VPN network adapter through the VPN. Without it the traffic you sent out on the VPN adapter via squid will still go out through your default gateway. This quick introduction to policy based routing should get you started.

If you terminate your VPN on your firewall, you can just give two network adapters in your network to squid. and then reroute all traffic from one adapter through the VPN  with firewall rules.

In either case the squid configuration will be the same. Once you set up your VPN connection in the most suitable way for you, you can start writing the acls that will be used to redirect some of your traffic.
The acl elements themselves are actually the same as in the example for using a squid cache peer. They will simply be applied to another access list.

# redirect whatismyip.com for testing only
acl domain_to_remote_proxy dstdomain .whatismyip.com
acl ref_to_remote_proxy referer_regex [^.]*\.whatismyip\.com.*
# redirect cbs
acl domain_to_remote_proxy dstdomain .cbs.com
acl ref_to_remote_proxy referer_regex [^.]*\.cbs\.com.*

The first acl entry is needed to reroute traffic that is going directly to the target domain. The second entry reroutes all traffic that has the target domain as its referer. This is needed because some content providers might host their actual content on different domains and in that case the referer would be the domain you want to reroute.
Now that you have the acls ready, you can use them on access lists:

tcp_outgoing_address 192.168.3.1 domain_to_remote_proxy
tcp_outgoing_address 192.168.3.1 ref_to_remote_proxy

This configuration sends all traffic that matches the acls through the interface with the IP 192.168.3.1. This IP should obviously be replaced by the IP of your VPN.
Now just restart squid and enjoy your unblocked content:

service squid3 restart

Why stop here?

Your squid server is running and you have accomplished your goal. If you get bored again or if you want to improve your experience while browsing you can look at other things that can be done with squid.

• Block Ads
• Protect your privacy. For example by stripping the referer from http requests or by randomizing the user agent header.
• Fine tune the caching to conserve bandwidth
• Make squid a transparent proxy for your network so that all http and https traffic goes through automatically
• Block content from employees or kids

Unless you are already using ZFS, Btrfs or ReFS you don’t. If your file server is a couple of years old, there is a very good chance that your Data is NOT OK.

Your server hardware might tell you everything is OK but that does not really tell you much because none of your monitoring systems check the actual data.

ZFS on the other had knows if your data is ok and can even repair it. I upgraded my file server recently. All the old disks that were still in the pool at the time were OK according to their SMART values. yet look what the monthly check told me: Just think about that: In just one month almost 2MB of data got corrupted. And right now you don’t know if your server has similar issues. If you did not know it yet this particular issue has a name: Bit Rot and it gets worse the older your current hardware is and more likely the larger your disks are.
You might not realise it but 2MB is a lot of data. It can be an important document or picture. Or it can be that it is just 2MB of a huge CAD file that is totally unreadable without those 2MB.
Granted my old server is quite a bit of an extreme example because I postponed the upgrade quite a bit longer than I originally wanted to. And the disks in it were not meant for this workload.

But I have seen quite a few that storage servers or NAS boxes remained almost untouched until a disk failed or they were full. So old hardware is not all that uncommon, especially at home and in the SMB market.

Now you might think setting up a new file server is easy right? You could just take Server 2012 with ReFS and Storage pools and be done with it.

Technically that is correct, but even in Windows there is really quite a bit more to consider. That being said I want to make the case for Open Source because for this purpose it is simply better than Windows.
In particular I want to show you how to build your file server with ZFS and Linux, two proven Open Source technologies. When you are done the Server will integrate nicely into your windows environment. Your users will not know the difference and you don’t have to get headaches from trying to license Windows correctly.

Update 2015-08-29: Updated configuration advise for ZFS datapools. Using the xattr = on setting resulted resulted in a delay in freeing space after a file was deleted.

If you are a returning visitor or are looking for something specific just jump ahead:
A short introduction to ZFS
About the Seagate Archive Disks
Get rid of Windows Once and for All: why it just doesn’t cut it compared to the alternatives
Warning: Your current Active Directory might not be prepared for Linux yet
How to prepare your old ZFS based Server for Migration
How to Install a fully encrypted Debian
Debian is installed what now?
How to Install ZFSonLinux in less than 5 minutes
How to Make your Linux Server talk to Windows: Samba4
How to make your Linux Server share files with other Linux Servers
How to Encrypt without loosing all the awesome ZFS features
How to Transfer Data between ZFS file systems
How to transfer your data from Windows to Linux
Warning: do a few basic checks before moving on
How to get rid of obsolete zpools, datasets and snapshots
How to Be Smart in a World of Dumb users
Can’t Keep up? Automate common maintenance tasks
Some final advice

Before we start, it might be good for you to know how my old server looked and how the new one looks. I hope this will help you understand the background of this post a little better and maybe even help you recognize some of those issues in your environment.

If you want to skip this part, then remember at least this: Old hard disks are dangerous.

My old file server was based on OpenIndiana (a Solaris with the Open Iluminous Kernel). It ran Napp-It on top of that for easy management from the browser. The server did what it was supposed to and it was very reliable. So you might believe that there was no need for a change.

Unfortunately the OpenIndiana project seems to be a little stalled at the moment and updates don’t happen at a pace that I like. And since the hardware replacement was going to cause downtime anyway, a look at the software couldn’t hurt.
The only hardware that was replaced during the upgrade, were most of the hard disks. All the disks are still powered by a Xeon L3426 with GB ECC RAM on a Supermicro X8SI6 Mainboard. The board has two additional rebranded LSI HBAs in addition to the one onboard. And all of that is housed in a Linan Li D8000 cube tower. That is one of the few non rack cases on the market, that can easily house over 20 disks.

In the old configuration the Server had several Zpools made up of the following disks:

1. Media pool:
   • 8x2TB raidz2 mostly Samsung 204ui
   • 8x2TB raidz2 mostly Samsung 204ui
   • 4x3TB raidz Seagate NAS drives
2. Data and Server Backups
   • 3×1.5TB raidz Seagate Barracuda Drives (The ones that were famous for dieing)
3. Temporary Storage used during ripping and such:
   • 1x2TB WD Red

That is quite a few of drives and most of them are quite old and not meant to run 24/7. While I planned this upgrade there were even a few drives that failed:

Replacing them would have been somewhat difficult since almost all my old storage pools were created in a way that optimises for 512 byte sectors and getting drives with real 512 byte sectors can be quite hard or expensive these days.
So I opted for not replacing the failed drives in the hopes that the pools would hold out long enough (don’t do this if you don’t have backups or can’t afford down time).

After the upgrade all disks in the server are encrypted. All the old 2TB disks except the WD Red are no longer in the server. They got replaced by a raidz3 of 11x 8TB Seagate Archive disks. The 1.5TB disks of the Backup pool got replaced by the 3TB NAS drives there were in the media pool previously. The three SAS/SATA HBA controllers and the onboard disk controller did not have enough SATA ports for the upgrade. Because of that I used an Intel RES240V expander temporarily during the upgrade. I also lacked enough SATA power connectors, but I did some downright unholy stuff with 1 to 4 port power adapters to take care of that.
Naturally the result was a bit of a mess during the upgrade:

But in the end the Server had a massively larger storage pool and a few free disk bays fore future expansion:

But the cable management could be improved:

A short introduction to ZFS

Before we go on you should know a little bit about ZFS to understand just how awesome it is.

ZFS is a file system that was originally developed by Sun for Solaris. Though calling ZFS a file system does not convey the full picture. ZFS is not only a file system it is disk management, software RAID, Filesystem and share management rolled into one. ZFS was built for reliability and scalability.
Some of you might protest that ZFS is new and that using something this new could impact the stability of your systems and the security of your data. But you are wrong. ZFShas been around for over a decade and in the it world that is certainly not new anymore. It is just not ancient like many of the commonly used file systems.

Not using ZFS because it is new is like using a horse carriage instead of a car for your daily commute because cars are new and your don’t trust them.

If you are not yet convinced about ZFS just has a look at its features:

• ZFS is a copy on write filesystem: You change a file and ZFS will write a copy of that file to your disk. After the copy is written without errors ZFS will delete the original file. This will enable you to recover from server crashes during the write process because a ZFS volume is never in an inconsistent state
• There are checksums for all the data and metadata this allows ZFS to detect (and in many cases repair) any errors in the stored data.
• Scrubbing is the process that detects and corrects data errors based on the checksums. Unlike fsck scrubs can be run while the filesystem is mounted and in use.
• ZFS allows you to build traditional RAID combinations and combinations that can handle even more failed Disks
• You can stripe data across multiple ZFS virtual raid devices
• ZFS uses the RAM heavily for caching and even supports SSD caching(though you probably shouldn’t use ssd caching at home)
• ZFS supports snapshots. Due to copy on Wirte the snapshots are easy and fast to create and don’t need a lot of overhead in terms of storage space. The snapshots are possibly THE killer feature in ZFS.
• ZFS supports deduplication out of the box. With that being said Dedup needs a lot of ram to be performant and you will likely not need it in home use
• ZFS is a 128 Bit Filesystem. That means it can handle pools so large, that the number would not have any meaning to you. Lets just say you really wont be able to hit that limit in the foreseeable future.
• You can easily replicate entire file systems to to different Storage Pools or even different ZFS hosts with built in tools
• ZFS supports ACLs such as those of Windows (the Linux version does only POSIX ACL’s but this will not be noticable in the front end)
• ZFS supports compression at very little performance loss

But as with all software ZFS is not perfect. Here are a couple of limitations and issues that you should be aware of:

• There are currently two versions of ZFS in the wild, the oracle one and the Open source one. The Oracle version is only used in Oracles Solaris. All other ZFS implementations (even the free iluminous Solaris Kernels) use the Open Source implementation of of ZFS
• In their current state these two implementations are not compatible to each other anymore, since they developed different feature sets by now
• Currently only Oracles Solaris implementation supports encryption from within ZFS
• Disk management is not quite as flexible as some other abstraction layers out there. You can not add or remove disks from the virtual devices. And you can only add new virtual devices to a pool. But you can’t remove them
• While usually to install under Linux it is not part of the Linux kernel, and may never be part of it due to it’s licensing. Though there are people working on that.
• You shouldn’t run ZFS on top of a RAID. ZFS performs best, if it can manage its disks by itself
• Deduplication is always done over the entire zpool. You can’t do deduplication over only a single dataset.
• Neither the raid like features of ZFS, nor Snapshots replace the necessity for backups
• ZFS does not automatically rebalance pools when you add new vdevs. It will however rebalance the pool over time by doing more writes to the new vdev until the storage use on the different vdevs is balanced.

Even with those limitations ZFS is still better than any other file system currently out there because all other file systems have bigger drawbacks or don’t even attempt to solve problems that ZFS solves.
In order to understand this article and to work with ZFS you should know a few terms and tools that are important when working with ZFS:

• vdev: Stands for virtual device. This is an abstraction layer to which you can add any kind of block device. In most cases this will be disks. If you add multiple block devices into a vdev you can use them in non striping raid configurations. These configuration options are mirror and raidz. With a mirror you can build vdevs that function like the RAID1 usually does, but you can also mirror over more than two devices. Raidz is based on parity calculations like RAID5 and RAID6. Depending on the level you choose, ZFS allows for 1-3 parity disks in a vdev
• zpool: Is the logical volume presented to the operating system if you will. A zpool can consist of one or more vdevs. Data will be striped across all possible vdevs, this means combined the the raid like capabilities of vdevs, you can build all the usually used RAID configuration with ZFS. While it is recommended, that ever vdev in a zpool is build the same way, this is not necessary. You can build a zpool with one vdev as a 3x2TB raidz and later extend it with a 6x4TB raidz2 vdev without a problem. But naturally there won’t be much striping in that particular case.
  One general rule of thumb is, the more vdevs in a pool, the better the performance.
• dataset: A dataset is the equivalent to filesystems and partitions in the non ZFS world. A dataset is dynamic and sees the available storage on a pool. Though you can give it quotas to limit its growth. You can even attach properties such as smb share or nfs share to the dataset. This will try to create these shares when the pool is mounted.
• scrub: A scrub is the filesystem integrity test of zfs. A scrub can be run while the filesystem is in use will check the integrity of all the data and metadata in a zpool with the help of checksums. It will try to repair the data as far as possible. Unlike fsck checks in traditional filesystems that only check the metadata, a scrub can take several hours or even days to complete on large pools because it checks all data stored in a pool.
• resilver: A resilver is the repair process in ZFS. A resilver will happen if a scrub finds damaged data in a pool, or if one of the disks in a vdevs is replaced.

If you are more of a visual thinker, have a look at this diagram, that shows how which element of ZFS is used:

Before you go out and build something like that be warned: even if ZFS lets you build something like the above, it is not a good idea.
You should be aware of a few best practices before you start:

• Do not mix disk sizes and disk speed in a vdev
• Do not mix vdevs with different mirroring/ parity schemes in a single zpool
• Do not mix vdev with different disk sizes or disk counts in a single zpool
• Use the entire disk for ZFS instead of just an partition
• Scrub regularily
• Build multiple smaller pools rather than one large one if your storage has different performance requirements.
• Use compression. LZ4 compression has little to no performance impact.
• Avoid using deduplication because it requires hughe amounts of RAM. If you absolutely need deduplication, build an extra pool for it because deduplication is a pool wide feature.
• Use ECC RAM. This one is geared more towards  home users. ZFS is geared towards protecting you data from corruption without ECC RAM it would not be possible to detect or correct data corruption that happens in memory.
• Try too keep at least 20-30% of the pool free otherwise the performace of the pool can degrade.
• Use an even number of data disks in raidz configurations and then add the parity disks.
• ZFS snapshots do not replace your other backups.

About the Seagate Archive Disks

There have been quite a few discussions and very mixed reviews about the new Seagate Archive disks. I ultimately decided to buy some because they just seem to be made for large media archives. Since they are based on fairly new technology (that is not used by any other drives on the market) I wanted to say a few words about them.
They are based on a technology called SMR or Shingled magnetic Recording changes how the tracks on a hard disk are aligned on the Platter. They are not aligned in a 90 degree angle towards the platter surface anymore but rather a much lower angle and they overlap each other. You can imagine this like shingles of a roof.
Because of that the tracks can be thinner without the risk of loosing data.

And just like you can’t just change one row of shingles on your roof, you can’t just overwrite one single track of data.
In these HDDs, that is because the write heads on the disks are larger than the read heads and they are thicker than a single track now.
This means if data on the disk has to be rewritten, all neighboring tracks have to be rewritten as well. To limit the effect the tracks on the disks are grouped.
This makes SMR disks really bad for write intensive applications like Databases or File Storage that changes a lot.
But the Archive disks are great for cold storage or write once read many types of use cases (like an Media Archive). Especially if you consider the price per Terra byte compared to other 24/7 capable storage.
They seem even more perfect for copy on write filesystems, since the original data doesn’t have to be rewritten.
Due to the size of those disks I opted not to do a complete burn in test of the disks by filling them once before using them. Instead I transferred my existing pool(roughly 40TB to write in raidz3) over to those disks and then rand a scrub.
This resulted in a couples of days of permanent write and read operations. If the drives survive such heavy use, they are not DOA.

So far my experience with the disks has been very good. No unexpected drops from the pool or any such thing. If you need lots of relatively cheap storage and the limitations of the SMR technology are not a problem for your use case, you should go for these drives.

Get rid of Windows Once and for All: why it just doesn’t cut it compared to the alternatives

There are quite a few options for operating systems for file servers. Three of those looked like a a possible fit:

1. Omni OS: Honestly this is a good option. But the better hardware support and the larger community make Linux the better platform for most people. If you have somebody already familiar with Solaris though, you might want to try this instead of a Linux. If you want to try Omni OS but you don’t have anyone familiar with Solaris give Napp-It a try. This way you will have to touch the command like only rarely.
   But this might cause you problems if you have to troubleshoot and you don’t know the operating system.
   Pros:
   • ZFS is native to Solaris. Thus there is a better integration in some areas.
   • There is a choice between using the native SMB Server or Samba
   • Windows ACL’s are very easy to handle here.  And it can handle actual windows ACL’s not just posix ACL’s.
   • Napp-It works and is optimized for this platform. So you could basically do the entire server management through the Webgui

   Cons:

   • Far fewer people are familiar with Solaris compared to Linux and Windows. Which means it is harder to find qualified employees for business and the community is smaller
   • The Kernel based SMB server only seemed to do the old ntlm authentication by default. This one is based on my Experience with OpenIndiana and hopefully that is not the case anymore.
   • The Kernel based SMB server does not support following symlinks between different zpools. Samba capable of doing that and available for Omni OS but it is not the preferred smb server there.
   • Group policies don’t work even if domain member
   • Hardware Support on Solaris is not great. This is especially a concern for Home users.
2. Windows 2012R2:
   With ReFS and Storage spaces, you actually have features similar to ZFS such as CRC checking and building pools with redundancy. But those features are not as mature as ZFS.
   And the performance of ReFS and storage pools is not great.On top of that the licensing adds another layer of complexity. And the cost can also be quite a bit high if you are a home user. You also have to consider that you will probably need Client Access Licenses and you have to read all the license documentation very carefully if you don’t want to get screwed over in the long run. If you think that it is not that bad, just listen to this story that one of the fellow admins told me at a Microsoft licensing seminar:
   Their company had been using Microsoft office since the 80s and always used upgraded the office License to the next version. About a year after they migrated their users to Office 97 their user base grew quite a bit and they bought new Office 97 licenses to effectively double their Office 97 license pool. In 2010 they got an MS license audit and MS declared half their Office Licenses as illegal.
   It turns out that the second batch of Office 97 licenses they bought had a slight modification in the License Aggreement that explicitly forbade them from being upgraded by later versions.If you are not in the mood to double check the license for stuff like this you should avoid buying Microsoft and other proprietary software whenever possible.Even if that scenario is not your concern then think about this: how much time do you spend comparing the different valid licensing scenarios for a MS product to figure out what the best deal is?
   In any larger installation this time can get quite ridiculous.
   Pros:
   • Fairly easy to use for somebody who is familiar with Windows/ It has a GUI
   • Native AD support including GPOs ( and from MS itself)
   • Plays well with windows only environments
   • All the newest domain features and secures authentication mechanism are available
   • Quite a lot support available in different communities and vendors.
   • Sysadmin are dime and a dozen.
   • you can find the most common problems and their solutions on Microsoft technet

   Cons:

   • Storage Spaces still had some performance problems
   • It is not open source
   • NFS on Windows was never Ideal
   • Bitlocker — it is encryption but it is closed source. It is only safe as long as you absolutely trust Microsoft
   • Licensing is quite expensive for a Home Lab ( though acceptable considering what I spent on hardware)
   • Licensing is extremely time consuming, complicated and annoying.
   • You may not be allowed to use all the features of the product you bought because it is forbidden somewhere in the depths of the license. (Think RDP: you are only allowed to use RDP to connect to a Windows Client machine over RDP if it has an enterprise license)
   • Napp-It does not work for windows (to be fair it doesn’t make much sense for it and I would probably be able to find different tools to display system health and basic maintenance in the browser)
3. Debian:
   This kind of stands in for all Linux distributions and I choose Debian here because it is rock stable. None of the downside are truely big. And the advantages of having it can’t be overlooked:
   Pros:
   • ZFSonLinux is easy to install and the use is pretty much the same as on Solaris. (Which means Solaris documentation for ZFS is in large parts relevant to Linux as well)
   • It is is not that hard to find sysadmins that are comfortable with Linux and there is a huge community behind it.
   • Debian is extremely stable.
   • Debian is easy to upgrade once a new major release comes out.
   • Napp-It is available for Linux( but quite limited in functionality)
   • SMB and NFS Server are not a problem
   • Samba supports symlinks in SMB shares
   • LUKS for encryption is well documented and has been in use for year. Many people are familiar with it, so getting support is fairly easy.
   • Samba allows for something more secure than plain ntlm authentication

   Cons:

   • LUKS (used for the disk encryption) is single threaded(though I Encrypt the underlying disks anyway so that is not much of a problem really)
   • ZFS is not native in the Linux Kernel
   • Samba requires some Setup and configuration to join the domain and set up some shares(though that has gotten less since samba4 was introduced)
   • group policies don’t work even if domain member

While I recommend Debian, you can also use any other Linux you are already comfortable with. The general concepts and many of the configurations that you can see in the post will stay the same. You will probably have to change some commands to fit your distribution though.

Warning: Your current Active Directory might not be prepared for Linux yet

Let’s take a break from the file server for a minute and talk about Active Directory.
As you know you can add Linux servers to an Active directory as members. But Linux handles users differently than windows.
When you join your Linux server to an AD, Samba maps Windows users to Linux uids and Windows groups to gids.
That is pretty neat, but if you have more than one Linux server it can happen that one user will have different uids on different servers.
This will be worse the more Linux servers are connected to your Active Directory because your nifty little management scrips will return inconsistent results between servers.

There is already a simple solution for this exact problem.
The rfc2307 specifies Active Directory schema extensions that extent user and group objects with properties that are relevant to Linux.
With the rfc2307 schema extension in place the Linux uid of your user will be stored in the Active Directory. This allows you to have consistent uids for your domain users across all Linux servers.
And on top of that Microsoft already provides management tools for those extensions that integrate into the normal Active Directory management tools.

Active Directory Domains created with newer Samba4 versions do have these extensions installed by default.
If you created your Active Directory with a beta version of Samba4 then you need to apply a schema modification to your Active Directory.
On the windows side you have to install the “Identity Management for UNIX” on one of your domain controllers if you are running Server 2008. Server 2012 seems to come with rf2307 extensions in the schema by default.
The following steps will show you how to extend the schema of an Samba4 Active Directory domain.
This will likely also work on Windows if you run the ldap command against a Windows Domain Controller, but you should use the Microsoft provided way on them unless you have a good reason not to.
Just follow these simple steps to upgrade your schema:

1. Log into your Samba Domain Controller
2. Find the correct path for sambas schema extension files. The Debian default is “/usr/share/samba/setup/”
3. The next step would be to manually edit the file and then take your DC offline to run the file against the sam.ldb of your DC.
   But if you are as lazy efficient as I am you will appreciate this much simpler solution( there is also no downtime with this one):

   sed -e 's/${DOMAINDN}/dc=example,dc=com/g' \
   -e 's/${NETBIOSNAME}/EXAMPLE/g' \
   -e 's/${NISDOMAIN}/example.com/g' \
   /usr/share/samba/setup/ypServ30.ldif | ldbmodify -H ldap://dc.example.com --option="dsdb:schema update allowed=true" -k yes -i

   Just replace the references to the example.com domain with the correct values for your domain and run it.

4. You should see the following output after the command ran:

   Modified 55 records successfully

Once you have an Active Directory schema that can handle the RFC2307 extensions, you still need a way to manage the uids and gids that your Active Directory users and groups will have under your Linux installs. As mentioned above, Microsoft supplies a tool for that. You just need to install it:

1. Search “features” in the windows start menu search of your admin computer and select “Turn Windows features on or off”
   
2. The management window “Turn Windows features on or off” will open
3. Navigate to “Remote Server Administration Tools” and extend the tree. Extend the Role Administration Tools. Open AD DS and AD LDS Tools. Extend the tree for AD DS Tools.
   Select “Server for NIS Tools” and click OK
4. Now you have an additional tab in you Active Directory users and computers management:
5. Select your domain as NIS domain if you want a user or group to have UNIX Attributes. This will fill out the important UID or GID fields. You can prevent a user from logging into your linux servers altogether by giving him /bin/false as shell. If you do that he will still be able to use the samba shares from this server, so this is probably a good setting for most of your users.

How to prepare your old ZFS based Server for Migration

Armed with all that knowledge it is time to get your hands dirty. If you are not planning to migrate old data over to your new server you can skip this chapter.

If you are coming from another ZFS based system, you should export all zpools that you want to migrate. While this is not strictly necessary, it can save you some headache later on.

The export is fairly easy. It is just a few clicks if you use Napp-it:

1. Login
2. Navigate to the Pool Management
3. Click the export link on any pool that y ou want to bring over to your new system.

If you don’t run napp-It, simply get on the command line and run the following command:

zpool export poolname

That’s it you are done with the preparation.

How to Install a fully encrypted Debian

You might ask yourself: why you should I encrypt a server it is in a locked room?

Let me ask you: Why not?

My current file server manages to give me almost full gigabit throughput and it does not have AES-NI to speed up encryption. Any remotely modern server has this these days and Encryption will have no impact on your performance.
Also Servers do get stolen even if they are in locked rooms. It happened to a customer of mine three separate times in five years in three separate offices. Just consider what kinds of confidential data could get into the wrong hands.
Even more so do you have process to safely decommission your server that includes securely wiping the disks?
Because I have seen old electronics being repurposed by disposal facilities.

Don’t risk it!

Just follow these easy steps to get an fully encrypted Debian system(or skip ahead if you know how to install Debian):

1. Boot from the Debian net install CD. You will be greeted by this screen:
   
   Select Install and press Enter.
2. Now you will go through the basic system and user configuration first select you language by using the arrow keys on this screen:
3. After you press Enter you have to select your time zone:
   Confirm your choice with Enter.
4. On the next screen you have to select your keyboard layout and press enter to continue:
5. Now you have to choose a host name for your server.  This is what your server is called on the network:
   If you don’t have a naming schema yet, just do what I do and name your server after it’s primary function.
6. After that you have to choose the domain name for the server:
   Enter the full name of the Active Directory domain the server will be in later on.
7. Next you will have to choose a password for your root user. This user is similar to the default administrator in windows and should get a really strong password:
   After you entered the password and pressed enter there will come another very similar screen to verify your password. Just enter it again and press enter.
8. Next you have to create a new user account for yourself. Try not to use any account names that you are currently using in your Active Directory:
9. As you might have guessed, you have to choose a password for this user:
   There will also be a password verification screen again.
10. The last part of the basic system configuration is choosing your time zone:
11. Now you get to the disk configuration. You should stick with the guided configuration unless you have specific needs, that are not covered by the guided configuration. To create a fully encrypted server, choose Guided –  use entire disk and set up encrypted LVM:
12. Select the disk you want to install Debian on. This screen shot only shows one disk available, your file server will likely have more disk in it and show them all. Be careful to select the correct disk:
13. Next you have to select the partitioning scheme. Just leave this on the default setting, all your data will be stored on the ZFS volumes that you will have to create later on:
14. Now you will have to confirm your choices. Make sure that you selected the correct disk and choose Yes:
15. This will start a process that fully erases the selected hard disk by overwriting it with random data:
    This can take a while!
    If you are impatient, you can cancel this process and the installation will continue normally. But your disk encryption might be easier to break in the end and an attacker could potentially determine how much encrypted data you have stored on the disk.
16. Now you have to choose a passphrase for your disk encryption. This should be a secure password of at least 20 characters. There is no reason to go absolutely nuts with password, because the disk encryption can be managed with LUKS later on. LUKS allows you to add new keys(passwords) to the disk and delete old ones.
    Once you have entered the password you will have to confirm it again.
17. After your set up your disk encryption, you will have to confirm the partitioning of the disks, because the actual operating system partitions will be written to the encrypted part of the disk:
18. Once you confirmed the partitioning, you will be asked if you really want to write those changes to the disk:
    Choose Yes.
19. Now that you are done with the disks, you will have to configure the package system and choose the software to install. This is actually fairly straight forward. First you have to choose the country you want to download Debian from. Chose your country or any nearby country, that allows for fast download speeds:
20. Next you will have to choose a so called mirror. This is a server where Debian and all its software packages will be downloaded from. You can choose any server in the list:
    And if it should turn out to be a slow one, you can still change it later.
21. Next your will be asked to configure your proxy:
    If you are in your homelab you probably won’t need one, just leave the line blank and select continue. If you are in a business and you do not know if you need a proxy, ask your network team. Though you probably are not supposed to set up servers in your business if you do not know if you need a proxy.
22. After this you have to decide whether you want to take part in the automated Debian package survey. This will make no difference to the function of your server. You should however consider all your IT policies in a business environment before making this choice:
23. Afterwards you will have to select the software that will be installed on your server:
    The only software packages that you need right now are SSH server and standard system utility.  Select both with the space key and deselect any other package before you continue.
24. After the software is installed you will be asked to install the boot loader:
    Select Yes.
25. The next dialog will require you to select the boot disk:
    This should be the disk you installed Debian on. There could be exceptions if you try to create a multi boot system for example. Though this does not make a whole lot of sense on a server.
26. Afterwards you will be greeted by the install complete message:
    Remove the Debian CD from you server and select continue to reboot.
27. On you first boot into Debian you will be greeted by this screen:
    Enter your disk encryption password to finish booting.

Debian is installed what now?

After the basic installation, your first order of business is implementing a (sane) way to unlock your disk remotely.
Otherwise you would have to be in front of the server to enter the password every time you reboot. The simplest way to get remote unlock capabilities is dropbear, a minimal ssh server that can start from your boot environment.
If you have remote KVM through IPMI or a VM you can unlock your disks remotely already, but SSH is just more convenient and least Super Micro’s IPMI Implementation is in Java and is not exactly ideal.

1. Install dropbear with apt-get

   apt-get install dropbear

2. Edit the configuration of your boot process to start networking and the dropbear service:

   nano /etc/initramfs-tools/initramfs.conf

   Towards the end of the file you will find the configuration options that you want to change/add:

   DEVICE=eth0
   IP=192.168.0.2::192.168.0.1:255.255.255.0:storage:eth0:off
   DROPBEAR=y
   

   Make sure you get the colons right in the IP line. Use the following format:

   server.ip.address::gateway.ip.address:subnetmask:hostname:interface:off

3. On Wheezy I was able to just use echo -n to enter my encryption key. Unfortunately this does not work anymore in Jessie.
   A quick google turned up an awesome script that covers this exact problem. It has worked quite well so far and there is no need to reinvent the wheel. Just download the script and follow the steps in this blog to set it up.
4. Once you have the script in place, you still need a way to authenticate to the server during boot. The best way to authenticate to ssh is with your ssh key. Follow this guide if you do not have one already. Simply drop your public key as authorized key into the boot environment:

   cp /path/to/your/key /etc/initramfs-tools/root/.ssh/authorized_keys

   While you are at it you can remove some default keys just in case:

   	rm /etc/initramfs-tools/root/.ssh/id_rsa
   	rm /etc/initramfs-tools/root/.ssh/id_rsa.pub
   

5. Now you need to update your boot environment, otherwise your changes won’t take effect on the next restart:

   update-initramfs -u -k all
   update-grub

6. Now reboot and try to unlock your System disk via SSH. It is important that this works.
   Don’t be lazy and test it now. I will wait.
7. By default the Debian installer sets your network to DHCP. If that does not work for you, or if you need to set up additional network interfaces, you can do so by editing the /etc/network/interfaces file.

   nano /etc/network/interfaces

   A basic network interface configuration looks like this:

   auto eth0
   iface eth0 inet static
           address 192.168.0.100
           netmask 255.255.255.0
           gateway 192.168.0.1

   If you need to change the DNS settings, you can do so in /etc/resolv.conf

   nano /etc/resolv.conf

   At the very least you have to enter the IP addresses of your DNS Servers. As part of an Active Directory domain, you should also set the domain and search setting to the DNS name of your domain:

   domain example.com
   search example.com
   nameserver 192.168.0.15
   nameserver 192.168.0.16

How to Install ZFSonLinux in less than 5 minutes

With the basic setup of the operating system behind you,
Your server is running now but it is not much of a file server yet. In fact it can’t even access ZFS file systems yet or serve files for that matter.
To change that you will have to install some software.
The first and possibly most important piece of to install is ZFS.
Fortunately the ZFSonLinux Project has a Debian repository and it supplies a Package that adds their repository to your system..
This cuts the entire installation of ZFS down to two steps(which I shamelessly copied from here their website) :

1. Install the ZFSonLinux repository:

   apt-get install lsb-release
   wget http://archive.zfsonlinux.org/debian/pool/main/z/zfsonlinux/zfsonlinux_6_all.deb
   dpkg -i zfsonlinux_6_all.deb

2. Install ZFSonLinux:

   apt-get update
   apt-get install debian-zfs

If you have old zpools to import do so now:

zpool import poolname

After this command your pool is mounted as “/poolname”.

And all zfs dataset can be found under “/poolname/datasetname”

How to Make your Linux Server talk to Windows: Samba4

By now you are the proud owner of a perfectly fine Linux Server with ZFS. Unfortunately the server can’t share files with your windows computers yet. For that you need Samba4.
On Debian there are two options to get samba4: You can use the package that is supplied by the Debian Maintainers or you can use the package from SerNet.
The SerNet package tends to be more current than the Debian one, but at the time of this writing they do not have any packages for Jessie on their website.
If you go with the default Debian packages and you are still on Debian Wheezy, use the Samba4 backport package. Because the Samba4 package supplied by Wheezy is still a beta version and there have been major improvements to Samba4 since that version was current.
On Jessie there is no more Samba3, so you can ahead and use the “Samba” package. These instructions are for Jessie so modify the package names as needed for other distributions.

1. Install Samba and the supporting software that you will need to join the domain and use Posix ACL’s on your shares:

   apt-get install samba openntpd winbind libnss-winbind acl

2. Edit the ntp settings to make sure that your servers time stays in sync with your domain:

   nano /etc/openntpd/ntpd.conf

   Comment out or delete the lines for the Debian pool servers and instead configure at least one of your domain controllers as the sync target by using this line:

   server dc.example.com

   Now restart the ntp service:

   /etc/init.d/openntpd restart

3. Now it is time to join the server to your Active Directory. This example configuration will allow you to use the your domain users to access file shares and you will be able to use Posix ACL’s on the file shares. From Windows Clients the ACL’s will look like native NTFS ACL’s and they can be changed with the native Windows tools..
   You will not be able to log into the server with your windows accounts however. Additional configuration is required for that and this configuration is not in the scope of this post.
   First you have to edit the Samba configuration file. Since this will differ quite a bit from the distribution defaults, I find it easier to back up the original config file and start with a blank one.

   cp /etc/samba/smb.conf /etc/samba/smb.conf.bk
   nano /etc/samba/smb.conf

4. Adapt this configuration template to your domain and you are good to go.:

   #======================= Global Settings =======================
   
   #section for the global settings
   [global]
   
   #you have to specify your NTDOMAIN Name as work group (the one you use when authenticating with DOMAIN\User
   workgroup = EXAMPLE
   #the security level of the domain. ADS is the correct value for an active directory
   security = ADS
   #The realm is the dns name of your domain
   realm = EXAMPLE.LOCAL
   #the keytab contains the kerberos key of the computer it will be created in this path when you join the computer to the AD
   dedicated keytab file = /etc/krb5.keytab
   #How samba should use kerberos. Again this is the correct value for an active directory
   kerberos method = secrets and keytab
   
   #The idmap and winbind configurations tell samba how it should map active directory users to your domain.
   #This setup requires your domain to have rfc2307 schema extensions. I will explain later how to get those in if you do not have them.
   #If you have more than one Linux server that might connect to your active directory at some, I strongly urge you to do this.
   #IF you only have this one Linux server I would still recommend to use this configuration, but if you don't want to do this or can't due to policies make these changes:
   #change idmap config EXAMPLE:backend = ad to idmap config EXAMPLE:backend = rid
   #remove these lines:
   #idmap config EXAMPLE:schema_mode = rfc2307
   #winbind nss info = rfc2307
   
   idmap config *:backend = tdb
   idmap config *:range = 2000-9999
   idmap config EXAMPLE:backend = ad
   idmap config EXAMPLE:schema_mode = rfc2307
   idmap config EXAMPLE:range = 10000-99999
   
   winbind nss info = rfc2307
   winbind trusted domains only = no
   winbind use default domain = yes
   winbind enum users = yes
   winbind enum groups = yes
   winbind refresh tickets = yes
   winbind expand groups = 4
   
   #this server is no domain controller and thus can't possibly have any master roles. So set it that way.
   domain master = no
   local master = no
   #this configuration sets your shares to use extended acls. This setting is necessary to use the ntfs security settings dialog in windows to configure the filesystems security settings.
   #If you look at these shares from a windows client, the ntfs security tab will behave like the ones on any windows share
   vfs objects = acl_xattr
   #NTFS acls usually have inheritance, so you should enable it here as well. ACL inheritance means that all files and folders will have the security settings from their parent folder, unless you change them manually.
   map acl inherit = Yes
   #If you access the share with windows clients, it is a good idea to allow their file attributes.
   store dos attributes = Yes
   
   #I copied the log settings from the debian default, as they are reasonable
   #### Debugging/Accounting ####
   
   # This tells Samba to use a separate log file for each machine
   # that connects
      log file = /var/log/samba/log.%m
   
   # Cap the size of the individual log files (in KiB).
      max log size = 1000
   
   # If you want Samba to only log through syslog then set the following
   # parameter to 'yes'.
   #   syslog only = no
   
   # We want Samba to log a minimum amount of information to syslog. Everything
   # should go to /var/log/samba/log.{smbd,nmbd} instead. If you want to log
   # through syslog you should set the following parameter to something higher.
      syslog = 0
   
   # Do something sensible when Samba crashes: mail the admin a backtrace
      panic action = /usr/share/samba/panic-action %d
   
   
   #===================== Share Definitions =======================
   
   #In this last part of the configuration you can put all the shares that you want to set up.
   #zfs can actually do this by itself, but I prefer setting the shares up manually.
   
   #every share is its on configuration section started by [sharename]
   [sharename1]
   comment = share that I imported from my old ZFS File Server
   #the path the shares root has in your linux system
   path = /OldPool/ZFSdataset
   #this controls wether the share can be seen if you browse to the computer with windows explorer
   browsable = yes
   #Most likely you will want to allow your users to write to the share
   read only = no
   #disallow guests. In an Active Directory #environment every user should have an #legitimate account.
   guest ok = no
   #the admin users will always have the right to #change the security settings of the file system #one the share. Best specify a group that should #have this right.
   #This is especially important if you are comming #from a solaris system or a Linux System where #you were not using rfc2307 user mapping as you #may have to rewrite the the shares security #settings in those cases.
   admin users = @"EXAMPLE\File_Admins"
   

5. Restart Samba and join the domain. Be sure to specify a user with the rights to join a computer to the domain:

   /etc/init.d/samba restart
   net ads join -U administrator
   

   The last command will ask for the users password and once you supply it the server will be part of the domain.

6. In order to map the users and groups correctly you have to edit the nsswitch.conf.

   nano /etc/nsswitch.conf

   Find the following two lines and add winbind to them (they should be close to the top of the file):

   passwd: compat winbind
   group:  compat winbind

7. You have to restart all involved services for the changes to take effect:

    /etc/init.d/samba restart
   /etc/init.d/winbind restart
   /etc/init.d/nmbd restart
   

8. Verfiy that everthing works as expected by running these commands:

    
   #should output domain users
   wbinfo -u
   #should output domain groups
   wbinfo -g
   #should output the uid and the groups of this users
   id domainusername
   

   If that works try accessing you share with the admin user and try changing security settings/creating files.
   Once all of that checks out, you are done with the Samba setup.

How to make your Linux Server share files with other Linux Servers

NFS, especially before version 4 has far less fine graned access controls than Samba. But it is great if you want to share space for other Linux Servers because the protocol has less overhead than smb. I use it as a backup target for my Hypervisor for instance.

1. Install nfs

   apt-get install nfs-kernel-server

2. configure your nfs share(s)

   nano /etc/exports

   add a share like this:

   /DataPool/Backup        192.168.85.254(rw,sync,fsid=0,crossmnt,no_subtree_check,no_root_squash)

   this configuration line is build in the following way:

   /Path/To/Share ip.of.the.client(options for this client)

   The options in the example have following meaning:

   • rw: Allow read and write access
   • async: reply to requests before the changes are commited to the storage. This can lead to corrupt files in the situation of an unclean shutdown, but offers better performance. You can use sync if you do not like this behavior
   • fsid=0: defines this export as the root of all exports. Fsid is normally used to give each filesystem a unique identifier
   • crossmnt: Allows the client to move into onther file systems mounted under this share.
   • no_subtree_check: Subtree checks improve the security a little by checking the parent file system of an nfs export for their access permission. However they can cause a bunch of problems for example they don’t handle it well if one file was opened and then renamed by another user while still open. In general they cause more problems then they solve
   • no_root_squash: Disables root squashing, which means that requests from uid 0 and gid 0 do net get rewritten to root. Most shares will not need this option, however this shares backs up OpenVZ guests and the permissions of all files should stay exactly as they are.
3. restart the NFS service:

   service nfs-kernel-server restart

How to Encrypt without loosing all the awesome ZFS features

You are here to learn how to build a fully encryptet file server, yet ZFS on Linux does not have a file system level encryption feature.
The only ZFS that currently has that feature is the (non open source) ZFS implementation from Oracle.
You are in luck however because Linux comes with its own disk encryption known as LUKS.
The only real limitation of LUKS is, that it is still single threaded. This is only a minor concern however, since you use luks to encrypt the drives from which you built the ZFS vdevs.
Every drive will have its own LUKS thread and thus effectivly making LUKS multi threaded.

Before you can actually encrypt your disks, you have to find out, what your disks are called. Simply run this command:

lsblk

And you will get an output like this:

This output shows all block devices in your system and what names they are mapped to. In the example you can already see the encrypted drives for every disk in the system.
If you are building a ZFS Server with more than 4 or 5 disks, consider using the same password for all disks in one vdev or zpool.
While this lowers the security somewhat because only one disk per pool has to be decrypted successfully before the attackers know the password for the entire vdev or pool.
This is not a realistic concern if you use strong enough encryption and passwords.

Alternately you could built a master-slave like setup for all disks in the pool. LUKS does allow for that, but in that particular configuration your LUKS master key is stored only on the master disk and if that disk crashes a restore of the pool will be a pain.
If you go that route make sure to back up you master key.

A small warning before you start:
Please ensure that you do not encrypt the wrong disk. If you encrypt a disk, the old data on it will be lost.

Once you have identified all the disks that you want to encrypt, run this command on every one of them:

cryptsetup -v --cipher aes-xts-plain64 --key-size 512 --use-random --verify-passphrase luksFormat /dev/sdx

Lets have a look at the different options of the command used above:

• -v:is used to get verbose output from the command.
• –cipher:sets the encryption cypher the aes-xts-plain64 is one of the currently recommended cyphers. Please note that in XTS mode only half of your key is used. Set your key size accordingly.
• –key-size:This sets the Key size. If you are using an xts cipher, set this to twice the value you actually want.
• –use-random: defines the random source for the master key. The use-random option will create a slightly stronger key, but if you do not have a lot of entropy sources on the sytem, it will take some time to create the key.
• –verify-passphrase: queries for the passphrase twice, this prevents you from creating a passphrase with a spelling error in it.
• luksFormat: is used to initialize the LUKS partition
• /dev/sdx: This is the target device of the LUKS operation

If you used the same encryption password for every disk in the zpool, you can decrypt your entire zpool with a script like this:

echo -n "Enter the Media Pool Password: "
read crypt_pw

echo -n $crypt_pw | cryptsetup open --type luks /dev/sdf sdf_crypt
echo -n $crypt_pw | cryptsetup open --type luks /dev/sdg sdg_crypt
echo -n $crypt_pw | cryptsetup open --type luks /dev/sdh sdh_crypt
echo -n $crypt_pw | cryptsetup open --type luks /dev/sdi sdi_crypt
echo -n $crypt_pw | cryptsetup open --type luks /dev/sdj sdj_crypt
echo -n $crypt_pw | cryptsetup open --type luks /dev/sdk sdk_crypt
echo -n $crypt_pw | cryptsetup open --type luks /dev/sdl sdl_crypt
echo -n $crypt_pw | cryptsetup open --type luks /dev/sdm sdm_crypt
echo -n $crypt_pw | cryptsetup open --type luks /dev/sdn sdn_crypt
echo -n $crypt_pw | cryptsetup open --type luks /dev/sdo sdo_crypt
echo -n $crypt_pw | cryptsetup open --type luks /dev/sdp sdp_crypt

After you have run the script or mapped your decrypted devices manually, you can use them to create your new zpool with a simple one liner like this one:

zpool create NewPool -o ashift=12 raidz3 /dev/mapper/sdf_crypt /dev/mapper/sdg_crypt /dev/mapper/sdh_crypt /dev/mapper/sdi_crypt /dev/mapper/sdj_crypt /dev/mapper/sdk_crypt /dev/mapper/sdl_crypt /dev/mapper/sdm_crypt /dev/mapper/sdn_crypt /dev/mapper/sdo_crypt /dev/mapper/sdp_crypt

This looks a bit confusing, but I promise once you know what is going on here it is really straight forward.
Let’s have a look at the different parts of the zpool create command and what you need them for:

• NewPool: is the name of the Pool. This name will also be used as mountpoint for the pool when it is imported by your system. So the pool MediaArchive will be available under /MediaArchive
• -o ashift=12: -o are the options that are used to create the pool. In this case the only option used is the ashift option to ensure that the filesystem will be aligned for 4k sectors. Please consider this carefully, the ashift setting of a vdev can’t be changed later on.
• raidz3: Specifies the level of parity in the vdev.Possible options are:
  • none: leave this parameter out and list only a single disk and this disk will be added as is to the pool
  • raidz: This is a raid mode with parity. In normal raidz one disk will be used for parity. This means you can loose one disk without loosing any data. Due to the way the data is written, the vdev will only have the write performance of its slowest member. There are also the raidz2 and radiz3 which work in the same way but have two and three disks for pairity data respectively.
  • mirror: This works like a raid one. Every disk in the mirror will have the same data. The write performance is that of a single disk, but reads will be performed accross all disks. Unlike RAID1 ZFS allows you to mirror up to three disks.
• /dev/mapper/sdf_crypt /dev/…This is a list of every disk in the vdev. Please use the mapped paths of the decrypted devices.

How to Transfer Data between ZFS file systems

Now you can transfer the filesystem from your old zfs pool to your new pool. ZFS has the tools to accomplish this right out of the box. You can even easily transfer entire file systems between different computers.
One thing that might not be immediately apparent about those built in tools is that they only transfer snapshops.
You can not use them to transfer the current state of a file system without making a snapshot first.
This might sound unnecessarily complicated at first, but it is not a limitation due to the way snapshots work in ZFS. Snapshots do not cost any additional disk space until a file actually changes.
And by transferring snapshots you know exactly which files have been transferred in which state because any changes during the transfer do not affect the snapshot.
And the transfer of any subsequent snapshots of that file system is incremental. That mean only changed or newly added files will have to be transferred to the new volume on subsequent runs of the transfer job.
Combined these properties make the move very painless for your, because you can keep the old share online and run zfs transfer jobs until both zpools or datasets are identical and then you can switch your users to the new share.

You can use this simple command to create a snapshot:

zfs snapshot OldPool/dataset@snapshotname

Now you can send the Snapshot to your new pool by piping “zfs send” and “zfs receive” together:

zfs send -R OldPool/dataset@snapshotname | zfs receive -Fdvu NewPool

If you data happens to reside on two different server with ZFS, simply run the send command on the source server and the receive command on the receiving server.

How to transfer your data from Windows to Linux

If you happen to migrate from Windows, you can’t really transfer your old data this way. But you can use robocopy to transfer your data to its new location incrementally. It is not as straight forward as a zfs send and zfs receive but it gets the job done.

robocopy source destination /ZB /E /COPYALL /V /LOG:userfilestransfer.log

Warning: do a few basic checks before moving on

Once you have transferred all the data, you should check your future Windows shares because the acl options may not be what you need for samba and posix ACL’s.
Run following command to see how the ACL options are currently set:

zfs get all | grep acl

This will display two properties for all your zpools and datasets:

• aclinherit: this property defines if and how inheritance of ACL’s is supported. Windows acl’s rely on inheritance, so set this to passthrough if it is not already set to that.
• acltype: this property defines if the pool supports of ACL’s. It will likely be set to off and for windows like ACLs to work on your share, you will have to set this to posixacl

You can use these commands to change the acl properties for every pool that doesn’t have the required options set:

zfs set acltype=posixacl NewPool/dataset
zfs set aclinherit=passthrough NewPool/dataset

Update

You should also set xattr=sa for every pool. Otherwise deletes won’t be processed entirely until you remount the pool. This means that all the space a deleted file takes up will still be marked as in use. It took me a while to notice, because I rarely delete anything on my storage server and the volumes are quite large.
But if you have a higher turnover in files, this might be a problem.

In order to set this setting simply enter folloing command for every zpool. The datasets on the pool will inherit this option unless your explicitly change that:

zfs set xattr=sa NewPool

Now you can switch your users over to using the new pool, by editing the path of the samba share in your smb.conf and the path of your nfs export.
While your users are working on the new pool you may wish to see if the transfer caused any data errors and repair them if they happend.
You can do this by scrubbing the zpool:

zpool scrub NewPool

The scrub can take quiet some time, especially on large pools but your users can continue working during the scrub.
If you want to know how far the scrub has gotten, you can open another ssh session and run the zpool status command.

zpool Status

The output of this command looks like this:

How to get rid of obsolete zpools, datasets and snapshots

Once the scrub is done, your data should be ok again if any bits flipped during the transfer. If you want to keep using some of the drives from your old zpool, you should destroy the old zpool before you reassign the drives. BE CERTAIN NOT TO ENTER THE WRONG POOL NAME HERE:

zpool destroy OldPool

This will destroy the entire pool.

If you only want to destroy datasets that you no longer use you can use this command:

zfs destroy -r DataPool/tempdata

If you have snapshots that are no longer needed but do not want to touch the current data of the dataset do this:

zfs destroy -r DataPool/tempdata@snapshotname

This will delete all files that are not in another snapshot or your current data.

After you have destroyed the old pool and reassigned any of the old disks as you see fit, you can take any disks that are not needed any more out of your server.

How to Be Smart in a World of Dumb users

Have I told you already that snapshots in ZFS are awesome?

Yes?

Here is another reason why:
zfs-auto-snapshot

This little Tool from the ZFSonLinux developers can be configured to take snapshots at certain intervals and to keep the last x snapshots.

This might not sound like much but how often has a user asked you to restore a file that he accidentally deleted or overwritten? Now you won’t have to look for the correct tapes anymore.

Snapshots will also help you mitigate the impact of one of the horror scenarios in recent time: a cryptlocker infection. With snapshots you can simply roll back to a snapshot from a point before the infection happend. You don’t have to spend hours restoring data anymore. (You still have to clean the infected computers though, sorry I can’t help you with that)

Now that you are convinced have a look at how it works: By default ZFS auto snapshot will create cronjobs in the following directories: cron.hourly, cron.daily. cron.monthly and cron.d
The the job in cron.d will run every 15 minutes.

By default the job creates snapshots for all your zpools where the option “sun.com-autosnapshot” is not set to false.
This option is not set by default. If you want to disable automatic snapshots for any of your pools or datasets use the following command:

zfs set com.sun:auto-snapshot=false PoolName

If you want to explicitly allow automatic snapshots for a dataset or zpool use this command:

zfs set com.sun:auto-snapshot=true PoolName/dataset

This setting will automatically be inherited to all child datasets under the point you set it. But you can change it for every child and it will not be overwritten.
I use this on my documents pool. It is great, if I mess up and accidentaly overwrite or delete something important I can recover a very recent copy of the document.
And due to the way ZFS snapshots work, you will have very little overhead from this feature on your avarage document pool. Though you might not want to run this on a pool that holds the virtual disks for your Virtual Machines.

If that alone was not great enough it gets better. With one little change in your Samba share configuration you can access those snapshots from Windows Previous version feature.
Just change the share like this in your samba configuration:

[users]
comment = Private Folders for users
path = /NewPool/users
browsable = yes
read only = no
guest ok = no
admin users = @"Example\File_Admins"
follow symlinks = yes
wide links = yes
vfs objects = shadow_copy2
shadow: snapdir = .zfs/snapshot
shadow: sort = desc
shadow: format = zfs-auto-snap_hourly-%Y-%m-%d-%H%M
shadow: localtime = no

This loads the hourly snapshots as windows previous versions. Unfortunately samba does not accept wildcards in the shadow format and you can only configure one shadow format.

Despite this little limitation is this feature extremely practical. And from Windows it is so simple to use that most of your users will be able to learn it. Have a look:

1. Right click in a folder or on a file that is snapshotted regularly and select preferences:
2. Click the Previous Version tab. You will see a list of all snapshots with the time they were taken:
   You can immediately restore it or you can browse the folder or view the file by double clicking the snapshot in the list or by clicking open.

If you did’t realise what I just said or you think you read it wrong: The restore is so simple that you can tell your users to restore their stuff on their own.
Just write a small guide on how to use Windows Previous Version and send it out whenever a user asks you to restore one of his files from backup and close the ticket.
As awesome as snapshots are, they can’t replace your other backups. They can only be another layer in your backup strategy. They will not protect you from sudden massive hardware failure, floods, fire or theft. And yes I have had customers who had these issues. Trust me you do not want to argue with firefighters to enter a burning building to get the tapes out because the dumbass responsible for the backup stored them next to the server.

Can’t Keep up? Automate common maintenance tasks

Remember the disk decryption script that I have shown you earlier?

Well now that you have a zpool on these disks, extend the script to also import your zpool for you.
Just add this line to the end of the script:

zpool import PoolName

One of the greatest threats to File Server is an unnoticed disk failure. And since we all know that regular manual checks just won’t happen in the long run, you should script the check.
I wrote this little script to query and parse the status of every single zpool in the server and then send a status report. The script runs every morning on my server.

#!/bin/bash
#This script queries the status of all zpools on the system and sends the output to the admin

pool_error=0
pools_and_errors=''
body=''
admin_email='admin [at] example [dot] com'
#get the status for the pools
for pool in $(zpool list | cut -d ' ' -f 1 | tail -n +2); do
	pool_state=$(zpool status $pool | grep 'state:' | cut -d " " -f 3)
	if [ $pool_state != ONLINE ]
	then
		#there are pool errors count and get the error
		pool_error=$(($pool_error + 1))
		pools_and_errors=${pools_and_errors}${pool}" has the status "${pool_state}"\n" 
	fi
done

#build the email
if [ $pool_error != 0 ]
then
	#echo $pool_error
	#echo -e $pools_and_errors
	subject='ERROR: '$(hostname -f)' has '${pool_error}' zpools with errors'
	body="These are the pools with problems:\n"${pools_and_errors}"\n\n"
else
	subject='zpool status for '$(hostname -f)': all pools are OK'
fi
body=${body}'Here is the detailed status for all zpool on '$(hostname -f)':\n'$(zpool status)

#send the email
echo $subject
echo -e "$body" | mutt -s "$subject" -- $admin_email

This script relies on a correctly configured email subsystem on the server. I usually use ssmtp on my non mail servers servers, as this is a very convenient way to redirect all system E-Mail to my mail server.
You can schedule this script as a regular cron job.

You are still missing one last piece to benefit from all the awesome of ZFS. By now you are aware of the importance of scrubs, but you still have to run them regularily.
Just schedule them with cron because running them by hand would be tedious and prone to be forgotten. Depending on the size of your pools and the amount of write activity the exact schedule can vary. But the general recommendations are, smaller pools that get a lot of writes, should be scrubbed once a week. And larger pools with fewer writes should be scrubbed once a month.
In order to plan a custom cronjob you can edit the crontab.

nano /etc/crontab

An entry for a scheduled scrub can look like this:

1 0 1 * * root zpool scrub NewPool

This example will scrub the zpool “NewPool” on 00:01 every first of every month. The Basic cron syntax is very simple:

It starts with a definition of the time the job is supposed to run. All time definitions can also have the following entered instead of one specific value: “*”. The asterisk stands for every time this unit is reached(every minute, every day and so on). Wheras a construct like this “*/15” specifies a regular interval.
You have to use 5 space sepparated values to indiacate the time, they have to be in this order:

1. Minute: the minute a job should run. You can use any number between 0 and 59 here.
2. Hour: the hour a job should run. Valid values are numbers between 0 and 23.
3. Day of Month: The day of the month a job should run. Valid values are 1 to 31 but you should be careful not every month has a 31st day
4. Month: The month on which a job should run. It is not at all surprising that the numbers 1 to 12 are valid values here.
5. Day of the Week: The Weekday jobs should run. Valid valus here are 0-7. There is 8 possible values here, to account for countries that start the week with a sunday and those that don’t. Sunday is assined to 0 and 7.

After the time you have to state the user as whom the command is run. In this case it is “root”

And after that you can enter the command that should be run. In this case it is “zpool scrub NewPool”

Some final advice

After you have read this, it might seem like protecting your data is a lot of work. But consider that this process is a one time thing and much of the necessary maintenance is automated.
If I still have not convinced you ask yourself this:

What would you do if this important customer documentation was not available in a year from now and they have a big project that relies on it?

What would I do if those baby pictures were lost 10 years from now?

What would I do if somebody stole my storage device and put the private information on it on the internet?

In the end it always comes down to two points:

1. If your data is important to you you have to Protect it.
2. You probably do not realize how important your data is to you until you lost it. This post happend because customer ignored warnings about backups and the dangers of RAID0.

I hope that you think about how important your data is to you and that you take measures to secure. Even if you end up choosing a different solution because it is a better fit for you.

And since we are somewhat on topic. Do not forget Backups. If you don’t have remote backups you don’t have backups. If you have not tried to restore your backups successfully, you don’t have Backups.

The Story of my mistakes during the upgrade of my  Proxmox guests to Debian 8.0 Jessie and what you can learn from them

Debian Jessie has been released a couple of days ago and naturally I wanted to check it out. Especially since some of my machines could really benefit from some of the newer software for Jessie. So I upgraded a few test machines and the first upgrade, a KVM Machine went pretty much as expected. There were only a few minor issues with configurations that are no longer supported.
After that I upgraded one of my OpenVZ test machines. The upgrade itself went as expected again, but after a reboot most services on the machine (including the network) would not start anymore.
After a bit of searching in the log files (which did not actually log anything after the update), I started looking to the most obvious cause: The change to systemd.

I was under the impression that the Proxmox OpenVZ Kernel supported systemd in OpenVZ Containers but it made the most sense for the symptoms. I spun up a new container, upgraded it to Jessie and installed sysvinit-core immediately after the update.
And everything worked as expected. Now I just had to deal with the Machine I upgraded earlier. I wanted to know if I could still get it to a working state. After a bit of messing around with it I managed to get it upgraded as well, but restoring a Backup and starting the upgrade again would have been quite a bit faster.

Now that everything was working I checked my Proxmox again since systemd should have been supported for my containers. It turns out, that while I did have the current Kernel 2.6.32-37 installed, I didn’t reboot the hypervisor in quite a while. Thus the Kernel that I had running was still 2.6.32-33.  I could have saved myself a couple hours of troubleshooting if I had just restarted my Hypervisor before running the upgrade to Jessie on my OpenVZ container.

So if you plan on Upgrading your Debian Containers to Jessie, do yourself a favor and ssh to your Proxmox host and check what kernel version it is running:

uname -r
#the Output should be a current kernel like this one
2.6.32-37-pve

If the output of this commands returns an older kernel than this, you should consider upgrading Proxmox to its current Version and then restarting. If that is not an option for you, then you should scroll down to my guide to Upgrade OpenVZ and install sysvinit-core.
If you already bricked a Machine you have no backup for have a look at the end of this article.

Follow these 6 steps to Upgrade your Debian Wheezy to Jessie

1. Update your current wheezy install to the current state:

   apt-get update
   apt-get upgrade
   
   

2. Check if there are any issues with your current install, that you need to take care of:

   dpkg --audit
   

   You want the output of this command to be empty, if not fix the issues listed there.

3. Change your sources list towards the Jessie repositories:

   nano /etc/apt/sources.list
   

   The file might look like this:

   deb http://ftp.us.debian.org/debian/ wheezy main contrib non-free
   deb http://security.debian.org/ wheezy/updates main contrib non-free
   

   You should replace Wheezy with Jessie in the lines of the official Debian repositories:

   deb http://ftp.us.debian.org/debian/ jessie main contrib non-free
   deb http://security.debian.org/ jessie/updates main contrib non-free
   

   If you have third party repositories, check if they are compatible with Jessie and what their path to the Jessie repository is.

4. Now it is time to upgrade to Jessie:

   apt-get update
   apt-get upgrade
   apt-get dist-upgrade
   

   The update will ask if you want to disallow the use of the password for root logins over ssh. If you don’t use root for ssh or if you already authenticate with your private ssh key, then I suggest you choose yes. If not choose no for now and look into generating your own ssh key.
   If you are Updating an OpenVZ machine with an old kernel, continue here to exchange systemd for sysvinit

5. After the upgrade make sure that all services are still working as intended and verify that everything installed correctly. Now reboot to finish the upgrade.

   dpkg --audit
   reboot
   

6. Once the reboot is done, make sure that all services started properly again. I had a few issues with mount cifs for example.
   I will address any software issues that I find in the next part of this article.

Quick fixes for software issues I encountered after the upgrade

Mount cifs does not mount my smb shares anymore

This issue was a simple compatibility issue with my old configuration in /etc/fstab

It took me a while to figure out though, since there were 2 parts to the issue and many of the fixes suggested on the net did not work.

This is a line out of my old configuration:

\\smbserver.example.com\share3   /mnt/share3  cifs    password=pwd,uid=1006,gid=119,username=shareuser      0       0

This is the line for the same share now:

//smbserver/share3   /mnt/share3  cifs    password=pwd,uid=1006,gid=119,username=shareuser,sec=ntlm        0       0

What changed?

• the new version of mount cifs does not consider ntlm the default authentication algorithm anymore. My Solaris file server did not like that.
• I can’t specify the path to the share as unc path anymore.

Proftpd not starting anymore

In proftpd I was using a module that does not seem to be in Jessie anymore. The module is mod_vroot_c

Just remove it from the proftpd configuration at:

nano /etc/proftpd/modules.conf

And proftpd will start again.

Did you encounter any issues while upgrading to Jessie? If so I would like to hear about it in the comments.

Squid3 not starting anymore

After the Upgrade of my proxy box Squid 3 refused to start anymore. I had two issues with my old configuration file:

• the log_access directive was obsoleted and I still had one line with it in my configuration. I simply removed it since this particular statement was not doing much. If you need to replace it, use: access_log
• The name names of the authentication helpers in “/usr/lib/squid3/” have changed. The ones that I was using, that caused issue were:
  1. kerb_auth this changed to: negotiate_kerberos_auth
  2. wbinfo_group.pl this changed to: ext_wbinfo_group_acl

Spam assassin fails to compile

After the Upgrade spamassassin failed to compile with following error:

Running sa-compile (may take a long time)
May 16 01:51:19.956 [10909] info: config: failed to parse line, skipping, in "/etc/spamassassin/local.cf": use_dcc 1
sa-compile: not compiling; 'spamassassin --lint' check failed!
dpkg: error processing package sa-compile (--configure):
subprocess installed post-installation script returned error exit status 2
Errors were encountered while processing:
sa-compile
E: Sub-process /usr/bin/dpkg returned an error code (1)

I simply commented out the use dcc line in the spamassassin config and told apt-get to fix the issue:

nano /etc/spamassassin/local.cf
#in the file look for this line and delete it or comment it out:
use dcc 1
apt-get -f install

You don’t like the newfangeled stuff? Get rid of it!

If you can’t Upgrade the Kernel of your Proxmox host for some reason, you can follow these steps to install sysvinit-core instead of systemd.
This also works for KVM hosts, if you happen to dislike systemd.

1. If you completed step 4, your Jessie OpenVZ guest will be running just fine. But if you reboot it, it will not come up cleanly due to issues with systemd and old OpenVZ kernels. One of the things that will not start properly is the networking. Which is why you should change to sysvinit before rebooting.  This is how you exchange systemd for sysvinit:

   apt-get install sysvinit-core
   reboot
   

2. Next remove the remnant of systemd:

   apt-get remove systemd libsystemd0
   Enter: Yes, do as I say!
   apt-get autoremove
   reboot
   

3. After the reboot continue with step 5 of the normal update process.

So you bricked your OpenVZ guest

If you followed the guide, you won’t need this section.  If you did reboot before installing sysvinit you can still get that machine back to a working state. But you will need access to the Proxmox host and its file system. It it is usually faster to restore from a backup. If you can’t do that for some reason, follow these steps:

1. Connect to your Proxmox host and enter the OpenVZ machine.

   vzctl enter machineid
   

2. Try removing systemd-sysv. This won’t work, but it will give you the download links for the packages you are missing:

   apt-get remove systemd-sysv
   Reading package lists... Done
   Building dependency tree
   Reading state information... Done
   The following extra packages will be installed:
     cgmanager libcgmanager0 libnih-dbus1 libnih1 systemd-shim sysvinit-core
   Suggested packages:
     pm-utils
   The following packages will be REMOVED:
     systemd-sysv
   The following NEW packages will be installed:
     cgmanager libcgmanager0 libnih-dbus1 libnih1 systemd-shim sysvinit-core
   0 upgraded, 6 newly installed, 1 to remove and 1 not upgraded.
   1 not fully installed or removed.
   Need to get 493 kB of archives.
   After this operation, 1074 kB of additional disk space will be used.
   Do you want to continue? [Y/n] Y
   Err http://ftp.debian.org/debian/ jessie/main libcgmanager0 i386 0.33-2+deb8u2
     Could not resolve 'ftp.debian.org'
   Err http://ftp.debian.org/debian/ jessie/main libnih1 i386 1.0.3-4.3
     Could not resolve 'ftp.debian.org'
   Err http://ftp.debian.org/debian/ jessie/main libnih-dbus1 i386 1.0.3-4.3
     Could not resolve 'ftp.debian.org'
   Err http://ftp.debian.org/debian/ jessie/main cgmanager i386 0.33-2+deb8u2
     Could not resolve 'ftp.debian.org'
   Err http://ftp.debian.org/debian/ jessie/main systemd-shim i386 9-1
     Could not resolve 'ftp.debian.org'
   Err http://ftp.debian.org/debian/ jessie/main sysvinit-core i386 2.88dsf-59
     Could not resolve 'ftp.debian.org'
   E: Failed to fetch http://ftp.debian.org/debian/pool/main/c/cgmanager/libcgmanager0_0.33-2+deb8u2_i386.deb  Could not resolve 'ftp.debian.org'
   
   E: Failed to fetch http://ftp.debian.org/debian/pool/main/libn/libnih/libnih1_1.0.3-4.3_i386.deb  Could not resolve 'ftp.debian.org'
   
   E: Failed to fetch http://ftp.debian.org/debian/pool/main/libn/libnih/libnih-dbus1_1.0.3-4.3_i386.deb  Could not resolve 'ftp.debian.org'
   
   E: Failed to fetch http://ftp.debian.org/debian/pool/main/c/cgmanager/cgmanager_0.33-2+deb8u2_i386.deb  Could not resolve 'ftp.debian.org'
   
   E: Failed to fetch http://ftp.debian.org/debian/pool/main/s/systemd-shim/systemd-shim_9-1_i386.deb  Could not resolve 'ftp.debian.org'
   
   E: Failed to fetch http://ftp.debian.org/debian/pool/main/s/sysvinit/sysvinit-core_2.88dsf-59_i386.deb  Could not resolve 'ftp.debian.org'
   
   

   Download every single one of the packages that is listed. In the output of apt-get

3. Transfer the packages to your OpenVZ container. The exact way to do this depends on the storage backend you are using.
   If you are using the default Proxmox Virtual Machine directory, you can sftp to your Proxmox host and drop the files to this path:

/var/lib/vz/private/machineid/tmp

This will put the files in the /tmp folder of the container.

• Change into the container again and install the packages by hand:

  dpkg -i /tmp/libcgmanager0_0.33-2+deb8u2_i386.deb
  dpkg -i /tmp/libnih1_1.0.3-4.3_i386.deb
  dpkg -i /tmp/libnih-dbus1_1.0.3-4.3_i386.deb
  dpkg -i /tmp/cgmanager_0.33-2+deb8u2_i386.deb
  dpkg -i /tmp/systemd-shim_9-1_i386.deb
  dpkg -i --ignore-depends=systemd-sysv /tmp/sysvinit-core_2.88dsf-59_i386.deb
  

  The cgmanager and systemd-shim packages will fail configuration. That is ok for now.

• Restart the machine. The reboot command won’t work, so do this instead:

   exit
   vzctl stop machineid
   vzctl start machineid
   vzctl enter machineid
  

• Fix the packages that failed configuration earlier:

  dpkg --configure --pending
  reboot
  

• Continue with Step 5 of the upgrade guide

E-Mail sucks! Your users just keep clicking the links in those damn phishing mails. And you can’t do anything about it. Hell somebody might be sending spam in your name and you have no idea about it. Let me blow your mind: You can solve these problems, for free. And I will introduce you the tools you need.

The cure to your E-Mail headaches hides behind three small acronyms:

• SPF: Sender Policy framework, tells others which mail servers are authorized to send E-Mail for your domain.
• DKIM: Domain Keys Identified Mail, uses encryption and DNS to verify an E-Mail sender and that it was not altered in transit
• DMARC: Domain based Message Authentication, Reporting and Conformance. Builds on SPF and DKIM and implements a policy and reporting system around them

The goal of this post is to get you to implement all three of these technologies. This is not an entirely selfless proposition since you will not only make your life easier, but every other mail admins life as well. To do this I will tell you which tools you need, how to implement them and how they work. By the end of this tutorial, a lot less Phishing mails will arrive in your users inboxes and you will have a deeper understanding of the involved technologies. I already hear you screaming: Wait these cookie cutter tutorials don’t work for me! My environment is different.

Well guess what? Mine is as well. In this post I will not just give you a bunch of configuration examples and send you on your way. I will give you a small overview of my environment as a reference and then actually explain what you are doing and why. Armed with this knowledge, adapting this tutorial to your own infrastructure will be a  breeze.

If you already have some of the systems in place feel free to jump to the point you are interested in.

1. Let’s get on the same page –overview of my mail environment
2. Wait you are not the mailman – Validate E-Mail Senders with SPF
3. Not everyone should be allowed to speak for you – Create your own SPF Record
4. Place a bouncer – Validate SPF within Postfixx
5. But does it actually work? – Confirming your successful SPF implementation
6. A Virtual signature for your company – Authenticate E-Mails with DKIM
7. Teach your mail server to sign E-Mail – Implement DKIM with Postfix
8. Validate your DKIM configuration
9. How do I know if a message should be signed? – Enhancing DKIM with ADSP
10. Enforce your rules with DMARC
11. Publish your rules – Create your DMARC DNS record
12. Know the law – Integrate opendmarc into Postfix on Debian
13. You are not perfect – Verify your work
14. Help out your fellow admins – DMARC reporting
15. Know your Spam – get the most out of Spamassassin
16. The finishing Touch – 6 Easy Steps to use Dovecot and Sieve to redirect spam to your users junk folder
17. Helpful Links and more Information

Let’s get on the same page –overview of my mail environment

I promised you a look at my mail environment. Let me warn you it is “Historically Grown”. I hear you groan, well you are probably right. But If I can get my system to stop spam and phishing, so can you!

A quick overview of the building blocks used for this:

• Mail server: Debian 7, Postfix, Dovecot , responsible for one mailbox domain and one virtual alias domain
• Antivirus and Spam check: Debian 7, Amavis-New, ClamAV, Spamassassin
• Webmail: Debian 7, Apache, Horde
• Database: Debian 7 MySQL
• Domain Controller: Debian 7, Samba 4(Backport).

As promised this is not exactly ideal, I especially lack some redundancy. But I only have a fairly small user base and regular backups. While downtime is not ideal, I can restore a  Backup that is less then 12 hours old in less then an hour so cut me some slack.

The other thing you might notice, is that every service has it’s own (virtual) machine. This is due to two things:

1. Linux Virtual containers with OpenVZ have close to 0 overhead
2. When I started out and ran a bunch of services on one box, I quickly learned that the failure of one service can effect others in unexpected ways.

Since I am kind of using two domains, this tutorial will also cover how to implement these technologies for different domains. Even though I have only one real domain and one virtual alias domain, the steps are the same if you have two mailbox domains, or twenty.

Wait you are not the mailman – Validate E-Mail Senders with SPF

The first cornerstone that will help you get rid of phishing is SPF. SPF is an acronym for Sender Policy Framework and is in the simplest terms a way to ensure, that a Server is authorized to send an E-Mail for a domain. You probably wouldn’t open the door to a guy in a ski mask with crowbar in hand only because he tells you has your latest Amazon order. In the online world SPF is a way to recognize the guy with ski mask.

This is achieved with a set of special DNS records for your domain. These records contain information about which servers are allowed to send E-Mail for your domain and how certain you are about that information. This diagram shows you how SPF works:

As shown in the diagram your server does not need any special program to get your E-Mail SPF verified. Your Domain simply needs to have a SPF DNS record. On the receiving side you will however need a little more work. You have to implement a program that can check the SPF records of the domain you are receiving E-Mail from. With postfix the 3 easiest options are:

• Spamassassin: You probably have this running already. But in my opinion you will want to fine tune the score for the SPF tests a little. Spamassassin helps you to mark failed SPF checks as spam, but not in a way that is useful for DMARC verification.
• postfix-policyd-spf-python: This is the tool I recommend for the job. This allows you to rewrite the header to include the results of the SPF check or even outright reject E-Mail that fails the SPF check.
• opendmarc: As the name suggest this is mainly a tool to implement DMARC. Opendmarcs capabilities include running SPF checks, but due to its limitations I still recommend using postfix-policyd-spf-python.

All of these tools test the E-Mail and write the results in some way before forwarding the Mail to the next part of the mail processing queue. I actually recommend using all three of those tools, but each one for its primary purpose. But before I get into that, I will show you how to set up your DNS, as this is done quickly and your changes will need a little while to propagate through all of the internet.

Not everyone should be allowed to speak for you – Create your own SPF Record

And setting a SPF record for your domain empowers you to tell the world which mail servers are allowed to send your E-Mails. Setting this record will take you less than five minutes and potentially saves everyone you are corresponding with a lot of headaches. Some of you might say: “Why would I care about that?”
Well that is easy. Because you are responsible! You are the mail admin and if unwanted E-Mail is sent in your name, warning everyone about the imposters is your duty.

Now that you are convinced that you need to set a SPF record, let me get all technical on you. A SPF record is aDNS  TXT record attached to your entire domain (or subdomain). This record starts with the SPF version that you are using. At the time of this writing there is only version one. The start of the record will look like this:

v=spf1

After that you will use so called mechanism to specify the hosts you want allow or disallow from sending E-Mail for your domain. But for those mechanism to be useful, you need to combine them with a qualifier. A mechanism can have one of four qualifies to determine what will happen with messages from the hosts that match the mechanism.These four qualifiers are:

• Pass: expressed by “+” this qualifier is assumed if no other qualifier is used. The Server(s) listed with a pass qualifier are allowed to send mail for your domain.
• Fail: expressed by a “-“ The servers listed here are NOT allowed to send mail for your domain. The plus usually also implies that mail should be rejected if SPF fails. I personally am not a friend of dropping or rejecting mail unless I am 110% certain that I do not want it, but unlike many other people I still recommend using the fail qualifier instead of soft fail. You should know which hosts can conceivably send E-Mail for your domain and you should disallow all other hosts from sending messages in your name.
• Softfail: expressed by “~” basically states that those hosts are probably not allowed to send mail for your domain but you aren’t certain. As a result mail servers should accept but tag mail from those servers. I believe this is ok for testing SPF records while reducing the impact of mistakes.This qualifier can also be used during server transfers. (set the new server to pass and the old server to soft fail for a little while)
• Neutral: expressed by “?” you are telling everyone, that you have no idea whether these servers are allowed to send mail for your server. Mail from those server will usually be accepted, but unless you are just testing new additions to your SPF record, setting this is not much use for anyone.

SPF offers plenty of mechanics to describe E-Mail Senders in its DNS record. I will only list the more useful ones, since you can cover pretty much any scenario with them:

• “all” you will only want to use this one with a “-“ qualifier in front of it, after you listed your servers. As the name suggests this mechanism encompasses all possible senders.
• ipv4” You can use this mechanism to allow Ipv4 hosts and networks to send E-Mail for your domain. The network has to be specified in CIDR Notation. An entry could look like this:

  ipv4:127.0.0.1
  ipv4:127.0.0.0/24

• “ipv6” same as above but for ipv6. an entry could look like this:

  ipv6:fc00::/7

• “a” you can use this to specify the a record of the allowed mail server(s). This can be combined with CIDR notation to allow the entire subnet to which the returned address belongs. Here are a couple of examples

  # use the a record of the current domain to determine valid sender IPs
  a
  # all IPs in the /24 subnet of this a record are valid senders 
  a/24 
  # the next two work in the same way as above but they use the a record of example.com
  # no matter what domain the SPF record is on 
  a:example.com 
  a:example.com/24

• “mx” this mechanism works like the “a” mechanism, except that it uses all mx records of the current or specified domain as valid senders. Adding a subnet is possible as well.
• “redirect” refers to the SPF record of another domain. If you include a domain, all hosts considered valid senders for that domain, will also be valid senders for your domain. All changes to their SPF will also affect your domain. You might need this if you are using outbound gateways or hosted E-Mail. Another use for this would be if you are using multiple Mail domains on your own server(s). With this mechanism in place you would only have to keep one SPF record updated. A redirect is specified like this:

  redirect=google.com

Now that you know all the pieces, let me show you a few examples of complete SPF records:

1. The public record for my main domain allows only one host, defined by an a record, to send mail. All other hosts are explicitly forbidden to send mail for my domain:

   v=spf1 a:mail.skelleton.net –all

2. This is actually the record I use in my local network. This record allows two different subnets to send E-Mail in my name:

   v=spf1 a:mail.skelleton.net/24 ip4:10.0.0.0/24 –all

3. And on top of that I have a virtual alias domain that is hosted on my mail server. I set it up to redirect to my main domain:

   v=spf1 redirect=skelleton.net

This information should cover most use cases. Look up the record syntax page on openspf.org if you need even more information.

Place a bouncer – Validate SPF within Postfix

Now that everyone knows your mail servers are the real deal, you might want to check the mail servers of all your incoming E-Mail. The tool of choice for this task is “postfix-policyd-spf-python”. You can use this policy service to just rewrite the headers of your incoming mail (useful for further processing) or you can outright reject E-Mails that fail the SPF checks.

Let’s start:

1. The first step is obviously the installation of the policy service. This is a simple one liner in Debian Wheezy

   apt-get install postfix-policyd-spf-python

2. Next you will have to edit the configuration file with the editor of your choice:

   nano /etc/postfix-policyd-spf-python/policyd-spf.conf

3. As I explained earlier, I only want to add the SPF check headers to my E-Mails and I do not want to reject any messages for SPF failures. To achieve this I had to change following two lines of my default configuration:

   HELO_reject = False
   Mail_From_reject = False

   I personally choose not to reject any mail based on the SPF results, because I have a tiny user base and receive only relatively few phishing mails. If you receive lots of phishing and your users are likely to click all the nice links they get, you should consider rejecting mail that fails SPF checks. But this might lead to missing a few E-Mails if a legit sender did not configure his SPF properly.

4. Next you need to add the policy service to your Postfix configuration. This requires two changes to your Postfix main.cf file. Open this file with you editor:

   nano /etc/postfix/main.cf

5. You need to add the following option to your main.cf:

   policy-spf_time_limit = 3600s

   This setting prevents your policy server from timing out during while E-Mail is being processed. In addition you will have to change your smtpd_recipient_restrictions setting. To be safe, you should add the check policy service at the end of your current recipient restrictions.

   smtpd_recipient_restrictions = permit_mynetworks,permit_sasl_authenticated,reject_unauth_destination,check_policy_service unix:private/policy-spf
   

   Whatever you do, make sure that check policy comes after reject_unauth_destination! Otherwise your server will be an open relay to anyone with a correct SPF record. You will probably also want the policy check after you list your allowed senders, this way you wont check the SPF records of your outgoing mail.

6. After those two changes you need to edit the master.cf configuration file of your Postfix installation:

   nano /etc/postfix/master.cf

   Once you have the file open add the following to the end:

   policy-spf  unix  -       n       n       -       -       spawn
        user=nobody argv=/usr/bin/policyd-spf

   This actually adds the policy-spf service that you configured earlier in the recipient restrictions.

7. Now all you need to do is restart Postfix, so that your configuration changes can take effect:

   /etc/init.d/postfix restart

But does it actually work? – Confirming your successful SPF implementation

After this SPF should work for your domain. You can check this by testing with an external mail account. Gmail is a good candidate for this since they also have DKIM and DMARC set up correctly.
First send an E-Mail from your own domain to an external mailbox you control and check the message source. The header should contain a field named “Received-SPF” and this should contain pass. If this field has a fail or softfail, you need to look at your SPF Record again. If this header field is not present at all, the DNS record may not have propagated everywhere yet. In this case you have to wait a little. Your provider may not add SPF results to the header of incoming mails. In that case use another mail provider for this.

In order to check if SPF is validated for your incoming mail, simply check the headers of a few E-Mails that have arrived since the change. The received SPF field has to be present in the header now.

A Virtual signature for your company – Authenticate E-Mails with DKIM

In essence DKIM is a way to sign E-Mails coming from your domain and telling the world how your signature should look like. It achieves that outcome by encrypting the message body and certain header fields with asymmetric encryption and then hashing the result. The process is slightly more complicated than SPF. But don’t worry, just have a look at the diagram to get an overview of the entire DKIM process.:

1. The Outgoing Email’s body and a few of the header fields are hashed and then encrypted with your domains private key . The content of this field could look like this:

    v=1; a=rsa-sha256; c=relaxed/simple; d=skelleton.net;
       s=mail; t=1425776684;
       bh=g3zLYH4xKxcPrHOD18z9YfpQcnk/GaJedfustWU5uGs=;
       h=Date:From:To:Subject:From;
       b=YK3uKQweSaxXF5h0SkoTeHvgvVuR3yjkkFT1XEGIbwoP9ht9PTo0+0qvtKB/QhhaU
        HCakgk2QC/eE9R5RS4wqF0G/jDxGAFU5ZM2ULrNoVGCBkHTw3IfBBPVBR24tFjXKV2
        pN5UPbMBMiyS3AdtujqTma4STHOxHyYVf+L8PSfg=
   

   The content of the header has the following elements:

   • v: DKIM version
   • a: the used encryption and hash algorithms
   • c: the canonicalization algorithms that should be used for header  and body
   • d: domain name that issued this signature.
   • s: selector that is necessary to find the public key. In this case you would have to query mail._domainkey.skelleton.net in order to get the public key.
   • t: Time the signature was created on
   • bh: encrypted hash of the message body
   • h: a list of signed header fields. Even if it is not listed here, the DKIM header field is always included in this list
   • b: encrypted signature data
2. The receiving mail server gets the mail and retrieves both the public key and the hash algorithm via DNS.
3. Then the server takes all the information that was signed and hashes them in the same way the sender did
4. These hashes are compared to the result of decrypting the signatures.
5. If the two match DKIM will pass, else it will fail. The result of this DKIM test is added to the header of the E-Mail in a new “Authentication-Results” field.
6. Processing of the E-Mail continues

All of this sounds complicated, but all the work will be done automatically by your mail server(s) once you finished the setup. You also don’t need to run you own CA for this, nor do you have to buy a certificate from one of the major Certificate Authorities. As with SPF, Spamassassin does DKIM checks by default. But they are again useless for DMARC tests. And since opendmarc is not capable of checking the DKIM validity of a message, you have to use opendkim if you want to be able to get your mail server DMARC compliant.

Teach your mail server to sign E-Mail – Implement DKIM with Postfix

Now that you are properly hyped about DKIM, I have even more good news: all the tools you need are available in the official Debian Wheezy repository. Those tools are called opendkim and opendkim-tools and I will walk you through their installation and configuration in easy to follow steps.

1. Installing opendkim is a one liner on the shell:

   apt-get install opendkim opendkim-tools

2. Next you will need to edit the default configuration of opendkim:

   nano /etc/opendkim.conf

3. I will not list the entire configuration file here, but rather only the lines you need to edit. With this configuration opendkim is capable of signing E-Mails for multiple domains. But it also works for single domain scenarios:

   KeyTable           /etc/opendkim/key_table
   SigningTable       /etc/opendkim/signing_table
   ExternalIgnoreList /etc/opendkim/trusted_hosts
   InternalHosts      /etc/opendkim/trusted_hosts
   AutoRestart             Yes
   AutoRestartRate         10/1h
   Mode                    sv
   PidFile                 /var/run/opendkim/opendkim.pid
   SignatureAlgorithm      rsa-sha256
   Canonicalization        relaxed/simple
   UserID                  opendkim:opendkim

4. Next you need to create the directories that will be used for the keys and the additional configuration files:

   mkdir /etc/opendkim
   mkdir /etc/opendkim/example.com
   mkdir /etc/opendkim/example2.com 
   

5. Now you need to create the list of trusted hosts:

   nano /etc/opendkim/trusted_hosts

   The contents of this file could look like this:

   #local host
   127.0.0.1
   # local subnets that are trusted and do not need to be verified
   10.0.0.0/24
   10.0.1.0/24

6. Now is the time to create the signing keys for your domain. The ‘–s’ option in opendkim-genkey specifies the selector or name of the key. I suggest using mail here, as that is the keys purpose. This step has to be repeated for every domain that you wish to sign E-Mail for

   cd /etc/opendkim/example.com
   opendkim-genkey -s mail -d example.com
   chown opendkim:opendkim mail.private

7. Now that you have the key pairs, you can create the key table file. This file tells opendkim, about all the domains you want to sign and where to find their keys:

   nano /etc/opendkim/key_table

   An entry in this file looks like this:

   mail._domainkey.example.com example.com:mail:/etc/opendkim/example.com/mail.private
   

   Every entry consists of four parts:

   1. mail._domainkeyexample.com is the KeyID. The KeyID is build as follows: selector._domainkey.domain.tld
   2. example.com: the domain this entry is for
   3. mail: is the selector
   4. /etc/opendkim/example.com/mail.private: the path to the private key

    

8. Next you have to create the signing table:

   nano /etc/opendkim/signing_table
   

   In this file you will have to make one entry per domain you want to sign. Every entry consists of the domain and the corresponding KeyID:

   example.com mail._domainkey.example.com
   

    

9. With this done, you have to ensure, that opendkim starts at boot time and has a socket. Simply edit the default file for opendkim to get this done:

   nano /etc/default/opendkim
   

   In Debian you simply need to uncomment following line:

   SOCKET="inet:12345@localhost"
   

   Using local Unix sockets requires slightly more work since the socket has to be readable by Postfix(Which starts in a chroot).

10. Now you have to change the configuration of postfix.

    nano /etc/postfix/main.cf
    

    Depending on your current setup you may already have milters running. In this case you will only need to add the opendkim milter to the list of active milters. Otherwise add this to the end of the file:

    milter_protocol = 6
    milter_default_action = accept
    smtpd_milters = inet:localhost:12345 
    non_smtpd_milters = inet:localhost:12345
    

     

11. If you are running a filtering solution like amavis-new, you should consider excluding the milter from running when it returns your E-mails to postfix. This will not change the success or validity of your DKIM tests, but it will reduce system load:

    nano /etc/postfix/master.cf
    

    Look for the configuration options of the port on which the mails are returned by amavis-new. This port will probably have a few options listed with the –o tag. Add a new line at the end of the existing options and make it look like this:

    127.0.0.1:25025 inet    n       -       -       -       -       smtpd
    [....]
    -o smtpd_milters=
    

     

12. Before you can take your configuration live, you need to update the DNS records for your domain. This is simple since the DNS record you need has already been created for you during key creation. You simply need to copy the contents of following file:

    /etc/opendkim/example.com/mail.txt
    

    And add them as txt record for the host “mail._domainkey.example.com” in your DNS.

13. Once you are reasonably certain that your new record has propagated, you should restart opendkim and Postfix, so that your changes take effect:

    /etc/init.d/opendkim restart
    /etc/init.d/postfix restart
    

Validate your DKIM configuration

The validation is similar to what you did for SPF, E-mail an external web mail account that you have access to and then write an E-Mail back. You will need a mailbox at an provider that implements DKIM like Google Gmail.

Once you have done that look into the headers of both e-mails.You should be able to find a header field called “Authentication-Results”. In this field you will find a string containing “dkim=pass”. If not, check you configuration again.

In the setup that I am describing you will ultimately find two authentication results headers field. One for DKIM and one for DMARC. Other implementations like the one G-Mail uses create only one field for all tests.

How do I know if a message should be signed? – Enhancing DKIM with ADSP

While DKIM is a way to determine if a signed E-Mail is valid. It does not contain any way to deal with E-Mails that are not DKIM signed. You can solve this problem by implementing a DKIM extension called ADSP or Author Domain Signing Practices. ADSP is a DNS record that tells other mail servers what to do with unsigned E-Mail. You can also achieve this with DMARC, but I urge you to implement both. Simply because DMARC is not anywhere near as widely implemented as DKIM. Since opendkim does already check ADSP on the receiving side, you only have to set a DNS record to implement it fully.

You need to create a txt record for following host:

_adsp._domainkey.example.com

This txt record contains only one tag:

dkim=all

The three possible values here are:

• unknown: both signed and unsigned mail is valid. If this is the case for your domain, you do not need to publish an adsp record as this is the default behavior in DKIM anyway
• all: All mail from this domain is supposed to be signed. If you can manage to get all your mail signed this is a good setting. But check on your newsletters and such before setting this.
• discardable: all mail from this domain is supposed to be signed and any mail that is not should be discarded. This is usually not needed. But if your organization deals with financial data or any other sensitive information do everyone a favor and set this.

Enforce your rules with DMARC

The last technology I would like to introduce you to is DMARC: Domain based Message Authentication, Reporting and Conformance. DMARC builds on both of the things that you just implemented in your mail environment. In the simplest terms DMARC allows you to create policies as to what should happen to an E-Mail if either one or both of the other checks fail. And it provides a standard for reporting that allows postmasters to be aware of phishing or spam in their name.

As the previous two technologies DMARC relies on a specially formatted DNS TXT record. I will help you to create this record for your domain and after that I will show you how to implement opendmarc in a step by step guide. This will allow you to check incoming mail for DMARC compliance and to report aggregated statistics back to domains who want them. Here is a rough overview of how the everything will work once you implemented DMARC:

Publish your rules – Create your DMARC DNS record

The DMARC record is a txt record for the host “_dmarc.example.com”. Your DMARC record tells the outside world how they should handle E-mail that is coming from your domain. In addition it tells everyone how you would like to receive DMARC reports for your domain.
With that being said, DMARC does not require other mail servers to follow all the reporting guidelines you set. In addition none of these settings have any effect on how your mail server processes incoming messages. Creating a valid DMARC record is somewhat complex, due to all the information that supposed to go into it. I will use my own DMARC record as an example to explain all the pieces of a valid DMARC record. If you don’t care for this much in depth knowledge, you can use this DMARC record generator.

v=DMARC1; p=quarantine; rua=mailto:dmarc [at] skelleton [dot] net; ruf=mailto:dmarc [at] skelleton [dot] net; fo=0; adkim=r; aspf=r; pct=100; rf=afrf; ri=86400

As you can see the record contains several tags that are separated by semicolons. I will explain those in the order they appear in:

1. v: DMARC version. This is a required field. For now it will have the value of ‘DMARC1’
2. p: DMARC policy. This is an required field as well, you can set it to one of three values:

• none:  Means don’t do anything if the DMARC verification fails. This is a good setting while you are still testing your DMARC implementation as it will not disrupt your outgoing mail if you made a mistake in your configuration.
• quarantine: Mail that fails DMARC checks should be treated as suspicious. This is good middle ground setting if you want to ensure that none of your mail gets lost.
• reject: Mail should be rejected If the DMARC verification fails. This is a good setting if your domain is used for phishing or if trust in your E-Mails is more important than occasional lost messages.
• rua: This is an optional parameter and contains the address to which the DMARC aggregate report should be submitted. You can specify web addresses here, but I have not done so so far. If you do not set this option at all, you will not receive DMARC reports.
• ruf: This is similar to the rua field, but these are the addresses for DMARC forensic reports. These reports contain detailed information about failed DMARC verifications of E-mail claiming to be from your domain.
• fo: These are reporting options for the failure reports. This can have four possible values:
  • 0: generate a report if both SPF and DKIM tests failed
  • 1: generate a report if either the SPF or the DKIM test failed
  • s: generate a report if the SPF test failed
  • d: generate a report if the DKIM test failed
• adkim: This option is optional and controls how strict the result of the DKIM verification should be interpreted. It defaults to relaxed if it is not present. Possible values are:
  • s: strict
  • r: relaxed
• aspf: This option is optional and controls how strict the result of the SPF check should be interpreted. It defaults to relaxed if no value is set. Possible values are
  • s: strict
  • r: relaxed
• pct: This is also optional and determines how many percent of the messages from your domain should have the DMARC verification done by other mail providers. The possible values here are integers between 0 and 100. It defaults to 100 if it is not set.
• rf: This optional field lets you specify your preferred reporting format for forensic reports. It defaults to “afrf”. These are the possible values:
  1. afrf: Authentication Failure Reporting Format
  2. aodef: Accident Object Description Exchange Format
• ri: This optional field is the interval at which you want to receive DMARC reports in seconds. It defaults  86400 seconds (one day). According to the DMARC specification every participating organization should be able to send reports at least once every day. Intervals as small as one hour are within the specification. But those smaller intervals are generally served on a best effort basis.
• sp: The last field is also optional (and not present in my example). It is the subdomain policy. If you do not set this, the policy you set in the beginning will apply to your subdomains. If you set this you can use the same values as in the policy field. This can be useful if you know, that you never send mails from one of your subdomains. In that case you can set the subdomain policy to reject without any risk of your legitimate E-Mails being discarded.

Know the law – Integrate opendmarc into Postfix on Debian

Now that your DMARC record is set, you need to integrate some kind of DMARC verification into your Postfix server. I choose the milter opendmarc for this task. Unfortunately it is not in the Debian Wheezy default repository. But you can use Wheezy-Backports to get an installation package. Which makes it easy to upgrade later on. I will walk you through the installation step by step:

1. Since opendmarc is currently only available as a backport, you will have to add the Debian backports repository. To do that you need to edit your sources.list file:

   nano /etc/apt/sources.list
   

   Add following line to the file:

   deb http://ftp.debian.org/debian wheezy-backports main contrib
   

    

2. Now you need to update the list of available packages and then you can install opendmarc using the backports repository:

   aptitude update
   aptitude -t wheezy-backports install opendmarc
   

3. Once the installation is done, you need to edit a few things in the opendmarc configuration file:

   nano /etc/opendmarc.conf
   

   You need to add, change or uncomment the following lines in this file:

   AuthservID mail.example.com
   PidFile /var/run/opendmarc.pid #Debian default
   RejectFailures false
   Syslog true
   TrustedAuthservIDs mail.example.com,mail2.example.com
   UMask 0002
   UserID opendmarc:opendmarc
   IgnoreHosts /etc/opendmarc/ignore.hosts
   HistoryFile /var/run/opendmarc/opendmarc.dat
   #for testing:
   SoftwareHeader true
   

   Let me explain those options to you, so that you can change this to fit your needs:

   • AuthservID: Sets what is used as AuthservID when processing E-Mails. This should be the hostname of your mail server or another unique string
   • PidFile: Path to the PID file
   • RejectFailures: This is a Boolean, if this is true E-Mails that fail DMARC verification will be rejected by your mail server. I prefer simply tagging the mail so I set this to false.
   • Syslog: true or false. Tells opendmarc, whether it should log to syslog or not
   • TrustedAuthservIDs: these AuthservIDs are assumed to be valid inputs for DMARC assessment. This can prevent the DMARC tests from running several times if you have multiple mail servers in your organization
   • UMask: the PID file and the socket file are created with this umask
   • UserID: the user and group running the opendmarc service separated by a colon.
   • IgnoreHosts: The path to the Ignored Hosts list
   • HistoryFile: The path under which the History file should be created. This file is necessary if you want to be able to create aggregate reports to send out to other organizations
   • SoftwareHeader: adds a “Dmarc-Filter” header with the opendmarc version in every processed mail. This is good to have during testing
   • ForensicReporting options seem to be broken in the version of opendmarc that I used. When I tried to uncomment them, opendmarc would not start because of unrecognized parameters.

    

4. Now you need to create the Ignore hosts file that you specified in the configuration:

   mkdir /etc/opendmarc/
   nano /etc/opendmarc/ignore.hosts

   This file should contain a list of networks and hosts that you trust. Their mail will not be checked by opendmarc. Put one host or subnet per line into the file:

   localhost
   10.0.0.0/24

    

5. With this part of the configuration complete, you just need to make a little change to the default file:

   nano /etc/default/opendmarc

   Put following line in the file:

   SOCKET="inet:54321@localhost"

    

6. Now you can start opendmarc

   /etc/init.d/opendmarc start

7. Next you have to add opendmarc to the milers in postfix. To do that edit your main.cf configuration file:

   nano /etc/postfix/main.cf
   

   Find the milters you set up earlier for DKIM and add the opendmarc milter after DKIM(the order is important, as opendmarc needs the DKIM check results):

   smtpd_milters=inet:localhost:12345,inet:localhost:54321
   non_smtpd_milters=inet:localhost:12345,inet:localhost:54321
   

    

8. Finally you need to reload your postfix configuration:

   /etc/init.d/postfix/reload
   

Now you have a mail server that performs DMARC verification on incoming mail.

You are not perfect – Verify your work

The easiest way to verify your setup is the same as before. Use an external mailbox write an E-Mail to it and then write one to you from that mailbox. Check the headers for the Authentication-Results field, that contains dmarc=pass. Remember that you should find two “Authentication-Results” headers if you followed this guide.

If you are in Germany and you are using Gmail for your verification, make sure that you write from a @gmail.com address. Google’s German mail domain googlemail.com does not seem to have DMARC set up. This took me an embarrassing amount of time to figure out since everyone said that Google has DMARC implemented.

Help out your fellow admins – DMARC reporting

While your server now performs DMARC checks and has a DMARC record set, it is not DMARC compliant yet. For that you are missing reporting capabilities. I will show you how to send aggregate DMARC reports to get you compliant with the reporting part of DMARC. Trying to send forensic reports is a bad idea for two reasons:

1. There are privacy concerns with forensic reporting if your users subscribe to mailing lists. You can read a little more on this here.
2. As mentioned above the configuration options for forensic reporting were not recognized in the version of opendmarc that I used. So this would require spending on a lot of time on something that is potentially a huge risk to the privacy of your users.

You will need access to a working MySQL database to follow my step by step instructions. Setting up a MySQL server is not in the scope of this tutorial:

1. Copy the DMARC database schema SQL script to your database server(if that is not the same as your mail server). You can find the SQL script under following path:

   /usr/share/doc/opendmarc/schema.mysql
   
   

2. Edit the script to fit your needs. The default is mostly fine. But if you do not wish to create your database users by hand, you should uncomment and edit these two lines in the script (to uncomment them remove the leading –):

   -- CREATE USER 'opendmarc'@'localhost' IDENTIFIED BY 'changeme';
   -- GRANT ALL ON opendmarc.* to 'opendmarc'@'localhost';
   
   

   With this part you create a database user and allow him access to the database. The following parts of this statement may need to be modified:

   • ‘opendmarc’@: This is the username, you can choose whatever you like as username. But make sure that you put it into quotes. (The @ is not part of the username, but simply here to differentiate the username from the database name)
   • ‘localhost’: this the host from which the database user is allowed to connect. If the MySQL database runs on your mail server, you can leave localhost here. If your MySQL database runs on a different server, put the IP of your mail server here.
   • ‘changeme’: This is the user password. I suggest a nice strong randomly generated password here.
   • opendmarc.*: This is the database name. if you did not change anything in the upper part of the script, leave this alone.
3. connect to the database with an account with sufficient privileges to create a new database and run the script. One of the simpler ways of doing this would be the following commands on the database server:

   cd /path/to/schema.mysql/
   mysql -u root -p < schema.sql

   You can replace the root user with any user, that has the right to create new databases and users. The command will prompt you for the user password and then execute the script.

4. Once the database exists go back to your mail server and create a new script to read the history file into the database and send out the reports.

   /etc/opendmarc/report_script
   

   I used following script for this:

   #!/bin/bash
   
   DB_SERVER='database.example.com'
   DB_USER='opendmarc'
   DB_PASS='password
   DB_NAME='opendmarc'
   WORK_DIR='/var/run/opendmarc'
   REPORT_EMAIL='dmarc [at] example [dot] com'
   REPORT_ORG=example.com'
   
   mv ${WORK_DIR}/opendmarc.dat ${WORK_DIR}/opendmarc_import.dat -f
   cat /dev/null > ${WORK_DIR}/opendmarc.dat
   
   /usr/sbin/opendmarc-import --dbhost=${DB_SERVER} --dbuser=${DB_USER} --dbpasswd=${DB_PASS} --dbname=${DB_NAME} --verbose < ${WORK_DIR}/opendmarc_import.dat
   /usr/sbin/opendmarc-reports --dbhost=${DB_SERVER} --dbuser=${DB_USER} --dbpasswd=${DB_PASS} --dbname=${DB_NAME} --verbose --interval=86400 --report-email $REPORT_EMAIL --report-org $REPORT_ORG
   /usr/sbin/opendmarc-expire --dbhost=${DB_SERVER} --dbuser=${DB_USER} --dbpasswd=${DB_PASS} --dbname=${DB_NAME} --verbose
   

   You will need to fill in the database details of your previously created database here.

   Make the script executable:

   chmod +x /etc/opendmarc/report_script

   And run it under the opendmarc user as a test:

   su -c "/etc/opendmarc/report-script" -s /bin/bash opendmarc

    

5. When the script worked as expected you can add it to your cron jobs:

   nano /etc/crontab
   

   Add following line:

   1 0 * * * opendmarc /etc/opendmarc/report-script
   

   This example will execute the script every day at 00:01 under the user opendmarc.

6. During testing you probably want to receive a copy of every outgoing DMARC report. You can simply add one of your Mailboxes as bcc for every Message sent by the DMARC address. Add following line to your postfix main.cf:

   sender_bcc_maps = hash:/etc/postfix/bcc_map
   
   

   Create the file bcc_map with following content:

   dmarc [at] example [dot] com mailboxforbcc [at] example [dot] com
   

   Create the mapping database:

   postmap /etc/postfix/bcc_map
   

   Restart postfix:

   /etc/init.d/postfix restart
   

Now you are done with the DMARC implementation and people can be certain, that E-Mail coming from your domain is legit. The last goal left is: redirecting spam to your users junk folders.

Know your Spam – get the most out of Spamassassin

You have to recognize spam messages before you can redirect them to the Junk folders. My tool of choice for this task is amavis-new with Spamassassin. You might have noticed earlier, that I do not actually reject any incoming E-Mails, because I do not want to accidentally loose any important messages. Detecting malicious E-Mails is entirely up to amavis-new in my setup, as you can see in this diagram :

I will assume, that you already have Spamassassin running in some form. I am going to just suggest a few score tweaks and show you the basics of writing your own rules. This will enable you to use the changes you made earlier to filter spam more effectively. Both the score changes and the custom rules should be added to the end of the Spamassassin configuration file.

nano /etc/spamassassin/local.cf

The easiest part of this is changing the score of existing rules. The syntax for this looks like this:

score RULE_NAME 1.0

• score: tells the configuration that you wish to change the score of a rule
• RULE_NAME: is the name of the Spamassassin rule, this can be any default or custom rule
• 1.0: This is the score that any messages matching the rule get. A positive score, means that it is more likely to be spam. A negative Score means that the message is more likely to be ham. And a score of 0 means, that the test will not be run.

I made the following score changes to the default SPF and DKIM rules, as I found the scores of them to be not harsh enough:

#Adjust scores for SPF FAIL
score SPF_FAIL 4.0
score SPF_HELO_FAIL 4.0
score SPF_HELO_SOFTFAIL 3.0
score SPF_SOFTFAIL 3.0

#adjust DKIM scores
score DKIM_ADSP_ALL 3.0
score DKIM_ADSP_DISCARD  10.0
score DKIM_ADSP_NXDOMAIN 3.0

Avoid using huge negative scores for a SPF or DKIM pass. Even spammers sometimes use those two these days. Scarcely any legitimate mail servers have implemented them incorrectly though, so you can usually give harsher penalties to a fail of those tests.

I found however that not all tests that I whished to do were included Spamassassin. So I had to write my own rules. This is actually fairly simple. The most important thing to remember is, that you always have to escape special characters in your regex. If you don’t you might end up searching for mistakes in the wrong place for hours. Trust me….

Here are a few rule examples that I have written for further DKIM and DMARC checks:

#dmarc fail
header CUST_DMARC_FAIL Authentication-Results =~ /mail\.example\.com; dmarc=fail/
score CUST_DMARC_FAIL 5.0

#dmarc pass
header CUST_DMARC_PASS Authentication-Results =~ /mail\.example\.com; dmarc=pass/
score CUST_DMARC_PASS -1.0

meta CUST_DKIM_SIGNED_INVALID DKIM_SIGNED && !(DKIM_VALID || DKIM_VALID_AU)
score CUST_DKIM_SIGNED_INVALID 6.0

The rule syntax has the following elements:

• Type of the rule: header, meta, body, rawbody: The type of the rule describes what part of the E-Mail the rule matches. Meta rules are special. They let you link multiple rules with logical ‘and’ or ‘or’ operators and will be true if your entire construct returns a true. In order to deal with SPF, DKIM and DMARC, you will only need header and meta rules.
• Rule name: CUST_DMARC_FAIL: You need to specify a name for the rule. I recommend using a fixed prefix for your custom rules. This will make debugging the rules easier if something does not do what you want it to. You should watch out for this special naming convention: If you prefix your rule with ‘__’ it will only be evaluated as part of a meta rule.
• The rule itself is different for header rules and meta rules.
  • Header rules: usually consists of two parts. The header field that should be searched and a regex that matches the desired content in the header field. For example: ‘Authentication-Results =~ /mail\.example\.com; dmarc=fail/ ’ Matches the header field ‘Authentication-Results’ if it contains the string ‘mail.example.com; dmarc=fail’. The header field that you match is separated from the regex by ‘=~’. The regex that describes your search text is started and ended by ‘/’. All special characters like a dot or an ‘@’ have to be escaped with a ‘\’.
  • Meta rules: consist of rules linked by logical operators. For example: ‘DKIM_SIGNED && !(DKIM_VALID || DKIM_VALID_AU) ’,  The Operator ‘&&’ signals a logical ‘and’ and the operator ‘||’ signals a logical ‘or’. Expressions in parentheses will be evaluated first.

Once you tuned all the rules above, your spam will be tagged reliably.

The finishing Touch – 6 Easy Steps to use Dovecot and Sieve to redirect spam to your users junk folder

As I said a bunch of times already I don’t like loosing E-Mail. But I don’t want anybody to be annoyed by inboxes full of spam. While many E-Mail programs are able to to automatically move tagged spam, many smartphone clients do not have that ability. That is why I use Sieve server side filters as a plugin for Dovecot. Sieve allows you to write simple scripts, that change the local mail delivery. In this last part of the I will walk you through installing Sieve and setting up a global script that always gets executed first.

Sieve allows you to do many other cool things including letting your users write their own scripts through a mail client, but that is out of scope for this article.

1. Sieve is packaged in Debian, so it is fairly easy to install:

   apt-get install dovecot-sieve

2. Once the installation is done you need to activate sieve in your configuration:

   nano /etc/dovecot/conf.d/15-lda.conf

   Change this section to include sieve:

   protocol lda {
     # Space separated list of plugins to load (default is global mail_plugins).
     mail_plugins = $mail_plugins sieve
   }

    

3. Next you need to change the default sieve configuration of dovecot to run always run a script before running user scripts:

   nano /etc/dovecot/conf.d/90-sieve.conf

   Here you need to find the plugin section and uncomment or add the following option:

   plugin {
   sieve_before = /var/mail/SpamToJunk.sieve
   }

   If you happen to have an old monolithic dovecot.conf that does not include the /etc/dovecot/conf.d directory, you have to make all these changes in /etc/dovecot/dovecot.conf.

    

4. Now that you configured dovecot to use the Script, you still have to create it. But that is fairly easy:

   nano /var/mail/SpamToJunk.sieve

   Use this script to move all spam into the Junk folder of the corresponding Mailbox:

   require "fileinto";
   if header :comparator "i;ascii-casemap" :contains "X-Spam-Flag" "YES"  {
       fileinto "Junk";
       stop;
   }

    

5. Since this script will be run on every incoming E-Mail, it is a good idea to compile it:

   sievec /var/mail/SpamToJunk.sieve

6. Restart dovecot to enable your new configuration:

   /etc/init.d/dovecot restart

Now you have a mail server that is much better at detecting spam and phishing attempts and whose E-Mails can be authenticated by other mail servers.

Feel free to leave a comment or shoot me an E-Mail if you have any questions!

Helpful Links and more Information

DMARC

http://www.dmarc.org/faq.html
http://blog.hamzahkhan.com/2014/02/08/securing-postfix-mail-server-greylisting-spf-dkim-dmarc-tls/
https://domsch.com/blog/?p=641
http://www.trusteddomain.org/opendmarc/opendmarc.conf.5.html
http://dokuwiki.nausch.org/doku.php/centos:mail_c6:mta_13
https://agari.zendesk.com/hc/en-us/articles/202952769-What-are-the-different-Failure-Reporting-options-I-can-configure-in-my-DMARC-record-
http://dokuwiki.nausch.org/doku.php/centos:mail_c6:mta_13(German)
https://halon.io/blog/considerations-regarding-dmarc-forensic-reports-and-other-implementation-notes/
http://kitterman.com/dmarc/assistant.html

DKIM

http://linuxaria.com/howto/using-opendkim-to-sign-postfix-mails-on-debian
http://www.linuxlasse.net/linux/howtos/Postfix_with_DKIM_%28OpenDKIM%29_and_SPF
http://tecadmin.net/setup-dkim-with-postfix-on-ubuntu-debian/
http://www.gettingemaildelivered.com/dkim-explained-how-to-set-up-and-use-domainkeys-identified-mail-effectively

SPF

https://help.ubuntu.com/community/Postfix/SPF
http://www.openspf.org/Introduction
http://blog.iwader.co.uk/postfix-implement-spf-record-checking/

Basic Setup

Setting up a Mumble server is fairly simple, but I had a few problems to get it to play nice with my Active Directory. And since there wasn’t exactly a great deal of documentation on the subject, I figured that I should write a short How To.

I started with a new Debian Wheezy OpenVZ machine. I Installed the Mumble Server package from the Debian repository onto that:

apt-get install mumble-server

After the Installation is complete, you can configure the basics of the Mumble Server in its configuration file:

nano /etc/mumble-server.ini

If you configure it the way I did, only people in a specific Active Directory Group will be able to connect to the Server. However the Active Directory Authentication runs as its own Service and should that be down, anybody would be able to connect. If that is not something you are ok with, you could set a Server password in addition to the suggested configuration changes here.

1. Change the ice passwords. It is generally a good idea to have passwords on sensitive services. But these particular passwords will not be needed often, so there is really no reason not to use long randomly generated Passwords here:

icesecretread=Password1
icesecretwrite=Password2

2. You might want to change the maximum allowed bandwidth per user. Since my Server has a pretty decent Internet connection, I choose to set it to the maximum for option. I did not test if this value can be set higher, but I saw this described as the maximum value on several sites. This change does not effect clients with a low bandwidth Internet connection. When a connection is established, Mumble will check the server side and client side bandwidth settings and use the lower one of the two.

bandwidth=128000

That’s it. There is a bunch of other settings in the configuration, that allow you to customize your Server and quite a bit more, but you don’t have to touch them. The defaults are ok.

You also need to set the Superuser password with following command:

dpkg-reconfigure mumble-server

Now its time to start the mumble Server and test it with a client:

/etc/init.d/mumble-server start

Connecting with the client is easy:

1. Open Mumble and select Server –>Connect

2. Select Add New and enter the details of your server and use superuser as user:

3. A certificate error will pop up, you can accept this certificate.

4. You will be asked for a password. This is the Password you set in the dpkg-reconfigure step.

5. You should be logged in right now.

Integrating it into the Active Directory

With the basics working, it is time to get to the part that caused me a little headache. There is one Open Source Project that provides a bunch of scripts for Mumble including one for LDAP Authentication. The Script and config file are well commented, but there is very little documentation other than those comments. The Mumble Scripts Project can be found on this Github page.

Unfortunately it did not work out of the box for me. User authentication generally failed when I used the “cn=username,ou=users,dc=example,dc=com” way of specifying the user Objects. Tough I figured out fairly fast, that the NTDOMAIN\username way of providing usernames worked fine.

The next problem was, that this script doesn’t work great when the users are in different OUs in your AD. And to top it off the Group Membership check did not work for me with its original filter. This means I had to change a few parts of the script to get it to work with my Active Directory. These changes work great for me, but with those changes the script will probably not work with non Active Directory LDAP Backends.

I downloaded the Scripts and put the file LDAPauth.py in the directory “/opt/mumble-scripts” and the LDAPauth.ini into the directory “/etc/mumble-scripts”.

For the LDAPauth script to work correctly I needed 2 additional Python modules. So I installed them:

apt-get install python-ldap python-daemon

Now I will save you a little bit of headache and show you the parts of the script, I modified in order to get this working with my AD:

The first change I made was in the way the username was put together for the initial bind. With that change I can use the NTDOMAIN\user name scheme to authenticate against the AD. The original implementation was also limiting the usefulness of the script, if the users are in multiple OUs. This change has to be done twice in the file LDAPauth.py at Line 463 and 516:

original:
bind_dn = "%s=%s,%s" % (cfg.ldap.username_attr, name, cfg.ldap.users_dn)

replaced by:
bind_dn = "%s\%s" % (cfg.ldap.nt_domain, name)

Next up is the search filter for the user. The filter in the original script will work, but is a little basic. With the change, the filter only matches user objects that have not been deactivated. This is in line 478 of the LDAPauth.py:

original:
res = ldap_conn.search_s(cfg.ldap.users_dn, ldap.SCOPE_SUBTREE, '(%s=%s)' % (cfg.ldap.username_attr, name), [cfg.ldap.number_attr, cfg.ldap.display_attr])

replaced by:
res = ldap_conn.search_s(cfg.ldap.users_dn, ldap.SCOPE_SUBTREE, '(&(%s=%s)(objectCategory=user)(!(userAccountControl=514)))' % (cfg.ldap.username_attr, name), [cfg.ldap.number_attr, cfg.ldap.display_attr])

The last change that I needed to make was the LDAP search string for the group membership. The Original would definitly not work with users in different OUs and im not quite sure, if Active Directory even tracks group memberships that way. The new searchstring starts at the users_dn which I treat as a basedn and looks for any not deactivated user accounts that have the group listed in their memberOf attribute. This change has to be made at line 499:

original:
res = ldap_conn.search_s(cfg.ldap.group_cn, ldap.SCOPE_SUBTREE, '(%s=%s=%s,%s)' % (cfg.ldap.group_attr, cfg.ldap.username_attr, name, cfg.ldap.users_dn), [cfg.ldap.number_attr, cfg.ldap.display_attr])

replaced by:
res = ldap_conn.search_s(cfg.ldap.users_dn, ldap.SCOPE_SUBTREE, '(&(%s=%s)(&(objectCategory=user)(!(userAccountControl=514))(memberof=%s)))' % (cfg.ldap.username_attr, name, cfg.ldap.group_cn),[cfg.ldap.number_attr, cfg.ldap.display_attr])

Next is the configuration file “/etc/mumble-scripts/LDAPauth.ini”. Here I left the user settings as they were.

The ice settings needed adjustments. The slice path was initially not correct and the secret should be set to the ice write password.

The LDAP settings needed most adjustments. The first change is the ldap uri, this should be set to the address of your Domain Controller.

The next change is the users_dn with the changes I made to the script, this is more or less the same as the base dn in other ldap scripts. Choose a path from which all relevant users can be found.

After the users_dn I inserted a new option: nt_domain. The changes I made to the authentication script, require this option to be here and to be set.

The username_attr is the filed which contains the logon name. In Active Directory this is sAMAccountName.

The number_attr requires a filed with a unique Integer as content, because mumble uses this as user id and maps its own groups to this ID. I choose the pager field, since I don’t know anyone who owns a Pager. Since the field is usually free you have to remember enter a unique number in this field every time you add a user to mumble. Alternately you could try using the uSNCreated field in small Active Directories. This field should be unique as long as you only use one Domain Controller to create users.

The display_attr field points to the field containing the display name. This is pretty much the purpose of the LDAP displayName attribute. Alternately you could use sAMAccountName for this again.

The next option group_cn is the ldap path to the group mumble users should be members of.

The group_attr option on the other hand has no function anymore since the new search string for groups has this hard coded.

The rest of the Options can be left as they are. Here is an example ini file for reference:

;Player configuration
[user]
;If you do not already know what it is just leave it as it is
id_offset       = 1000000000
;Reject users if the authenticator experiences an internal error during authentication
reject_on_error = True
;Reject users that are not found when bind_dn is used with non-user credentials.
;Setting this to False will cause a fall-through when the user is not found in LDAP.
reject_on_miss  = True

;Ice configuration
[ice]
host            = 127.0.0.1
port            = 6502
slice           = /usr/share/slice/Murmur.ice
secret          = cPB3sjD5c_grkWEZOoxg
watchdog        = 30

; LDAP specific configuration
[ldap]
; Use bind_dn and bind_pass if you use non-user credentials for searches.
;bind_dn = SKELLDOM\searchuser
;bind_pass = password
ldap_uri = ldap://dc2.skelleton.net
users_dn = ou=Local,dc=skelleton,dc=net
nt_domain = SKELLDOM
username_attr = sAMAccountName
number_attr = pager
display_attr = displayName
group_cn = cn=Mumble_Users,ou=Groups,ou=Remote,dc=skelleton,dc=net
group_attr = memberOf
;group_cn =

;Murmur configuration
[murmur]
;List of virtual server IDs, empty = all
servers      = 

;Logging configuration
[log]
; Available loglevels: 10 = DEBUG (default) | 20 = INFO | 30 = WARNING | 40 = ERROR
level   = 10
file    = /var/log/LDAPauth.log

[iceraw]
Ice.ThreadPool.Server.Size = 5

With all those changes you can start the script by entering:

/opt/mumble-script/LDAPauth.py

You should now be able to use the mumble client to log into the server with an authorized Active Directory account. If this works you are almost done. If not look in the logfile to see where the problem is.

Starting the Authentication Script as a Service

I am using a slightly modified version of the /etc/init.d/skeleton script to run the LDAP authentication as a service. One of the things I had to modify was the Required-Start line. I added “$all” to this line, to ensure, that the LDAPauth script starts after mumble. Otherwise it will not start. The major problem in this step was something else however. The start-stop-daemon created a pid file with the wrong process id in it. Because of this the init script was not able to shut down the running LDAP auth daemon. I added a small command to the script that creates the pid file with the correct process id to the script and now everything works as expected.

Here is the start/stop script I use:

#! /bin/sh
### BEGIN INIT INFO
# Provides:          mumble-ldap
# Required-Start:    $remote_fs $syslog $network $all
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: LDAP Authentication for Mumble
# Description:       LDAP Authentication for Mumble
#                    placed in /etc/init.d.
### END INIT INFO

# Author: skelleton
#
# Please remove the "Author" lines above and replace them
# with your own name if you copy and modify this script.

# Do NOT "set -e"

# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="Mumble LDAP Authentication"
NAME=mumble-ldap
DAEMON=/opt/mumble-scripts/LDAPauth.py
DAEMON_ARGS="-d -i /etc/mumble-scripts/LDAPauth.ini"
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
DAEMON_USER=mumble-server

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.2-14) to ensure that this file is present
# and status_of_proc is working.
. /lib/lsb/init-functions

#
# Function that starts the daemon/service
#
do_start()
{
    # Return
    #   0 if daemon has been started
    #   1 if daemon was already running
    #   2 if daemon could not be started
    start-stop-daemon --start --chuid $DAEMON_USER --pidfile $PIDFILE --startas $DAEMON --test > /dev/null \
        || return 1
    start-stop-daemon --start --chuid $DAEMON_USER --pidfile $PIDFILE --startas $DAEMON -- \
        $DAEMON_ARGS \
        || return 2
    # Add code here, if necessary, that waits for the process to be ready
    # to handle requests from services started subsequently which depend
    # on this one.  As a last resort, sleep for some time.
    pgrep -f /opt/mumble-scripts/LDAPauth.py > $PIDFILE
}

#
# Function that stops the daemon/service
#
do_stop()
{
    # Return
    #   0 if daemon has been stopped
    #   1 if daemon was already stopped
    #   2 if daemon could not be stopped
    #   other if a failure occurred
    start-stop-daemon --stop --retry=TERM/30/KILL/5 --pidfile $PIDFILE 
    RETVAL="$?"
    [ "$RETVAL" = 2 ] && return 2
    # Wait for children to finish too if this is a daemon that forks
    # and if the daemon is only ever run from this initscript.
    # If the above conditions are not satisfied then add some other code
    # that waits for the process to drop all resources that could be
    # needed by services started subsequently.  A last resort is to
    # sleep for some time.
    start-stop-daemon --stop --oknodo --retry=0/30/KILL/5 --exec $DAEMON
    [ "$?" = 2 ] && return 2
    # Many daemons don't delete their pidfiles when they exit.
    rm -f $PIDFILE
    return "$RETVAL"
}

#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
    #
    # If the daemon can reload its configuration without
    # restarting (for example, when it is sent a SIGHUP),
    # then implement that here.
    #
    start-stop-daemon --stop --signal 1 --pidfile $PIDFILE 
    return 0
}

case "$1" in
  start)
    [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
    do_start
    case "$?" in
        0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
        2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
    esac
    ;;
  stop)
    [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
    do_stop
    case "$?" in
        0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
        2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
    esac
    ;;
  status)
    status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
    ;;
  #reload|force-reload)
    #
    # If do_reload() is not implemented then leave this commented out
    # and leave 'force-reload' as an alias for 'restart'.
    #
    #log_daemon_msg "Reloading $DESC" "$NAME"
    #do_reload
    #log_end_msg $?
    #;;
  restart|force-reload)
    #
    # If the "reload" option is implemented then remove the
    # 'force-reload' alias
    #
    log_daemon_msg "Restarting $DESC" "$NAME"
    do_stop
    case "$?" in
      0|1)
        do_start
        case "$?" in
            0) log_end_msg 0 ;;
            1) log_end_msg 1 ;; # Old process is still running
            *) log_end_msg 1 ;; # Failed to start
        esac
        ;;
      *)
        # Failed to stop
        log_end_msg 1
        ;;
    esac
    ;;
  *)
    #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
    echo "Usage: $SCRIPTNAME {start|stop|status|restart|force-reload}" >&2
    exit 3
    ;;
esac

:

Once the script is created, there is only one thing left to do:

update-rc.d mumble-ldap defaults

This ensures, that the LDAP authentication is started when the server boots.

Things that should be considered before you attempt to restore the data yourself

1. Make sure that there is nothing that writes to the disk from which you want to recover data. If you are recovering data from RAID volumes, make sure that there are no writes to any of the disks in the volume. Even if your data recovery is just about a simple deleted file, you will want to avoid writes to the disk. This is because any write cloud potentially overwrite recoverable data. Since some file systems write to the disk even when files are read, you should avoid mounting the disk directly or mount the disk as read only.
   If the hard drive is still in the Computer/NAS it is usually in, shut that system down or do not boot it the normal way. There may be programs or services that write to the affected drive. Starting that computer from a Live CD is usually OK though.
2. If it is clear that there is a serious mechanical or electronic defect ,often identified by problem descriptions such as: “sounds funny” or “smells funny”, ask the user to consider a data recovery lab before you start working on it yourself. Anything you do in such a situation could negatively effect the results of the lab.
3. Always image or clone the disk you are working on and only use the clone going forward. If you suspect that you wont be able to read the original disk a 2nd time, only work with copies of your image/clone. If your recovery gets somewhat more complex it is easy to do something irreversible that will negatively effect your chances of getting the data back.
   Even if creating the images takes a lot of time, just let it run and do something else meanwhile. Having a means to go back to to the original state is just basic cover your ass.

The Situation

I got the entire NAS-Box delivered. It was a Buffalo LinkStation DUO. It still booted (In retrospect I should have cloned the disks before trying to boot) up ok and the RAID was shown as normal, all the shares were listed when I accessed it over SMB, but I could not open any of the shares. The web interface showed the RAID to be OK. But I did not see any values for total storage and free space, which these LinkStations usually show prominently in the Web GUI. The disk settings showed both disks in the list, but the only information shown about them was they product number. This was very odd.

The disks were from Seagate so pulled them and put them in the USB dock on my desktop. I ran the Seatools and Crystal Disk Info to display the SMART values. One of the disks was fine, the other had almost 4000 reallocated sectors and over 700 sectors that were defective but could not be reallocated. The problem was fairly obvious….

Cloning the disks with dd

Next I put those disks (one at a time) and two larger disks (also one at a time) into a Linux machine and cloned the original disks to the other disk I put in with it. The linux tool dd does a great job at copying hard drives at block level and it is my go to tool if I need a block for block image from a disk. Especially if I need to work with that Image.

The dd command is fairly simple and short, but it will run quite some time (especially when there are bad sectors on one of the hard disks) and only give you an output if it encounters an error. The following example is assuming that your source disk (the ones from the NAS) is /dev/sdb and your target disk (the one you put in with it) is /dev/sdd:

Be absolutely certain that you selected the correct output before sending the command!

dd if=/dev/sdb of=/dev/sdd conv=noerror,sync bs=512

The options have the following meaning:

• if: the input for dd. In this case the entire disk sdb.
• of: the output for dd, this can be a block device or a file. Here the target is sdd. Be absolutely sure that you selected the correct output!
• noerror: tells dd to continue even if errors are encountered
• sync: tells dd to fill the block with zeros if an error is encountered, this means that the size of the original will be kept. Otherwise a block that could not be read will be skipped in the output and next readable block will be written directly after the last readable block. This would mess with file allocation tables.
• bs: This specifies the block size. If the disk has bad sectors try to make this match the size of the disks sectors (in this case 512 byte) otherwise pick a larger block size (64k is a good value) to speed up the process. If you pick a larger block size on a faulty drive, the entire block will be written as zeros when a sector can’t be read.

Since the cloning process can take a lot of time and only gives you any output when it encounters an error, you might want to get a status from time to time. To do this open a second terminal and find out the process id of the running dd process with following command:

pgrep -l '^dd'

Once you know the Process ID you can run following command to produce an “ERROR” message in the first terminal with the current status of the dd (substitute the 38862 for the process id you got in the previous step):

kill -USR1 38862

Getting the RAID back online

After the cloning was done, I stored the source disks safely and made sure both clones were in my Linux comupter. The disks were using gpt and had 6 partitions each. The last partition was clearly the one with the data stored. The other 5 partitions seemed to be system and swap partitions for the NAS. The system partitions were set up as a RAID1. That seems sensible and I still have no idea why RAID0 is the default for the data partition.

After looking at the general structure of the disks and partitions, I had a closer look at the data partitions. I first tried to simply add both partitions to a new RAID. This will recognize existing RAID superblocks on the partition and recreate the an already existing RAID:

mdadm -A /dev/md123 /dev/sdb6
mdadm -A /dev/md123 /dev/sdd6

If this works for you, you can continue reading here.
But unfortunately there was no mdadm superblock found on the clone of the defective drive. I tried to find it using a bunch different tools to see if I could find the superblock, but I failed horribly.

So I decided to continue using more drastic measures (read this as potentially destructive) and recreate the RAID by force. But first I needed more Iiformation on what settings the RAID was created with. For that I simply read out the RAID superblock on the data partition of the good disk:

mdadm --examine /dev/sdb6

The output looked similar to this (I got this output when I ran the examine on one of the cloned drives in a USB dock a few weeks later):

/dev/sdf6:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 73bfe225:b30b764b:faf5b20a:9ac09813 (local to host skelsrv)
  Creation Time : Thu Mar 20 23:54:11 2014
     Raid Level : raid0
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 2

    Update Time : Thu Mar 20 23:54:11 2014
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : c04d5a9 - correct
         Events : 1

     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8       22        1      active sync

   0     0       8       54        0      active sync
   1     1       8       22        1      active sync

even though its not exactly the output I had originally, it still shows all the relevant information:

• Version: the metadata Version of the RAID, it is important that this matches if you recreate the RAID.
• RAID Level: In case you don’t already know what RAID Level was used
• Total Devices: In this case it was clear from the beginning, but with 4 Bay NAS Boxes, the RAID might not use all the Devices
• Chunk Size: This one is also very important to get right
• The Order of the disks: You should specify the disks in the correct order in the create statement. In case of the output above, the disk is the second drive in the array.

Once you have all that information, you can create the RAID again:

mdadm --create /dev/md123 --assume-clean --level=0 --verbose --chunk=64 --raid-devices=2 --metadata=0.90 /dev/sdd6 /dev/sdb6

Mounting the XFS file system

Once the RAID is recreated you will have to mount the partition to see if you can get any data back. I strongly recommend mounting the partition as read only (for this to work you need to have xfs support, on Debian that is provided by the xfsprogs packet):

mkdir /mnt/md123
mount -o ro -t xfs /dev/md123 /mnt/md123

If this works, great. If your disaster is as bad as the one I was dealing with, you might get an error message along the lines of: the partition can’t be mounted because of log file corruption. I was not dissuaded and decided to ditch those pesky log files and all the files that had not been completely written to the NAS at the time of the crash with them. After all I was pretty far already and getting back some or even most files seemed to be better than getting none of them.But before you try this be aware that this form of repair pretty much drops the existing log files and can cause further corruption to the file system.:

xfs_repair -L /dev/md2
xfs_check /dev/md2
mount -o ro -t xfs /dev/md2 /mnt/md2

This last mount command worked for me and I could copy files off the newly mounted RAID. But as pretty much expected some of the files were corrupt. (You could copy them but not open them with their respective programs or you could open them and the content was garbage). Lucky for me I don’t have to figure out which files got corrupted, because that seems like a real pain in the ass.

Anyway once you retrieved the data, make sure to stress the importance of Backups before handing out the restored files.

If you do happen to have a convenient method to check over 700000 files of different formats (MS Office, PDF, single saved E-Mail messages from outlook, jpg files, Autocad files and a bunch of others) please share. But even if you don’t I hope that this article could help you.

A while ago somebody came to me with a problem. A bunch of Services on their Server stopped working. The admin password in the documentation didn’t work. The Server in question was a SBS with a bunch of additional software that was critical to the business. My questions for Backups were answered with silence.

Click here to skip all my gabbing and warnings. And go directly to the guide.

The Options

Obviously backing up all the data With a Boot CD and reinstalling the Server was to be avoided at all costs, but that Option is on the table if there are actually any recent backups (check them before trying to restore them to the server). I had reset Local Windows user accounts with before without much of a problem. But domain accounts are a very different issue and the tools for resetting local accounts don’t work for them. It did not help that Microsoft’s official stance on this problem is something along the lines of: if you loose the password to your only domain admin it can’t be recovered or reset.

Well that is not entirely true since you could restore a Backup of the Active Directory from a point time when your documented password still worked. Though this could cause a whole bunch of other issues. And in this scenario it is likely, that you will be told that it worked yesterday. Of course this will turn out to be wrong.

After a little bit of research (search Google for the problem and ignore Microsoft websites) I found a fast, easy and almost painless solution. The only issue with solution is, you will need to reboot the server from a Boot CD.

A few words of warning

I tested this with a Windows Small Business Server 2008. While this guide should also work for other Windows 2008, 2008R2 and even Windows 2012 based Domain Controllers, I have not had to test it yet and I can’t guarantee that this will work on those Operating Systems.. I am especially not sure if Windows Server 2012 still executes the ease of use utility with system rights and if the executable file is still named the same.
You should also know, that there is no guarantee that Microsoft won’t fix this in the future, since this is a security loophole that could be used to gain unauthorized access to the network.

The Solution

I am providing this step by step guide for fellow tech support people, who might be confronted by such a situation by no fault of their own. If you bring all the needed Tools and the Server boots decently fast, you will be done within a few minutes. And you should be aware, that doing this is illegal in many countries if you are not authorised by the owner of the server. So get the job in writing with a signature from the companies boss before you start.

With that said, just follow these steps to save the day:

1. First you need to gain local access to the server (though an IPMI that allows you to boot from an iso file should work as well).
2. Now you need to boot from any kind of Boot CD that allows you write access to NTFS File systems. For this you can use one of the many Recovery Linux Distributions out there or even the Windows Install CD that came with the server. In case of the Windows CD you want to open the windows repair console.
   I should mention here, that the Operating System you are booting into should support the disk controller that is connected to your windows drive(s)
3. Now navigate to the system32 Folder within your Windows Folder. In the Windows Recovery Console It might look something like this (though your windows drive does not necessarily have to be C:):

   cd C:
   cd C:\windows\system32

4. Once you are there you have to replace the Windows Ease of Access Tool with the Windows command line. Since you will want to reverse this change once you are done, I would advise against any action that would permanently remove this utility. The Ease of Access Utility is the file “utilman.exe” and the command line is “cmd.exe”. With the Windows Recovery Console you would accomplish this with the following commands:

   rename utilman.exe utilman.exe.old
   copy cmd.exe utilman.exe

5. After you replaced the utility, you need to reboot into the windows Installation.
6. Click The Button shown in the Windows 7 logon screen. While this is not one of the Server Operating Systems, The logon Screen is pretty much the same for them.
   
7. Clicking this Button will open a cmd window with system privileges. In this window you should type the following (replace Administrator with the username of you Domain Admin):

   net user Administrator newpassword123

8. You can log in now. You should put to original utilman.exe back in its original place. The easiest and cleanest way to do it is rebooting and doing it with your boot cd again.

Once you are done, you might want to inform the person running the server about the importance of Backups and proper documentation.

What is Apex?

APEX is a rapid application development tool from Oracle. It lets you design, implement and deploy database applications with a very easy to use web gui. For common task like showing reports or creating forms for data entry you can use one of the many prepared templates. With APEX you could create an entire application around your database with little to no knowledge of any programming or scripting language.
If you need more specialised pages, then the default templates offer to you , you can build a page from scratch with any combination of predefined elements, custom PL/SQL code and custom javascript / HTML code.

What are the requirements of APEX?

Apex needs an Oracle database to work with. But since the free Oracle XE comes with APEX already, there wont be any licensing fees for smaller Apex Applications. Oracle XE has some limitations on the amount of data you can store with it, the amount RAM it can use and it is limited to one core, but at the very least you can use it to check out Oracle and Apex before buying an Oracle License.
While Oracle XE come with a version of APEX, it does not come with the current version. I recommend to update the APEX version of Oracle XE since the new version has a few nice features.
Apex itself is a free download from the Oracle website, but you need to register an account on the website to download it. The same goes for the Oracle XE Database.

Anything that I need to watch while installing Oracle Linux?

The Oracle Linux installation is fairly standard for the most part. The only thing that you might stumble over is the network configuration. You have to explicitly enable every Network Adapter you want to use, otherwise you will not have any network access with the newly installed Linux until you enabled a Network Adapter from there.

Below is a small step by step guide for the installation. If you know how to install Linux you can skip all the pictures and continue reading at the next Heading.

Linux is up and running what now?

Before you try to install apex, you should make sure that Hostname of your Oracle Linux Machine can be resolved by the DNS Server it is using. The installation failed for me when that was not the case on my Server.
If your don’t have a local DNS Server or if you have no control over the DNS Server, you can also use the /etc/hosts file for name resolution. Open the file in you editor of choice and add a line like this:

# IP address of the host        hostname
192.168.0.1                     oraclelinux

How do I install Oracle XE and APEX?

Next on the Agenda is installing and configuring Oracle XE.

1. Download Oracle 11g XE from Oracles website and copy it to your Server.
2. Navigate to the directory you copied the download into and extract and install the package. The following example assumes /tmp as directory and the current version of Oracle XE at the time of this writing:

   cd /tmp
   unzip -x oracle-xe-11.2.0-1.0.x86_64.rpm.zip
   cd Disk1
   rpm -ivh oracle-xe-11.2.0-1.0.x86_64.rpm

3. Once the installation is done, you have to configure the Database. This is done with following command:

   /etc/init.d/oracle-xe configure

   You will be asked a few questions and after that the configuration will be done. The configuration will fail, if it can’t resolve the hostname of the machine Oracle is on.
   The Script will ask for the following Information:
   – The HTTP Port for APEX: the default is 8080
   – The Port for the Oracle Listener: the default is 1521
   – The Password for the Administrative accounts of the Database (SYS and SYSTEM): choose a secure Password and write it down
   – Do you want the Database to start automatically.

4. Once the configuration is complete, you need to log into your database with SQLPlus. But before you can do that, you will need to set some Environment variables. Oracle does provide a script to set them for you, but at least on my Oracle Linux test machine, the script didn’t work.
   Since the environment variables only need to be set, when you want to access the database from the command line, I set them by hand:

   export ORACLE_HOME=/u01/app/oracle/product/11.2.0/xe
   export ORACLE_SID=XE
   export NLS_LANG=`$ORACLE_HOME/bin/nls_lang.sh`
   export PATH=$ORACLE_HOME/bin:$PATH

5. Now you can run the sqlplus command from the command line to enter the Database.  You need to enter the Database as system and tell Oracle to respond to network connections. Run following command:

   sqlplus system

   You will be promted for the password that you set during the configuration. Once you have entered the password, you are in the command line interface of Oracle XE. Enter following line to enable the webserver for the network:

   EXEC DBMS_XDB.SETLISTENERLOCALACCESS(FALSE);

   Exit from sqlplus after that.

6. While Apex is up and running now, you can still not access it, because of the firewall. To change the firewall settings execute following command on from your Linux command line:

   system-config-firewall-tui

   This will open a configuration menu for the firewall. Here you must first select the Option “Customise” and then press enter. Next you select “Forward”. On this screen select “add”. A new mask will pop up. fill it in like in the picture below:
   Select “OK” when you are done and then select “Close” on the next screen. After that confirm all your changes with “OK”
   Now you should be able to contact the Oracle Apex Server from your workstation with the following url: http://ip.of.your.oraclelinux:8080/apex/apex
   If you can reach this page your Apex installation is good to go. You could start to work on your application, but I recommend installing the newest update for apex, before you start creating your own applications.

Updating apex to Version 4.2.2

Download the current APEX release from the Oracle website and copy it to a directory on your Oracle Linux Server. At the time of this writing the current Version of APEX is 4.2.2. This process will probably stay the same for all minor updates. The following steps will assume that you copied apex into the directory  “/tmp”

1. Download the current APEX release from the Oracle website and copy it to a directory on your Oracle Linux Server.
2. Enter the directory APEX is in and unpack it. and then enter the newly created directory:

   cd /tmp
   unpack -x apex_4.2.2.zip
   cd apex

3. Start sqlplus and connect with the user system. Please note that, as stated above, the correct environment variables need to be set for this.

   sqlplus sys as sysdba

   Enter your password when prompted and execute the following command:

   @apexins.sql sysaux sysaux temp /i/

   This runs the apex installation script with the following parameters:
   Tablespace for APEX user: sysaux
   Tablespace for APEX Files user: sysaux
   Temporary Tablespace: temp
   Virtual image directory: /i/
   The Values in the example command are correct for default apex installations. Don’t worry if this takes a while, that is normal.

4. After the script is finished, you should check if the upgrade was successful. You can do this by executing following command in sqlplus:

   select status from dba_registry where comp_id = 'APEX';

   The out put should look like this:
   

5. Since this was an Upgrade of APEX, it is necessary to load the images of the new version into the database.
   In order to do this, you need to start sqlplus as sysdba and run following command:

   @apxldimg.sql /tmp

   You need to specify the folder where you extracted apex to. If you followed the tutorial, you extracted apex to “/tmp”.

6. If you want to change your password already you can do it with the following command in sqlplus:

   @apxchpwd

7. The update is complete. The apex Web Interface is available at http://ip.of.your.oraclelinux:8080/apex/apex

As you may have noticed, this Mail Server is not just a simple Mail Relay, it also has user mailboxes. If you are just looking for a Mail Relay for you Exchange Server, you will only need the Postfix part of this Article.
The entire configuration is somewhat unusual if you already have a Mail Server on your premises, but there were some requirements for it.

This Sounds pretty specialised, what do I need that for?

The Answer is simple: A Server like this can be incredibly practical if you have Mailboxes with providers, that only allow for POP3 connection, or which have some very strict space restrictions.
Using a Server like this will allow you to Synchronise your E-Mail across multiple devices and you wont have to worry about the Mailbox size restrictions your provider set for you.

Ok that seems useful. How does it work?

The Mail System I will describe to you in this article, has four basic components:

1. Fetchmail: Downloads the E-Mails from your Providers Mailbox to your Server
2. Procmail: Is used by Fetchmail to put the E-Mails in the correct directory on your Server
3. Dovecot: Is the IMAP Server that your E-Mail clients will use
4. Postfix: Is used to send the outgoing E-Mails to the correct smart host or Provider Mail Server

I made a little Diagram to show how all the Parts work together:

In this Scenario all Clients with the default E-Mail Domain use Outlook to work with the Exchange Server, like they normally would. The Exchange gets his Mails with the POP3 Connector and sends all outgoing Mail to the Debian Mail Relay Server. Postfix checks the sender address and then Sends the Mail to the correct smar thost in the internet.
The E-Mails of the other Maildomain are downloaded with Fetchmail and locally delivered with Procmail to the Maildir Folder of the corresponding local user.
The users use Outlook and other E-Mail clients to work in their Mailboxes. When they send E-Mail, they use SMTP directly from their E-Mail client to connect to the Mail Relay server. It is again postfix who checks the sender of the outgoing E-Mail and sends it to the correct Smart Host.

Ok I want something like that How do I start?

Well that depends on what you have to work with, you may setup a small server that will run permanently, or you install all the needed programs on a server that is already running. To be quite honest, it is probably a waste to run this on dedicated hardware and a virtual machine or container is probably your best option.
So the as a first Step install Debian into the (Virtual) Machine of your choice. Since this entire Mail Server has no special networking requirements I choose to run it in an OpenVZ container.

The OpenVZ Template I used for this Debian install came with postfix out of the box.
If you installed Debian from a normal install medium, you will likely need to install postfix after the OS installation:

apt-get install postfix

The next things you will need are libsasl2 and libsasl2-modules. The former was already installed in my Template so I only had to install the modules:

apt-get install libsasl2-modules

#probably not needed for dovecot auth test by disabling saslauthd and sending E-Mail

#new after messing with smtpd auth

install sasl2-bin

/etc/default/saslauthd

edit enable to yes

start saslauthd

Since Dovecot and Fetchmail will be needed later on, you should install them now:

check the commands and if they get all other needed packages

apt-get install fetchmail dovecot-common dovecot-imapd

And as last part of the basics, you need a user to receive the mail with. For that run following commands:

useradd -G mail username
passwd username

OK all the stuff is installed. How do I get postfix running?

The Postfix configuration for this server is fairly simple. Since Postfix will only be used to relay mails to the smart hosts of every mail domain, there is no need to set up a local delivery of the E-Mails. Most of the configurations done in the file “/etc/postfix/main.cf”. Here is the main.cf file used for this server:

# See /usr/share/postfix/main.cf.dist for a commented, more complete version

smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h
#alias_maps = hash:/etc/aliases
#alias_database = hash:/etc/aliases
#mydestination = $myhostname, localhost.$mydomain, localhost
smtp_sender_dependent_authentication = yes
sender_dependent_relayhost_maps = hash:/etc/postfix/relayhost_map
smtp_sasl_auth_enable=yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options =
smtp_sasl_type = cyrus
smtp_sasl_mechanism_filter = login,plain
relayhost = smtp.mainmaildomain.com
mynetworks = 192.168.0.0/24 127.0.0.0/8
inet_interfaces = all
recipient_delimiter = +
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_recipient_restrictions = permit_mynetworks,permit_sasl_authenticated,reject_unauth_destination

If you have never used postfix, you might not understand what all those values mean. So let me give you a quick rundown.
All options that start with “smtp_” tell postfix what it should do with the E-Mails it has to deliver to the outside world. The options that start with “smtpd_” tell postfix under what conditions it should accept the E-Mails from your clients for delivery. Now a brief overview of the most important options:

• smtp_sender_dependent_authentication: this Option tells Postfix if it should deliver E-Mails outgoing based on the sender. You need to set this to yes or the server will send your outgoing Mail only to your primary smart host.
• sender_dependent_relayhost_maps: with this options you tell postfix where to find the file that links your E-Mail addresses and Domains with the corresponding smart hosts.
• smtp_sasl_auth_enable: Set this to yes in order to tell postfix, that the smart hosts want a username and password.
• smtp_sasl_password_maps: This tells postfix where the file with the login information for your smart hosts is.
• smtp_sasl_: these options tell postfix how to log into the smart hosts. The given examples should work in most cases, but they are not the most secure way to do it.
• relayhost: this is the smart host for your Primary E-Mail domain. All E-Mails from senders that are not listed in the relayhost_map file will be sent over this smart host.
• mynetworks: lists all networks that are considered local networks by postfix.
• inet_interfaces: Unless postfix should only listen to specific interfaces, setting this to all is fine.
• smtpd_sasl_: These options tell Postfix, if and how users sending E-Mail over your server should authenticate themselves. The settings shown in the file, allow your users to use the credentials used in dovecot. This way you will only have to make changes in dovecot, if you wish to change the user backend.
• smtpd_recipient_restrictions: Tells postfix who to accept mail from. In the given configuration it accepts E-Mail from the networks configured in mynetworks and from users who authenticated themselves with their username and password. If your local networks are not secure, you should probably only allow E-Mails from authenticated users. Otherwise your Server might end up as a spambot.

The configuration links to the file /etc/postfix/relayhost_map, this file has to be created by you. As mentioned earlier, this file contains the mappings for maildomains and their corresponding smart hosts. If you only have a single E-Mail Address that needs to be rerouted, use the example from the first line. In order to reroute E-Mails from an entire domain use the 2nd line. If the SMTP server is a server of a free mail provider you might have to use the submission port instead of the normal SMTP port.

#if the mail provider requires you to use the submission port for outgoing mail

#add :submission to the end of the server address usermailaddress [at] maildomain1 [dot] com smtp.maildomain1.com @maildomain2.com mail.maildomain2.com

After creating the file, you need to transform it into something that postfix can use. Do so by calling the postmap command. This needs to be repeated whenever there is a change to the file:

postmap hash:/etc/postfix/relayhost_map

Now postfix only needs a file to tell it how to authenticate to all those smtp servers. This is done with the file linked under smtp_sasl_password_maps in the main.cf. The contents of this file should look something like this:

#smtp sender dependent auth
usermailaddress [at] maildomain1 [dot] com    username:password
@maildomain2.com    username:password

#relayhost login
smtp.maildomain.com username:password

As with the last file you need to run the postmap command on this one as well. And since this is the last file for postfix you need to change, you should restart postfix after running the postmap command.

postmap hash:/etc/postfix/sasl_passwd
/etc/init.d/postfix restart

This concludes the configuration of postfix. If you only need a sender based E-Mail relay, you are done here. The rest is about getting you providers E-Mails into your own IMAP Server.

I also want to fetch the E-Mails from my providers mailbox. What do I do?

Lets start with setting up Dovecot. Dovecot is the IMAP in all this. If you have used Dovecot before, you might notice that the configuration looks a little different in Debian 7.0 wheezy. This is because wheezy uses Dovecot 2.x and they spread the configuration of Dovecot from one large file to several smaller ones. Each of the new configuration files controls one aspect of dovecot. This makes it easier for beginners to get everything running. Since the Dovecot configuration files are still pretty long, I will only show you the changes that I made to every file. The first file that needs changing is “/etc/dovecot/conf.d/10-master”:

service imap-login {
  inet_listener imap {
     port = 143
  }

service auth {

  unix_listener /var/spool/postfix/private/auth {
    mode = 0666
  }
}

These changes tell dovecot to have its IMAP server listening on the default IMAP Port and the second part there is needed if postfix uses Dovecot for authentication.

The next file that you need to edit is: “/etc/dovecot/conf.d/10-mail”. In this one you need to change the mail location, if you do not want to use the default. In my case I wanted to use maildir storage under /var/mail/username. So I changed the mail_location line to this:

mail_location = maildir:/var/mail/%n

The last file you might have to change is “/etc/dovecot/conf.d/10-auth”. Since this is just for a quick test setup I enabled plaintext authentication. You should probably try to avoid using it for security reasons under normal circumstances.:

disable_plaintext_auth = no

Once all the config files have been changes, you will have to restart Dovecot or the changes won’t take effect.

/etc/init.d/dovecot restart

As you can see I did not actually tell you to enable to enable one of the many authentication backends dovecot support. That is because Dovecot uses the local system users for authentication out of the box. This is fine if the server is just for you or a very small company. If you expect some more users on this server, you might want to think about a different authentication backend.

Ok my server can give me IMAP mailboxes now, but where are my E-Mails?

You still have to get you mails from your provider and that is what Fetchmail is for. The global configuration file for Fetchmail is “/etc/fetchmailrc”. This file does not exist after installing it, so you need to create it. While you are at it you should also make sure that it is only readable as root, since the file contains the login information for your E-Mail Accounts.

touch /etc/fetchmailrc
chmod 600 /etc/fetchmailrc

After you created the file you need to give it a few parameters for the deamon and configure all the E-Mail Accounts you want to retrieve:

set daemon 300
set syslog

poll pop.maildomain1.com protocol pop3:
    username "user1" password "password", is systemuser1 here
    #keep    
    #fetchall
    #use procmail to deliver local mail so we dont need to hassle postfix
    mda "/usr/bin/procmail -f %F -d %T"

poll mail.maildomain2.com protocol    pop3:
    username "user2"    password "password" is systemuser2 here
    #keep
    mda "/usr/bin/procmail -f %F -d %T"

This example tells Fetchmail to download the mail from the listed mail servers via pop3 and then to deliver them to the respective system users via Procmail. The set daemon option is tells Fetchmail how often it should run in seconds and the set syslog tells it to write its log in the syslog.
I suggest to uncomment the keep option when you first test the system, since it tells Fetchmail not to delete the E-Mails on your mail providers servers. Once everything is running fine, you can remove the keep option, unless you want to leave copies of the E-Mails on the servers of your provider. You might get trouble with the size of your mailbox, if you let the keep option commented in.

After you prepared the configuration file, you still need to tell Fetchmail to run as daemon. You can do that by adding the line “START_DAEMON=yes” to the file  “/etc/default/fetchmail”

You said I need Procmail?

Procmail is the last piece of the puzzle. I suggest Procmail for the delivery of the E-Mails, because it comes with the system and using Procmail means, you don’t have to set up Postfix to deliver E-Mails via Dovecot. This is definitely the quick and dirty option for this kind of server it is enough.

The configuration of Procmail is fairly similar to Fetchmail. It is also done in a global configuration file called “/etc/procmailrc”. This file has to be created by you as well.

SHELL="/bin/bash"
LOGFILE="/var/log/procmail.log"
# / at the end of default needed for maildir
#needs to be the mail location of dovecot

MAILDIR=/var/mail
DEFAULT=/$MAILDIR/$LOGNAME/

:0

My Example file is very simple. It defines the location of the logfile and the target for the E-Mail delivery. The one thing that you need to watch is, that the DEFAULT line has to end with an “/” if your dovecot uses maildir and naturally the MAILDIR needs to be the base mail directory of dovecot.

Ok I followed it all. Is there anything else I have to do.

No if you followed this tutorial you should have a working mail relay and IMAP server. But this is a fairly basic setup and you should probably consider using encrypted connections.
This server is also extendable, you could add spam and virus filtering for your E-Mails or if you receive large amounts of mail you may want to look a little closer at Procmail. Procmail can be used to categorize certain mails into subfolders of your inbox. If you expect a larger amount of users on your server and possibly some fluctuation, you may want to look at different user backends.

The great thing about this system is that it can be extended with all those functions. It can grow with your needs.

Preparing the Server

Before I could start installing a new Horde, I had to uninstall the old one. The easiest way to do that is using this command from the horde wiki:

pear uninstall `pear list -c horde | tail -n +4 | awk '{ print "horde/"$1 }'`
mv horde horde_bk

After getting rid of my old Horde Installation, I moved the old horde folder. I figured it might be a good idea to still have those config files around, if I need to look something up.
With all the uninstalling out of the way, I could start a fresh installation of Horde5. Since I had a Horde Server Running before, I was sure that my Server had all the requirements of Horde installed/configured.  If you need to look them up, the Horde wiki is a good place to do that. But the Basic requirements are PHP, a database (MySQL in my case) and a working mail server with IMAP or POP3 support.

Preparing the Database

First order of business was getting a new MySQL Database ready for Horde. Horde creates its database tables later in the installation, so all it needs prepared is an empty Database and an user to connect to it. I logged onto my MySQL Server and prepared this with the following two commands in the MySQL console:

CREATE database horde;
GRANT ALL ON horde.* TO 'horde'@'hostip' IDENTIFIED BY 'password';

The GRANT command creates a new user that is allowed to connect from the host ‘hostip’ with the password ‘password’. I use IP Addresses to specify the hosts allowed to connect, but DNS names should work fine as well. The Database should have utf-8 as charset, if your MySQL does not use that by default, you can change it with the command “ALTER DATABASE horde charset=utf8;”. The next task was the actual installation of horde.

Installing Horde

For that I logged back onto my web server and ran the following four commands:

pear install horde/horde_role
pear run-scripts horde/horde_role
pear install -a -B horde/webmail
webmail-install

The when you run the horde_role script, you will be asked for a installation directory. This should be a directory that is served by your web server. For instance “/var/www/webmail”.
The last command starts a small script, that does the basic configuration of horde by asking a bunch of questions. In the next segment I list all the questions, the answers I typed in (bold) and my comments (green). Please note that the script will also tell you all possible answers.

• 1st Question: What database backend should we use: mysql
  Other options are false, mysqli, pgsql and sqlite
• 2nd Question: Request persistent connections: 0
  1=Yes and 0=No. If you don’t have a good reason to turn them on leave the default of 0.
• 3rd Question: Username to connect to the database as: horde
  The user was created after the MySQL database.
• 4th Question: Password to connect with: password
  The password used to identify the MySQL user.
• 5th Question: How should we connect to the database: tcp
  Since my database runs on a different server I choose tcp. If your database runs on the same server as horde, you should use unix sockets by typing ‘unix’
• 6th Question: Database server/host: mysqlserver.yourdomain.com
  The FQDN or IP Address of the MySQL server.
• 7th Question: Port the DB is running on, if non-standard: press Enter
  The standard Port 3306 will be used if you just press Enter. If your MySQL server listens on a different port, you need to specify that port here.
• 8th Question Database name to use: horde
  The name of the Database that was created earlier
• 9th Question: Internally used charset: press Enter
  The standard is utf-8. This setting should match the charset of the database to avoid errors.
• 10th Question: use SSL to connect to the server: 0
  1=Yes and 0=No. I switched it off because I am the only user on my local network and ok with the additional security risk. In most cases this should be on though.
• 11th Question: Certification Authority to use for SSL connections: press Enter
  I pressed Enter because I do not use SSL connections. you have to specify a CA here if you plan to use SSL connections. Please note that outside connections are still handled via SSL because horde is a part of my apache ssl configuration.
• 12th Question: Split reads to a different server?: press Enter
  The options are false (disabled) and true (enabled), with false being the standard choice. I do not have multiple horde servers and also no where near the load for that option to be needed.
• 13th Question: Specify an existing mail user who you want to give administrator permissions (optional): hordeadmin
  The username of the user that can access the admin panel in horde.

Basic Authentication via IMAP

My fresh Horde install wanted to authenticate against a mail server as default, so I copied back my old backends configuration. It is recommended, that you do not change the original backends.php file in the imp/config directory. Instead you should copy the backends.php to backends.local.php and and make your changes there. I disabled the default IMAP backend and enabled the secure-imap backend. There is already an example for the secure IMAP backend in the file. With my changes to enable it, the configuration for the secure IMAP backend looks like this:

$servers['secure-imap'] = array(
    // Disabled by default
    'disabled' => false,
    'name' => 'yourdomain.com',
    'hostspec' => 'mail.yourdomain.com',
    'hordeauth' => full,
    'protocol' => 'imap',
    'port' => 993,
    'secure' => 'ssl',
    'maildomain' => 'yourdomain.com',
    // 'smtphost' => 'mail.yourdomain.com',
    // 'smtpport' => 25,
    'acl' => true,
    'cache' => false,
);

Testing the Installation

Now would be a good time to test your installation by accessing your horde on a browser and calling the test.php. If you installed horde in the location webmail without SSL, you can access the page with the address “http://yourdomain.com/webmail/test.php”.
Before you can do this however, you will have to enable this script in the Horde configuration. To enable the test script log into horde as admin user and go to Administration –> Configuration and click on horde in the list of applications. Now deselect “$conf[testdisable]” in the PHP Settings and click Generate Horde Configuration.
If you can not login or if there is some other error, you can also enable the test script on the command line. To do that you need to edit the config.php file in the config directory of your horde installation. Look for the line “$conf[‘testdisable’] = true;” and replace the true with a false. After saving the test script is enabled. The line should be close to the beginning of the file.

Configuring the Active Directory Authentication

With the IMP Backend in place, horde is functional and since my mail server authenticates against my Active Directory it is actually usable at this point. But I wanted to use horde to Authenticate against my Active Directory on its own.
The LDAP configuration can be done as Horde administrator from the Web Interface. Start by going to the Horde administrator panel and then click Horde. Here click the LDAP tab to set up a LDAP backend for Horde. Here you have to decide if you want to be able to do some user management with Horde or if you simply want to use LDAP for authentication.
I choose to go with the management path:
The user you configure as binddn should have the right to create and edit users and groups in your Active Directory. If you select a user without those privileges, the authentication will still work, but you will not be able to create or change users within Horde. As you can see the fields given are pretty standard for LADP authentication, this is a good thing because it seems to me that the Horde documentation for this is a little outdated.

Like the LDAP Settings most of these Settings are fairly straight forward, but I did encounter a problem with one Option. Unlike any other LDAP authentication I have used so far, you can not specify the Base DN as “DC=yourdomain,DC=com”. This will cause a fatal error when trying to log in. You need to specify at least one OU as well. After I changed this setting to a form that it can work with like “OU=users,DC=yourdomain,DC=com”, everything worked as it was supposed to. Unfortunately I could not find this mentioned anywhere in the Horde wiki.
As said in the picture, the LDAP filter can be used to restrict the access to only certain users in your directory. In my case that is only users that are classified as Persons and in the group webmail.  So my filter ends up as (the objectClass=Person in the Screenshot is still from testing and while it works, it is not optimal):

(&(objectClass=user)(objectCategory=person)(memberOf=cn=webmail,ou=groups,dc=yourdomain,dc=com))

Depending on how many users you have and how often they change, it might also be a good idea to implement additional restrictions. One very useful addition could be excluding deactivated user accounts like this.:

(&(objectClass=user)(objectCategory=person)(memberOf=cn=webmail,ou=groups,dc=yourdomain,dc=com)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))

More useful Active Directory LDAP Filter options can be found in the Microsoft Technet wiki.
As a warning I should mention that I did not find any documentation (mostly because I did not bother looking for it further than the Horde wiki site I was on at the time), which object Classes a new user in Active Directory needs. So I am not entirely sure what to fill in there and I doubt that it is a good idea to create new users with the horde web interface using the settings in the screen shot.

Making sure Horde can send E-Mails

Now that the Authentication worked, I still had to do some general configuration changes but I did not encounter any problems there. The important Settings are:

Since Horde is used as webmailer, it should be able to send E-Mails. To enable Horde to do so, you need to go to the Horde configuration in the Mailer tab and fill in the information for your Mail Server.

Enabeling Hordes Cache

Navigate to the Cache System Tab of the Horde configuration and enable CSS Caching, Javascript Caching and Theme Caching, in order the ensure better performance for Horde.

Allow Horde to show HTML E-Mails in the web frontend

Next I wanted to enable the display of HTML mails in the web interface. This is disabled by default, because it is somewhat of a security risk. The additional convenience is worth the risk for me though. First I went into the folder horde –> imp –> config and copied the file “mime-drivers.php” to mine-drivers.local.php. Afterwards I opened the file in an editor and change the line “ ‘inline’ => false” to “ ‘inline’ => true” in the html section of the file. The section looks like this now:

    /* HTML driver settings */
    'html' => array(
        /* NOTE: Inline HTML display is turned OFF by default. */
        'inline' => true,
        'handles' => array(
            'text/html'
        ),
        'icons' => array(
            'default' => 'html.png'
        ),

        'limit_inline_size' => 1048576,

        /* Check for phishing exploits? */
        'phishing_check' => true
    ),

Enabling Hordes Alarm System

Now I had to enable the alarms. This was a fairly easy process. The following command created a crontab for my www-data user and opened it for editing:

crontab -u www-data -e

I used the editor to add the cron job by adding following line to then end of the file:

#Horde Alarms
*/5 * * * * /usr/bin/horde-alarms

Setting up ActiveSync

Finally it was time to enable some advanced functionality. One of my main reasons for originally choosing Horde was the ActiveSync support. ActiveSync is the functionality used by Microsoft Exchange Servers to Synchronise E-Mails, Contacts and Calendar Data to mobile devices. While I personally use IMAP for receiving my E-Mails from my Server and owncloud to sync my calendar and contacts, I still believe ActiveSync is a nice feature to have.
To get ActiveSync working, you need an Alias for /Microsoft-Server-Active-Sync in your webserver, that points to the remote.php of Horde. I still had that prepared from my previous Horde Installation, but you can create the alias by adding following lines to the configuration of Hordes virtual host in Apache:

# Enable Active Sync for Horde
Alias /Microsoft-Server-ActiveSync /var/www/webmail/rpc.php

After creating the Alias you should reload the Apache configuration. Now all that was left was enabling ActiveSync in the Horde configuration

Using the Active Directory as Global Address Book in Horde

Finally I wanted my users to have a Global Address Book, much like the one you would have with a Microsoft Exchange Server. Horde has the functionality for this prepared, but it is not activated by default.
In order to activate it, go into the horde->turba->config directory on your horde server. Once there copy the file “backends.php” to “backends.local.php” and edit the new file. I listed the relevant part and what it should look like below:

/**
 * A local address book in an LDAP directory. This implements a public
 * (shared) address book.
 *
 * To store distribution lists in the LDAP directory, you'll need to include
 * horde.schema from Horde in your LDAP configuration.
 *
 * To store freebusy information in the LDAP directory, you'll need to include
 * rfc2739.schema from Horde in your LDAP configuration.
 */
$cfgSources['localldap'] = array(
    // Disabled by default
    'disabled' => false,
    'title' => _("yourdomain.com Global Addresses"),
    'type' => 'ldap',
    'params' => array(
        'server' => 'ad.yourdomain.com',
        'port' => 389,
        'tls' => false,
        'root' => 'ou=users,dc=yourdomain,dc=com',
        'bind_dn' => 'YOURDOMAIN\systemuser',
        // For Active Directory:
        // 'bind_dn' => 'username [at] example [dot] com',
        'bind_password' => 'password',
        'sizelimit' => 0,
        // For Active Directory:
        // 'sizelimit' => 0,
        'dn' => array('cn'),
        'objectclass' => array('organizationalPerson',
                               'user',
                               'group',
                               'contact'),
                               // Add 'turbaContact' to this array if using
                               // 'turbaType' attribute below, and 'calEntry'
                               // if using 'freebusyUrl'.
        // For Active Directory:
        // 'objectclass' => array('organizationalPerson',
        //                        'user',
        //                        'group',
        //                        'contact'),
        'scope' => 'sub',
        // For Active Directory:
        // 'scope' => 'sub',
        'charset' => 'utf-8',
        // Consult the LDAP schema to verify that all required attributes for
        // an entry are set and add them if needed.
        'checkrequired' => false,
        // Value used to fill in missing required attributes.
        'checkrequired_string' => ' ',
        // Check LDAP schema for valid syntax. If this is false an address
        // field is assumed to have postalAddress syntax; otherwise the schema
        // is consulted for the syntax to use.
        'checksyntax' => false,
        'version' => 3,

        // For Active Directory you probably want to also set the following
        // parameters:
        'deref' => LDAP_DEREF_ALWAYS,
        'filter' => '&(SAMAccountName=*)(mail=*)',
        'referrals' => 0,
    ),
    'map' => array(
        '__key' => 'dn',

        // Remove this mapping if using Active Directory server:
        //'__uid' => 'uid',

        // From horde.schema.  Make sure you have 'turbaContact' objectClass
        // included above:
        // '__type' => 'turbaType',
        // '__members' => 'turbaMembers',

        //'name' => 'cn',
     'name' => 'displayname',        
     'email' => 'mail',
        'homePhone' => 'homephone',
        'workPhone' => 'telephonenumber',
        'cellPhone' => 'mobiletelephonenumber',
        'homeAddress' => 'homepostaladdress',

        // From rfc2739.schema:
        // 'freebusyUrl' => 'calFBURL',

        // For Active Directory servers:
        //'name' => 'displayname',
        'title' => 'title',
        'cellPhone' => 'mobile',
        'department' => 'department',
        'company' => 'company',
    ),
    'search' => array(
        'name',
        'email',
        'homePhone',
        'workPhone',
        'cellPhone',
        'homeAddress'
    ),
    'strict' => array(
        'dn', 'uid'
    ),
    'approximate' => array(
        'cn',
    ),
    // For Active Directory servers:
    // 'approximate' => array(
    //     'displayname',
    //     'samaccountname',
    // ),
    'export' => true,
    'browse' => true,
);

Basically you just have to set the ‘disabled’ parameter to false and fill in the information of your Active Directory. The one thing that has to be watched is the root, because it will not work with the example given in the file. Just like with the Authentication, Horde needs a root of “ou=something,dc=yourdomain,dc=com”. If you use the “dc=yourdomain,dc=com” configuration of the example file, the Address Book will not work.
And since you are using Active Directory, you will have to comment in or replace the settings with settings appropriate for Active Directory wherever the file tells you to.

With all that done Horde was ready to use. If you have any questions about this leave a comment or contact me via E-Mail.