Home Server

My first home server back in 2014, was an old cast-off supermicro which had been used as a router, for many years at work. I hand-crafted into LXC containers on a 64GB SSD and a 1.5TB spinning rust drive (also rejects from work) to install various services on. The server lasted a few years until it's SSD died - Hands up if you saw that coming... Luckily anything important was stored on the spinning rust drive.

Writing this post reminds me of the fun I had with that machine - including the times when I came home in the summer and it was strangely quiet - because it's cooling fan had ceased. Ah, they were the days.

When it came to replacing the server, I remember wishing I had notes about how I set services up, what decisions had I made during the hand-crafting process years before.

Hardware

Continuing the trend of using cast-off hardware, I was given a surplus workstation which a colleague had taken home some time before. So it's been a cast off twice over!

For those playing along at home, it is an HP ProLiant ML110 G6. Among it's stunning features, it includes:

• a Intel G6950 - two cores running at 2.8GHz
• 16GB of DDR3 memory
• 2x 128GB SSDs in a RAID 1 configuration for the OS and containers
• a Dell PERC H200 (a Dell-branded LSI Logic hardware raid controller)
• 4x 2TB SAS spinning rust drives arranged in RAID 10 configuration

Perhaps one day I'll follow the hardware set up posted by people online who have 5 to 10 u of rack-mounted servers. Presumably costing quite a sum to purchase, and equally scary cost to power.

Configuration

Following my revelation that it might be good to have some sort of documentation about how a service was set up, I looked into options for some sort of configuration management. I had a play with Puppet and Chef for a while, but in the end decided to use Ansible as it was the tool that I'd be using with a change of role at work, so it seemed like a good way to learn it.

Networks

I think I may have gone overboard with my home network set up. There are three main networks, each NATed to my public IP address:

• LXD - the network that all LXD containers on this machine are added to
• Home - the main home network used by the phones and computers of the people living in the house
• IoT - a somewhat untrusted network for IoT devices which I don't necessarily trust being on the main network

As these networks go through the same switch I have three VLANs set up:

• one for the ISP provided modem
• another for the home network
• and the last for the IoT network

My wireless access point supports assigning a VLAN to SSIDs.

As well as these networks, I also use WireGuard for a couple of VPNs:

• A management VPN to allow my workstations to connect to the containers on various machines
• An inter-server private network to allow remote machines to communicate privately over the public internet

Overall network config is defined in a variable in my Ansible plays.

LXD containers

The OS is currently Ubuntu 18.04, and is carved into several LXD containers, one per service. Services I currently run on this machine:

• Bitwarden RS - A Rust implementation of the Bitwarden password manager - read the blog post about switching from Bitwarden to Bitwarden RS
• A DHCP server
• DNSS - DNS over HTTPS
• Drone CI - A continuous integration service which integrates nicely with Gitea
• Gitea - Self-hosted Git service (think a self-hosted Github or a light-weight Gitlab)
• Grafana - Pretty monitoring dashboards
• Go Graphite - An implementation of the Graphite Time Series DataBase (TSDB) written in Go
• Home Assistant - Open source home automation
• Jellyfin - A self-hosted media system, forked from Emby when Emby changed it's licence and closed it's source - Read about the Electron app I wrote for Jellyfin
• MySQL - You've heard of MySQL, right?
• Nextcloud - A self-hosted cloud system - I mostly use it to exfiltrate photos from my phone to my home server
• Nginx - while there are a few instances of Nginx on the server, this container acts as a reverse proxy for the other services to be available outside the home network using Let's Encrypt for TLS certificates
• Pi-hole - A network-wide DNS-based ad-blocking service
• Samba - Exports filesystem directories natively supported by Windows, macOS and Linux
• Sensu Go - A monitoring platform to schedule checks and metrics of my infrastructure

Why not docker?

When I starting working on the Ansible for my home server, I toyed with deploying services using docker containers. The main reason I didn't continue down this path was because I disliked the apparent ease of grabbing the docker image and running it. It seems that the common process is to search dockerhub for the service you want to run and just pick the first image (though the method to run the containers varies - docker compose, k3s, Kubernetes, etc). My main dislike of this pattern is that although it's great when it's easy to get your services up and running, it seemed as though I wouldn't get an understanding of how the service hung together this way. If a service broke, I'd be a step behind being able to diagnose and fix the problem.

I'm also not a huge fan of the Dockerfile format, especially after using Ansible. While it is possible to build docker containers using Ansible using tools like Packer (and in fact, I've used it to build some docker containers at work) I didn't want to have a nightly docker container building infrastructure set up to be able to run my services to keep on top of security updates.

The last issue I took with docker (a fix for which I understand is in the pipeline) is the lack of UID mapping for containers. Root in a docker container is effectively root on the host, while with LXD, UIDs and GIDs are mapped to higher ranges so root in an LXD container is actually UID 100000 on the host.

Having said all this, a few years down the line, I'm far less against the idea. Perhaps my next home server iteration will be docker-based, whether that's docker compose, k3s or some other setup.

Ansible

It has been great using Ansible to configure the server, it means that I can rebuild services very quickly, or run a quick Ansible playbook to update config of several machines at the same time.

For example; I realised that I'd left my Nginx TLS config at its default, so my two reverse proxies accepted TLS 1.0 connections (I don't take any card details through either of them so it doesn't really matter). I updated my Nginx Ansible role to specify the TLS settings, then run the playbook, and all instances of Nginx that use TLS are updated. Of course I could have updated the settings manually - it's only two servers, right? It would have involved updating 18 virtual host files - not to mention remembering to add them next time.

Though, really the benefit of Ansible is more obvious at larger scales. At work, I manage maybe 500 hosts, and about 400 of those have some version of MySQL installed on them. We run quite an optimised stack, so occasionally I'll find a setting to tune to improve security, performance or reliability or whatever. Instead of logging into each of the 400 machines individually to tweak the setting one by one - boring! - I would update the MySQL Ansible role to make the change (conditionally on other factors if applicable) and run the update through some sort of test-stage-live process.

The original idea for my home setup was to consider each container to be ephemeral, so the container could be deleted and another quickly stood in its place and configured in the same way. All Ansible tasks are written to update packages where available (unless a specific version is required) to make a complete rebuild equivalent to updating the existing setup. Unattended upgrades covers package updates between Ansible playbook runs where possible. Monitoring is set up for packages which can't be updated through unattended upgrades.

If I want to run a service I don't already have an Ansible play for, I'll write one. On the whole, it's just Ansible-ifying the installation instructions for whatever service I want to run. This is usually installing a package, writing config file, then starting a service. I will often take the default config file, copy it into a template in the ansible play and edit it there. I don't bother supporting all the various features of the service to be configured using Ansible unless I think it may be beneficial - it can always be added a the point of use at a later time.

Example service setup

For example, say I want to add another Samba instance, this time accessible to the IoT network for some reason.

First of all, I would assign the IP addresses in the network config in my Ansible inventory.

Next, I can add an item to the lxdhost_containers list to have the Ansible play create the new container on the right networks:

lxdhost_containers:
  - name: iot-samba
    network_config:
      - address: "{{ _networks.home_lxdbr.ips['iot-samba.example.com'] }}"
      - address: "{{ _networks.iotbr.ips['iot-samba.example.com'] }}"
        netmask: 255.255.255.0
        parent: "{{ _iot_bridge }}"
        gateway: false
    volumes:
      - name: iot-data
        path: /iot-data
        source: /data/iot-data

Running the lxdhost Ansible play against the home server with this config will create a new container called iot-samba. It'll be given two NICs and given the pre-configured static IPs. It will also have the /data/iot-data directory mounted into it at /iot-data. Though the use of cloud-init, the new container already has python installed, an admin user with authorised SSH public keys pre-configured.

Next, I can write an inventory file for this new machine to indicate it should have the samba Ansible role run against it, configure samba and the firewall ready for use (several other roles are always run against this type of machine):

groups:
  - samba

vars:
  firewall_open_ports:
    - name: samba
      interface: eth1
      protocol: udp
      port: 137
    - name: samba
      interface: eth1
      protocol: udp
      port: 138
    - name: samba
      interface: eth1
      protocol: tcp
      port: 139
    - name: samba
      interface: eth1
      protocol: tcp
      port: 445

  samba_users:
    - name: mydevice

  samba_shares:
    - name: iot-data
      description: IoT Data
      path: /iot-data
      valid_users: [mydevice]

The above config allows UDP ports 137 and 138, and TCP on ports 139 and 445 to be accessed on the eth1 interface (eth0 is the LXD network interface). A user will be created called mydevice, and a random password generated for it. It is only able to login over Samba.

Once the Ansible playbook has been run, the samba server will be configured and running.

Exposing services

To expose HTTP(S) services to the world, I have configured port-forwards on the host for ports 80 and 443 to my proxy container.

firewall_enable_forwarding: true
firewall_port_forwards:
  - name: proxy http
    in_interface: "{{ _isp_vlan }}"
    in_port: 80
    to_host: "{{ _networks.home_lxdbr.ips['proxy.example.com'] }}"
    to_port: 80
  - name: proxy https
    in_interface: "{{ _isp_vlan }}"
    in_port: 443
    to_host: "{{ _networks.home_lxdbr.ips['proxy.example.com'] }}"
    to_port: 443

Then virtual hosts for the services can be generated on the proxy container, and letsencrypt certificates will be configured:

nginx_use_letsencrypt: true
nginx_vhosts:
  - name: gitea.example.com
    state: present
    ssl_only: true
    server_name: gitea.example.com
    locations:
      - type: proxy
        backend: "http://{{ _networks.home_lxdbr.ips['gitea.example.com'] }}:3000/"

Backups

I use Restic to take nightly off-site backups.

Warts

Not everything is wonderful with my home server/network setup.

The worst offender that comes to mind is my lack of NAT loopback / hairpinning - where traffic originating from a NATed network destined for the server's public IP address does not end up going to the server. In theory, I should be able to add firewall rules to handle this traffic, but I have not found a way to fix this, and I've spent enough time on it.

Instead I added my reverse proxy container to my home network, and added DNS records to my Pihole server to hand out the home network IP of the reverse proxy container. This allows me to use the same hostname internally and not have to go though my NAT "properly". I hope to fix this one day, but now that it's not broken I suspect it may stay like this for a while.