Most of my development work was being done in my local machine. Since I worked
with a variety of technologies over a period of time and depending on what I
worked on at the time it mostly involved heavy IDEs like Visual Studio .NET or
Eclipse which is hard to work with via a remote machine. Also I never dealt with
code bases big enough that it takes hours to build or had tens of thousands of
files scattered in deep folder structures, well at least not until I got into
BSD development.

Suddenly the code base was big, and I was not confident enough to switch to a
BSD flavor to do development and testing, which means I still need to rely on my
Windows 7 laptop to do my daily activities whilst I do the development inside a
virtual machine running the OS. NetBSD being what it is, atleast allowed me to
do build stuff from within the Cygwin environment. But the builds stretched the
system to it’s limits especially when doing a kernel build, high CPU load and
the rising temperatures were not helping. And using a Raspberry Pi 2 as a
development environment is not exactly comfortable.

Couple of friends of mine decided to rent out a bare-metal server from
Hetzner. The configuration was not too bad

Hardware Specifications

  • CPU - Intel Core i7-3770 CPU @ 3.40GHz (Ivy Bridge)
  • RAM - 32 GB DDR3-1333
  • Graphics Card - Intel HD Graphics 2500 (Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller)
  • Network Card - RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
  • HDD - Toshiba DT01ACA300 3TB x 2

The asking rent (per-month) for the system was not too expensive. Even though I
could not afford to split the pay at that point in time, I volunteered to help
do the sysadmin stuff and maintain the server.

The only condition I insisted was to install FreeBSD on the bare-metal. Though
my friend was a bit reluctant at first but I convinced him to have FreeBSD on
the bare-metal and I could configure an Ubuntu instance if needed via bhyve the
FreeBSD hypervisor if need be for his Linux needs. After some thought he went
ahead and decided to have a FreeBSD bare-metal.

The Install

Not as simple as it sounded. At the time FreeBSD 11.0-RELEASE was just out and
the image provided by Hetzner was not installing correctly, not a good start
especially when I convinced him to have FreeBSD installed because it is less
hassle :-P

Anyway I asked him to install FreeBSD 10.4-RELEASE and told him I can take care
of the upgrade process to FreeBSD 11.0-RELEASE. So once the install was done, I
did a freebsd-update and upgraded directly to FreeBSD 11.0-RELEASE before
proceeding to set up the rest of the system

# freebsd-update upgrade -r 11.0-RELEASE --currently-running 10.4-RELEASE

Once the upgrade was completed and the reboots done it was time to set up the
machine.

Setting up the basics

Once the system was running, we needed to have the basic services up and running

  • sshd - Your friendly SSH daemon
  • pf - The industry standard firewall
  • sshguard - To block off unwanted people knocking at server’s ports

Setting up the basic rules in pf(4) was not too hard and even for a first timer
like me the handbook[1] helped me through most of the rules. And I had a handy
set of basic rules from working with Trouble in his server.

After that we created some more non-root accounts for each of us using the server.

Dhananjay helped setup sshguard(8) and it’s pf(4) configuration.

In addition to the primary IP we also got a secondary IP, this was configured in
rc.conf(5)

Setting up the Jails

The first step was to configure the cloned interface to put the jail IPs on in rc.conf(5).

cloned_interfaces="lo1"
ifconfig_lo1="inet 192.168.1.1 netmask 255.255.255.0"

Then this would be NAT-ed out to the external interface via pf(4)

nat on re0 from $jail_net to any -> (re0)

Next I set up ezjail(5) for jail management, once again the FreeBSD
handbook[2] came in use.

I had a strange warning when starting up the jail service

/etc/rc.d/jail: WARNING: Per-jail configuration via jail_* variables  is obsolete.  Please consider to migrate to /etc/jail.conf.

Even though nothing was mentioned in the handbook about this warning, it seemed
that jails were working as expected. So I put it off to investigate another
time.

Setting up the bhyve VMs

After some looking around I felt that vm-bhyve(8)[3] is a good choice to manage
bhyve based virtual machines. Especially for an in-experienced person like me.

With simple to follow instructions I was able to setup 4 VMs

  • Ubuntu 16.04
  • FreeBSD 11
  • NetBSD 7.1
  • OpenBSD 6.2 (I just provisioned another friend is using it for his purposes)

The only addition was that I had to set up an if_bridge(4) so that the various
VMs could communicate via tap(4), dhcpd(8) was enabled so that automatic IP
assignment can take place over the bridge interface.

cloned_interfaces="${cloned_interfaces} bridge0"
ifconfig_bridge0="inet 172.16.0.1 netmask 255.255.255.0"
rtadvd_enable="YES"
rtadvd_interfaces="bridge0"

NOTE: The ${cloned_interfaces} variable is a nice way to physically separate
the various cloned_interfaces statements to their relevant sections. For
example jails vs bhyve VMs here.

Setting up DNS server

I went with unbound(8) as the DNS server and set it up referring the unbound
manual[4].

Luckily I also had a boiler plate configuration file once again thanks to
Trouble.

Conclusion

The overall experience was smooth and we got a basic server up and running. As
time passes I would like to shift all my kernel development and package
development stuff to the respectively NetBSD / FreeBSD VMs and do builds / tests
there.

Meanwhile another thing to consider is with so much resources to spare one could
host several useful services and I will need to look into these as I get more
time.

References

  1. https://www.freebsd.org/doc/handbook/firewalls-pf.html
  2. https://www.freebsd.org/doc/handbook/jails-ezjail.html
  3. https://github.com/churchers/vm-bhyve
  4. https://nlnetlabs.nl/documentation/unbound/howto-setup/