When you are starving and do not know much about how to do programming with
systems, it is usually a welcome sight to get a setup and configuration job. I
got to play around with a DigitalOcean instance of FreeBSD to setup proof of
concept nginx(8) based load balancer[1] using FreeBSD jails[2].

So here is a description of the job

Configure a set of jails in my FreeBSD 11.x VPS.

Jail 1-4 contains applications with web interfaces / APIs.
Jail-5 is a webserver that proxies request to all of these.

Acceptance test / criteria:

On jail 1-4

Start a python -m SimpleHTTPserver on each jail and have each serve a
different text file indicating the jail#

On Jail #5

Setup nginx and point traffic in different subdomains (list of 5, will be
provided) to each jail the last one for itself.

Sounds simple enough, so I went ahead and accepted it.

Part I - Setting up the jails

After setting up the usual environment I work with I went ahead and installed
ezjail(8) for administering jails. One of the reasons for choosing ejzail(8) was
that there is nice documentation about it in the FreeBSD handbook[3].

Following the instructions there, I was able to get ezjail(8) up and
running. The one issue that I faced was it kept popping up the below warning

/etc/rc.d/jail: WARNING: Per-jail configuration via jail_* variables  is obsolete.  Please consider to migrate to /etc/jail.conf.

Since I was doing a proof of concept, I did not give too much attention to the
above warning.

  • Setting up a new interface lo1 to which Jail IPs will be assigned.

  • Assigned 192.168.0.1/24 as the network.

  • Configure pf.conf(5) so that jails could talk to the outside world

  # Allow outbound connections from within the jails
  nat on vtnet0 from lo1:network to any -> (vtnet0)

Once that was done, I created 5 jails with IPs 192.168.0.2-6. Jails with IP in
the range 192.168.0.2-5 will be the jails running the simple python based
SimpleHTTPServer and nginx(8) will runin Jail IP 192.168.0.6.

After this I made sure the jails could talk with each other and can talk
outside, for updating and setting up the necessary softwares. pkg(8) was pretty
slow for some reason, I think this could be some issue with DNS resolver, but I
did not investigate further, since I had enough time to setup things. One pkg(8)
boostrap was done, I setup python 2.7 in jails 1-4 and setup nginx(8) in jail 5.

In addition to this I also setup port forwading of 80 and 443 to the nginx(8) jail.

external_ip="11.22.33.44"

# nginx(8) jail at 192.168.0.6
rdr on vtnet0 proto tcp from any to $external_ip port 80  -> 192.168.0.6
rdr on vtnet0 proto tcp from any to $external_ip port 443 -> 192.168.0.6

I also had each of the jail from 192.168.0.2-5 run python -m SimpleHTTPserver
with a simple html file indicating the jail number.

Part II - Doing the load balancing

This was easier than expected and all I had to do was to follow the documentation at the NGINX website and the load balacing.

I removed the default config’s “http” block and replaced it with

http {
  upstream jail-load-balancer {
    server 192.168.0.2:80;
    server 192.168.0.3:80;
    server 192.168.0.4:80;
    server 192.168.0.5:80;
  }

  server {
    listen 192.168.0.6:80;
    server_11.22.33.44;
    location / {
      proxy_pass http://jail-load-balancer;
    }
  }
}

One of the important things to keep note here is that, the listen part of
their server block, when running in a jail should bind to the IP, that is
assigned to the jail, if no IP is specificed the normal working is that it binds
to localhost aka 127.0.0.1, but since we did not assign a local loopback address
to the jail, we have to explicitly mention the IP address we are binding
to. Else things may not work as expected.

Conclusion

At the end, it was nice to get paid a bit of money to pay the bills and get some
decent food for the month. And the work was something new I got to try out and learn.

References

  1. https://www.nginx.com/resources/wiki/start/topics/examples/loadbalanceexample/
  2. https://www.davd.eu/posts-freebsd-jails-with-a-single-public-ip-address/
  3. https://www.freebsd.org/doc/handbook/jails-ezjail.html