Crux

I’m tired of managing different virtual machines when I can use hierarchial jails to achieve my requirement of separation and isolation. After all, using virtual machines while using same FreeBSD OS for different tasks plus having ZFS on every VM is pointless. I’ll stop ranting. Let’s get to work.

I want to migrate each of my virtual machines to a separate jail for each of them and I’ll use hierarchial jails in order to migrate jails on those VMs.

Network

I was using standard switches (A.K.A bridge) on my hypervisor with uplinks of my server. So in order to make things easier, I want to use JIB to manage each of the jails interfaces under its required uplink. In order to accomplish that, I have to use VNET to virtualize network stack.

Install JIB

It’s in the base, so:

install -o root -g wheel -m 0555 /usr/share/examples/jails/jib /usr/local/sbin/jib

Firewall

I will use the pf firewall for my jails. Of course we have to deal with devfs and bridge for jib in order to make this happen.

DevFS Rules

If you do not want to use ZFS, there is no need for adding new rules. Use can use the devfsrules_jail_vnet=5 which is already configured by default. However, I want my jails to have access to zfs in order to create datasets. So I will create new devfs rules at /etc/devfs.rules.

[jail_vnet_zfs=6]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_jail
add include $devfsrules_jail_vnet
add path zfs unhide

Use this under your jail configuration:

  # unhide pf
  mount.devfs;
  devfs_ruleset = 6;
Bridges

I do not want the pf to care about any bridge interfaces in my network. So we have to tell it somehow. Add these configuration to your /etc/sysctl.conf and apply it using sysctl -f /etc/sysctl.conf.

net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_member=0

Storage and ZFS

As the matter of fact, We mostly like the snapshot ability on the hypervisors and that is one of the main reasons that we put EVERYTHING under a separate VM. Then we realize ZFS under FreeBSD bring us this ability perfectly. As well as, It’s ZFS, you probably can name many other reasons why use the ZFS for each of the jails.

I will create a separate dataset for jails directory of my server, and then for root directory of each jails.

zfs create -o mountpoint=/usr/jails tank/jails
zfs create -p tank/jails/vm1/usr

Jailed ZFS and VNET

Attach the ZFS dataset under your jails and create your vnet interfaces. This is how i do it:

  vnet;
  vnet.interface = "e0b_${name}";

  exec.prestart += "jib addm ${name} $pif";
  exec.poststart += "zfs set jailed=on tank/jails/${name}/usr || echo ZFS already jailed";
  exec.poststart += "zfs jail ${name} tank/jails/${name}/usr";
  exec.poststart += "jexec ${name} zfs mount -R tank/jails/${name}/usr";
  exec.poststart += "jexec ${name} zfs list -H -o name -r tank/jails/${name}/usr | xargs -I '{}' jexec ${name} zfs mount {} || echo ZFS already mounted";
  exec.poststart += "service -j ${name} jail onestart";
  exec.prestop  += "zfs unjail ${name} tank/jails/${name}/usr || echo zfs dataset is already unjailed";
  exec.poststop += "jib destroy ${name}";

Jail configurations of the host

I do not like the jail managers everytime I do not have the CI/CD pipelines. In like manner, I found it easier to use the jail.conf.d itself in the base system. Remamber to replace $pif variable with your own uplink interface of the host. Simply create the base jail configuration at /etc/jail.conf:

path = "/usr/jails/${name}";
exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.clean;

# FQDN
host.hostname = "${name}.spmzt.net";
host.domainname = "spmzt.net";

# Global Variables
$pif="vmx0";

Then create the configuration of your jails in the separate files under /etc/jail.conf.d/:

.include "/etc/jail.conf";
mayuri {
  persist;

  # unhide pf
  mount.devfs;
  devfs_ruleset = 5;

  # parent permissions
  children.max = 12;
  allow.mount.devfs;

  # required by zfs jailed
  allow.mount;
  allow.mount.zfs;
  enforce_statfs=0;

  # parent permission (required by netflow grandchild)
  allow.sysvipc;

  # parent permission (required by ocserv1 grandchild)
  allow.mount.procfs;

  # decrease securelevel to be able to mount
  securelevel=0;

  # Network Configurations
  vnet;
  vnet.interface = "e0b_${name}";

  exec.prestart += "jib addm ${name} $pif";
  exec.poststart += "zfs set jailed=on tank/jails/${name}/usr || echo ZFS already jailed";
  exec.poststart += "zfs jail ${name} tank/jails/${name}/usr";
  exec.poststart += "jexec ${name} zfs mount -R tank/jails/${name}/usr";
  exec.poststart += "jexec ${name} zfs list -H -o name -r tank/jails/${name}/usr | xargs -I '{}' jexec ${name} zfs mount {} || echo ZFS already mounted";
  exec.poststart += "service -j ${name} jail onestart";
  exec.prestop  += "zfs unjail ${name} tank/jails/${name}/usr || echo zfs dataset is already unjailed";
  exec.poststop += "jib destroy ${name}";

Note

You need the command below in your jail configuration to start child jails.

  exec.poststart += "service -j ${name} jail onestart";

Kernel Modules

I need some kernel modules inside my virtual machine when I want to use them as jails inside my host. I add to add those kernel modules inside the host itself.

sysrc kld_list=+"netgraph ng_ether ng_tee ng_socket ng_netflow ng_ksocket ng_eiface ng_iface ng_bridge ipsec"