Route metrics are a fundamental networking concept that determine which path traffic takes when multiple routes to the same destination exist. FreeBSD CURRENT now has full metric support in the routing stack.
Let me walk you through how it works with a practical example.
A metric is a numeric value assigned to a route that influences route selection.
When you have multiple routes to the same destination, the kernel prefers the route with the lowest metric.
Think of it as a “cost”, lower cost = preferred path.
When you create a route without specifying a metric, FreeBSD assigns a default metric of 1.
We don’t allow you to create a route with metric of 0, That one is used for wildcard. You can delete all of the nexthops without specifiying route metric.
Its netlink behavior is also consistent.
Either you don’t have RTA_PRIORITY or you have one with metric of zero.
If you’re creating a route, it means you’re not specifying its metric, which means default metric (1). If you’re deleting a route, it means you don’t care about its metric, you want to delete all of them.
# Create a route without specifying metric
route -6 add -net 3fff:a::/64 -gateway 3fff::1
# The route automatically gets metric 1
route -n6 get 3fff:a::/64
netstat -rn6W
This is useful because it means all “equal” routes start with the same priority, and you can selectively add routes with higher metrics as fallbacks.
Let’s say you’re a network operator managing traffic to network 3fff:a::/64. You want:
You start by creating an Equal-Cost Multi-Path (ECMP) setup with two gateways at metric 10:
For the sake of this example, I want this ECMP route to be unbalanced.
% route -6 add -net 3fff:a::/64 -gateway 3fff::2 -metric 10
add net 3fff:a::/64: gateway 3fff::2 fib 0
% route -6 add -net 3fff:a::/64 -gateway 3fff::3 -weight 2 -metric 10
% route -on6 get 3fff:a::/64
route to: 3fff:a::
destination: 3fff:a::
mask: ffff:ffff:ffff:ffff::
fib: 0
flags: <UP,GATEWAY,DONE,STATIC>
nhops: 2
via gw 3fff::2 iface vtnet1 metric 10 weight 1 mtu 1500 table inet6.0
via gw 3fff::3 iface vtnet1 metric 10 weight 2 mtu 1500
The -weight 2 parameter indicates these are NOT equally balanced within the ECMP group.
Traffic to 3fff:a::/64 now load-balances between the two gateways.
See the value of Slots column:
% netstat -On6W
Nexthop groups data
Internet6:
GrpIdx NhIdx Weight Slots Gateway Netif Refcnt
14 ------- ------- ------- --------------------------- ----- 2
12 1 1 3fff::2 vtnet1
13 2 2 3fff::3 vtnet1
Current best route: 3fff::2 or 3fff::3 (metric 10) Backup: (none yet)
Next, you add a backup gateway with a higher metric (20), so it’s only used if the primary routes fail:
% route -6 add -net 3fff:a::/64 -gateway 3fff::1 -metric 20
add net 3fff:a::/64: gateway 3fff::1 fib 0
% # Verify the results
% route -on6 get 3fff:a::/64
route to: 3fff:a::
destination: 3fff:a::
mask: ffff:ffff:ffff:ffff::
fib: 0
flags: <UP,GATEWAY,DONE,STATIC>
nhops: 3
via gw 3fff::2 iface vtnet1 metric 10 weight 1 mtu 1500 table inet6.0
via gw 3fff::3 iface vtnet1 metric 10 weight 2 mtu 1500
via gw 3fff::1 iface vtnet1 metric 20 weight 1 mtu 1500
% netstat -On6W
Nexthop groups data
Internet6:
GrpIdx NhIdx Weight Slots Gateway Netif Refcnt
16 ------- ------- ------- --------------------------- ----- 2
12 1 0 3fff::2 vtnet1
13 2 2 3fff::3 vtnet1
15 1 0 3fff::1 vtnet1
The system now has three routes, but only uses the metric 10 paths:
Current best route: 3fff::2 or 3fff::3 (metric 10) Backup: 3fff::1 (metric 20), waiting in the RIB
If you try to add the same gateway with the same metric again, the system rejects it:
% route -6 add -net 3fff:a::/64 -gateway 3fff::1 -metric 20
route: message indicates error: File exists
add net 3fff:a::/64: gateway 3fff::1 fib 0: route already in table
This prevents accidental duplicates.
Now your new, faster gateway 3fff::1 becomes available. You can immediately promote it to metric 5, better than the current metric 20 routes:
% route -6 add -net 3fff:a::/64 -gateway 3fff::1 -metric 5
add net 3fff:a::/64: gateway 3fff::1 fib 0
% route -on6 get 3fff:a::/64
route to: 3fff:a::
destination: 3fff:a::
mask: ffff:ffff:ffff:ffff::
fib: 0
flags: <UP,GATEWAY,DONE,STATIC>
nhops: 4
via gw 3fff::2 iface vtnet1 metric 10 weight 1 mtu 1500 table inet6.0
via gw 3fff::3 iface vtnet1 metric 10 weight 2 mtu 1500
via gw 3fff::1 iface vtnet1 metric 5 weight 1 mtu 1500
via gw 3fff::1 iface vtnet1 metric 20 weight 1 mtu 1500
% netstat -On6W
Nexthop groups data
Internet6:
GrpIdx NhIdx Weight Slots Gateway Netif Refcnt
17 ------- ------- ------- -------------------------- ----- 2
12 1 0 3fff::2 vtnet1
13 2 0 3fff::3 vtnet1
14 1 1 3fff::1 vtnet1
15 1 0 3fff::1 vtnet1
The routing table now has four routes, but the best one changes:
Current best route: 3fff::1 (metric 5, NEW PRIMARY) Alternate: 3fff::2, 3fff::3 (metric 10) Backup: 3fff::1 (metric 20, This one is shadowed now)
The metric system also enables fine-grained route deletion. You can delete routes by:
After the gateway 3fff::1 at metric 5 has proven stable, you can remove the old metric 20 routes without affecting your ECMP setup:
% route -n6 delete -net 3fff:a::/64 -metric 20
delete net 3fff:a::/64 fib 0
% route -on6 get 3fff:a::/64
route to: 3fff:a::
destination: 3fff:a::
mask: ffff:ffff:ffff:ffff::
fib: 0
flags: <UP,GATEWAY,DONE,STATIC>
nhops: 3
via gw 3fff::2 iface vtnet1 metric 10 weight 1 mtu 1500 table inet6.0
via gw 3fff::3 iface vtnet1 metric 10 weight 2 mtu 1500
via gw 3fff::1 iface vtnet1 metric 5 weight 1 mtu 1500
Now you have:
Or, remove a specific gateway from ECMP without touching other metrics:
% route -n6 delete -net 3fff:a::/64 -gateway 3fff::3 -metric 10
delete net 3fff:a::/64: gateway 3fff::3 fib 0
% route -on6 get 3fff:a::/64
route to: 3fff:a::
destination: 3fff:a::
mask: ffff:ffff:ffff:ffff::
fib: 0
flags: <UP,GATEWAY,DONE,STATIC>
nhops: 2
via gw 3fff::2 iface vtnet1 metric 10 weight 1 mtu 1500 table inet6.0
via gw 3fff::1 iface vtnet1 metric 5 weight 1 mtu 1500
% netstat -On6W
Nexthop groups data
Internet6:
GrpIdx NhIdx Weight Slots Gateway Netif Refcnt
15 ------- ------- ------- -------------------------- ----- 2
12 1 0 3fff::2 vtnet1
14 1 1 3fff::1 vtnet1
You can now see metric values in netstat output:
netstat -rn6 -W # Wide output includes metric
netstat -rn6 --libxo json # Structured output with metric field
You can find the metric attribute in libxo output.
% netstat -rn6 --libxo json | jq '.statistics."route-information"."route-table"."rt-family".[]."rt-entry".[] | select(.destination | contains("3fff:a::/64"))'
{
"destination": "3fff:a::/64",
"gateway": "3fff::2",
"flags": "UGS",
"flags_pretty": [
"up",
"gateway",
"static"
],
"weight": 1,
"metric": 10,
"nhg-kidx": 15,
"interface-name": "vtnet1"
}
{
"destination": "3fff:a::/64",
"gateway": "3fff::1",
"flags": "UGS",
"flags_pretty": [
"up",
"gateway",
"static"
],
"weight": 1,
"metric": 5,
"nhg-kidx": 15,
"interface-name": "vtnet1"
}
You can now see all the nexthops of a route in route output:
% route -on 3fff:a::/64
route to: 3fff:a::
destination: 3fff:a::
mask: ffff:ffff:ffff:ffff::
fib: 0
flags: <UP,GATEWAY,DONE,STATIC>
nhops: 3
via gw 3fff::2 iface vtnet1 metric 10 weight 1 mtu 1500 table inet6.0
via gw 3fff::3 iface vtnet1 metric 10 weight 2 mtu 1500
via gw 3fff::1 iface vtnet1 metric 5 weight 1 mtu 1500
By default bird uses metric value of 32 during route installation, you can overwrite it by setting metric keyword in kernel section:
protocol kernel kernel6 {
metric 1;
ipv6 {
import all;
export all;
};
learn all;
merge paths;
}
Let’s look at the manual first:
UMA (Universal Memory Allocator) provides an efficient interface for managing dynamically-sized collections of items of identical size, referred to as zones.
Looks like UMA directly responsible for zones. But, what are zones? The manual continues with:
Zones keep track of which items are in use and which are not, and UMA provides functions for allocating items from a zone and for releasing them back, making them available for subsequent allocation requests.
Ok, I should start looking for its code.
As defined in SYNOPSIS of UMA(9), the source code resides in /usr/src/sys/vm/. So let’s list files with uma in their names.
[spmzt@cornelia] [/usr/src/sys/vm] % ls *uma*
uma_align_mask.h uma_core.c uma_dbg.c uma_dbg.h uma_int.h uma.h
By seeing their namees, I instantly know I should start with uma_core.c for the main code and uma.h for it’s header.
Now, I probably should looking for zone structures, let’s open the uma.h first.
/* Types and type defs */
struct uma_zone;
uma_zone structure is the first structure defined in the system. However, as you can see, it’s a forward declaration.
We should find it. If you familiar with style(9), you can easily find your desired code in the FreeBSD kernel.
[root@cornelia] [/usr/src/sys/vm] # grep -ri '^struct uma_zone {'
uma_int.h:struct uma_zone {
Yup, it did the trick. our main zone structure is defined in uma_int.h.
After, I opened the uma_int.h file, I saw more than hundered of documentation at the beginning of the file.
Here is the break down of what it says.
Zone
…Zones describe unique allocation types. Zones are
- organized into per-CPU caches which are filled by buckets.
Bucket
…Buckets are
- organized according to memory domains. Buckets are filled from kegs which
- are also organized according to memory domains.
Keg
… Kegs describe a unique
- allocation type, backend memory provider, and layout. Kegs are associated
- with one or more zones and zones reference one or more kegs.
Slab
…Kegs provide
- slabs which are virtually contiguous collections of pages. Each slab is
- broken down int one or more items that will satisfy an individual allocation.
At the end of its brief summary, there is an ASCII representation of a slab.
/*
* This is the representation for normal (Non OFFPAGE slab)
*
* i == item
* s == slab pointer
*
* <---------------- Page (UMA_SLAB_SIZE) ------------------>
* ___________________________________________________________
* | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ___________ |
* ||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i| |slab header||
* ||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_| |___________||
* |___________________________________________________________|
*
*
* This is an OFFPAGE slab. These can be larger than UMA_SLAB_SIZE.
*
* ___________________________________________________________
* | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |
* ||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i||i| |
* ||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_| |
* |___________________________________________________________|
* ___________ ^
* |slab header| |
* |___________|---*
*
*/
So, we have a slab type which called normal or non OFFPAGE slab that its slab header is located at the end of the allocated page.
Also, there is an OFFPAGE slab, it can be larger than UMA_SLAB_SIZE which is currently defined as a PAGE_SIZE (which is 4096 bytes in my laptop.).
How I know that? using sysctl hw.pagesize you can findout your own pagesize. Typically, it is 4096 bytes in x86 architectures.
Back to OFFPAGE slab, it uses a separate slab header outside the allocated page.
Continuing on uma_zone structure, it has a lot of fields inside its struct. Some of them catches my attention:
uint32_t uz_size; /* Size inherited from kegs */
uint64_t uz_max_items; /* Maximum number of items to alloc */
We have size of each object and the maximum number of objects in here.
uma_ctor uz_ctor; /* Constructor for each allocation */
uma_dtor uz_dtor; /* Destructor */
ctor and dtor is described in UMA(9) manual:
The ctor and dtor arguments are callback functions that are called by the UMA subsystem at the time of the call to uma_zalloc() and uma_zfree() respectively. Their purpose is to provide hooks for initializing or destroying things that need to be done at the time of the allocation or release of a resource. A good usage for the ctor and dtor callbacks might be to initialize a data structure embedded in the item, such as a queue(3) head.
In summary, we use them to initialize a data structure in the items at the time of allocation or release of a resource.
uint16_t uz_bucket_size; /* Number of items in full bucket */
uint16_t uz_bucket_size_max; /* Maximum number of bucket items */
These are used for defining the number of items in full bucket of a zone and its max limitation of items in that those buckets.
const char *uz_name; /* Text name of the zone */
The name of the zone defined here.
uma_init uz_init; /* Initializer for each item */
uma_fini uz_fini; /* Finalizer for each item. */
The init and fini are described in UMA(9) too:
The zinit and zfini arguments are used to optimize the allocation of items from the zone. They are called by the UMA subsystem whenever it needs to allocate or free items to satisfy requests or memory pressure. A good use for the zinit and zfini callbacks might be to initialize and destroy a mutex contained within an item. This would allow one to avoid destroying and re-initializing the mutex each time the item is freed and re-allocated.
memory pressure is refered to pageout daemon, which is the only process that ask the UMA to free its items to free memory.
LIST_ENTRY(uma_zone) uz_link; /* List of all zones in keg */
uma_keg_t uz_keg; /* This zone's keg if !CACHE */
Finally, here is a pointer from zone to keg. We probably should move to the keg structure to see what actually are they.
You can find the uma_keg structure in the file we are currently looking at.
LIST_HEAD(,uma_zone) uk_zones; /* Keg's zones */
This field in uma_keg shows us the keg can have multiple zones connected together using a linked list.
uint32_t uk_size; /* Requested size of each item */
uint32_t uk_rsize; /* Real size of each item */
These struct members are the one that zone inherited the size from them. So which one it uses?
You see these two lines under the zone_ctor function. The one who initializes all fields, lock, etc for a zone:
zone->uz_keg = keg;
zone->uz_size = keg->uk_size;
So it’s the requested size of each item stored in zone struct, not the real size.
uma_init uk_init; /* Keg's init routine */
uma_fini uk_fini; /* Keg's fini routine */
As you can see, we have uma_init and uma_fini for kegs too.
uma_alloc uk_allocf; /* Allocation function */
uma_free uk_freef; /* Free routine */
uint16_t uk_ipers; /* Items per slab */
As we follow the structure, we can see a clear hierarchy in normal zone. Zones have number of items, and keg have their own number of items per slab.
const char *uk_name; /* Name of creating zone. *
Here is the name of the keg, as defined in comment of the code, it uses the same of zone name. I can confirm it in the keg_ctor function:
keg->uk_name = zone->uz_name;
And last but not least:
LIST_ENTRY(uma_keg) uk_link; /* List of all kegs */
All of the kegs linked together.
Now it’s time to move to the uma_slab structure. The structure is simple and self explaining. Here is the full structure:
/*
* The slab structure manages a single contiguous allocation from backing
* store and subdivides it into individually allocatable items.
*/
struct uma_slab {
LIST_ENTRY(uma_slab) us_link; /* slabs in zone */
uint16_t us_freecount; /* How many are free? */
uint8_t us_flags; /* Page flags see uma.h */
uint8_t us_domain; /* Backing NUMA domain. */
struct noslabbits us_free; /* Free bitmask, flexible. */
};
The only struct member that may need explaining is free bitmask. Bitmask simply represent free and allocated items.
To be continued…
]]>I’m tired of managing different virtual machines when I can use hierarchial jails to achieve my requirement of separation and isolation. After all, using virtual machines while using same FreeBSD OS for different tasks plus having ZFS on every VM is pointless. I’ll stop ranting. Let’s get to work.
I want to migrate each of my virtual machines to a separate jail for each of them and I’ll use hierarchial jails in order to migrate jails on those VMs.
I was using standard switches (A.K.A bridge) on my hypervisor with uplinks of my server. So in order to make things easier, I want to use JIB to manage each of the jails interfaces under its required uplink. In order to accomplish that, I have to use VNET to virtualize network stack.
It’s in the base, so:
install -o root -g wheel -m 0555 /usr/share/examples/jails/jib /usr/local/sbin/jib
I will use the pf firewall for my jails. Of course we have to deal with devfs and bridge for jib in order to make this happen.
If you do not want to use ZFS, there is no need for adding new rules. Use can use the devfsrules_jail_vnet=5 which is already configured by default.
However, I want my jails to have access to zfs in order to create datasets. So I will create new devfs rules at /etc/devfs.rules.
[jail_vnet_zfs=6]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_jail
add include $devfsrules_jail_vnet
add path zfs unhide
Use this under your jail configuration:
# unhide pf
mount.devfs;
devfs_ruleset = 6;
I do not want the pf to care about any bridge interfaces in my network. So we have to tell it somehow.
Add these configuration to your /etc/sysctl.conf and apply it using sysctl -f /etc/sysctl.conf.
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_member=0
As the matter of fact, We mostly like the snapshot ability on the hypervisors and that is one of the main reasons that we put EVERYTHING under a separate VM. Then we realize ZFS under FreeBSD bring us this ability perfectly. As well as, It’s ZFS, you probably can name many other reasons why use the ZFS for each of the jails.
I will create a separate dataset for jails directory of my server, and then for root directory of each jails.
zfs create -o mountpoint=/usr/jails tank/jails
zfs create -p tank/jails/vm1/usr
Attach the ZFS dataset under your jails and create your vnet interfaces. This is how i do it:
vnet;
vnet.interface = "e0b_${name}";
exec.prestart += "jib addm ${name} $pif";
exec.created += "zfs set jailed=on tank/jails/${name}/usr || echo ZFS already jailed";
exec.created += "zfs jail ${name} tank/jails/${name}/usr";
exec.poststart += "jexec ${name} zfs mount -R tank/jails/${name}/usr";
exec.poststart += "jexec ${name} zfs list -H -o name -r tank/jails/${name}/usr | xargs -I '{}' jexec ${name} zfs mount {} || echo ZFS already mounted";
exec.poststart += "service -j ${name} jail onestart";
exec.prestop += "zfs unjail ${name} tank/jails/${name}/usr || echo zfs dataset is already unjailed";
exec.poststop += "jib destroy ${name}";
I do not like the jail managers everytime I do not have the CI/CD pipelines. In like manner, I found it easier to use the jail.conf.d itself in the base system.
Remamber to replace $pif variable with your own uplink interface of the host.
Simply create the base jail configuration at /etc/jail.conf:
path = "/usr/jails/${name}";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
# FQDN
host.hostname = "${name}.spmzt.net";
host.domainname = "spmzt.net";
# Global Variables
$pif="vmx0";
Then create the configuration of your jails in the separate files under /etc/jail.conf.d/:
.include "/etc/jail.conf";
mayuri {
persist;
# unhide pf
mount.devfs;
devfs_ruleset = 5;
# parent permissions
children.max = 12;
allow.mount.devfs;
# required by zfs jailed
allow.mount;
allow.mount.zfs;
enforce_statfs=0;
# parent permission (required by netflow grandchild)
allow.sysvipc;
# parent permission (required by ocserv1 grandchild)
allow.mount.procfs;
# decrease securelevel to be able to mount
securelevel=0;
# Network Configurations
vnet;
vnet.interface = "e0b_${name}";
exec.prestart += "jib addm ${name} $pif";
exec.poststart += "zfs set jailed=on tank/jails/${name}/usr || echo ZFS already jailed";
exec.poststart += "zfs jail ${name} tank/jails/${name}/usr";
exec.poststart += "jexec ${name} zfs mount -R tank/jails/${name}/usr";
exec.poststart += "jexec ${name} zfs list -H -o name -r tank/jails/${name}/usr | xargs -I '{}' jexec ${name} zfs mount {} || echo ZFS already mounted";
exec.poststart += "service -j ${name} jail onestart";
exec.prestop += "zfs unjail ${name} tank/jails/${name}/usr || echo zfs dataset is already unjailed";
exec.poststop += "jib destroy ${name}";
You need the command below in your jail configuration to start child jails.
exec.poststart += "service -j ${name} jail onestart";
I need some kernel modules inside my virtual machine when I want to use them as jails inside my host. I add to add those kernel modules inside the host itself.
sysrc kld_list=+"netgraph ng_ether ng_tee ng_socket ng_netflow ng_ksocket ng_eiface ng_iface ng_bridge ipsec"
To enable IPFW with logging:
# IPFW
firewall_enable="YES"
firewall_script="/etc/ipfw.rules"
firewall_logging="YES"
firewall_logif="YES"
Configure IPFW by adding rules inside /etc/ipfw.rules as shell script:
#!/bin/sh
# Flush out the list before we begin.
ipfw -q -f flush
# Delete tables and NATs
ipfw -q table all destroy
ipfw -q nat64lsn all destroy
ipfw -q nat64stl all destroy
# Replace config
ipfw set disable 1
# Set rules command prefix (for test add -n)
cmd="/sbin/ipfw -q add"
# interface name of NIC attached to Internet
pif="vmx0"
epairs="epair*"
# Non-routable IPv4 addresses
## Enable this for RFC1918
#martians="{ 192.168.0.0/16 or 172.16.0.0/12 or 10.0.0.0/8 \
## Note: 204.152.64.0/23 is for Sun cluster interconnect
martians="{ 127.0.0.0/8 or 169.254.0.0/16 or 192.0.2.0/24 or 0.0.0.0/8 or 240.0.0.0/4 \
or 204.152.64.0/23 or 224.0.0.0/3 }"
# IPv6
## Allow IPv6 DAD
$cmd 00031 set 1 pass icmp6 from :: to ff02::/16
## Allow IPv6 RS, RA, NS, NA, & redirect
$cmd 00032 set 1 pass icmp6 from fe80::/10 to fe80::/10
$cmd 00033 set 1 pass icmp6 from fe80::/10 to ff02::/16
# Allow ICMPv6 destination unreachable, NS, NA, & too-big
$cmd 00034 set 1 pass icmp6 from any to any icmp6types 1,2,3,4
$cmd 00035 set 1 pass icmp6 from any to any icmp6types 135,136
# NAT64 Rules (50-69)
# No restrictions on Loopback/me Interfaces
$cmd 00070 set 1 pass log all from any to any via lo0 record-state
$cmd 00071 set 1 pass log all from { me or me6 } to { me or me6 } record-state
# Scrub
$cmd 00099 set 1 reass all from any to any in
# NAT Rules (100-149)
# Passes the packet through if it matches an existing entry in the dynamic rules table
$cmd 00150 set 1 check-state
# Pass from me
$cmd 00161 set 1 pass tcp from { me or me6 } to any established keep-state
$cmd 00163 set 1 pass tcp from { me or me6 } to any setup keep-state
$cmd 00165 set 1 pass udp from { me or me6 } to any keep-state
$cmd 00167 set 1 pass icmp from me to any keep-state
$cmd 00168 set 1 pass icmp6 from me6 to any keep-state
# Deny RH0 (RFC5095)
$cmd 00201 set 1 drop log ipv6 from any to any ext6hdr rthdr0
# Deny fragmented NDP packets (RFC6980)
$cmd 00211 set 1 drop log icmp6 from any to any ext6hdr frag icmp6types 130,131,132,133,134,135,136,143
# uRPF (RFC)
$cmd 00221 set 1 drop log ip from any to any not antispoof in
# Deny ICMPv6 Redirect and Renumbering
$cmd 00231 set 1 drop log icmp6 from any to any icmp6types 137,138
# Deny all inbound traffic from non-routable reserved address spaces
## IPv4
$cmd 00301 set 1 drop log ipv4 from $martians to any in recv $pif
$cmd 00302 set 1 drop log ipv4 from any to $martians out xmit $pif
## IPv6
$cmd 00321 set 1 drop log all from ::1/128 to any via $pif
### Site-local (RFC3879)
$cmd 00322 set 1 drop log all from fec0::/10 to any
### IPv4-Compatible IPv6 (RFC4291)
$cmd 00323 set 1 drop log all from ::/96 to any
### Deny Teredo IPv6 (RFC4380)
$cmd 00324 set 1 drop log all from 2001::/32 to any
$cmd 00325 set 1 drop log udp from any to any 3544
### Deny IPv6 Documentation (RFC3849)
$cmd 00326 set 1 drop log all from 2001:db8::/32 to any
### Deny Discard-Only IPv6 Address (RFC6666)
$cmd 00327 set 1 drop log all from 100::/64 to any
### Deny ORCHID (RFC5156)
$cmd 00328 set 1 drop log all from 2001:10::/28 to any
### Deny Benchmarking IPv6 (RFC5180)
$cmd 00329 set 1 drop log all from 2001:2::/48 to any
### Deny 6Bone (RFC3701)
$cmd 00330 set 1 drop log all from { 3ffe::/16 or 5f00::/8 } to any
### Deny AYIYA (draft-massar-v6ops-ayiya-02)
$cmd 00331 set 1 drop log all from any to any 5072
### Deny TSP (RFC5572)
$cmd 00332 set 1 drop log all from any to any 3653
### XMAS tree
$cmd 00341 set 1 drop log tcp from any to any in tcpflags fin,psh,urg recv $pif
### NULL scan (no flag set at all)
$cmd 00342 set 1 drop log tcp from any to any in tcpflags !fin,!syn,!rst,!psh,!ack,!urg recv $pif
### SYN flood (SYN,FIN)
$cmd 00343 set 1 drop log tcp from any to any in tcpflags syn,fin recv $pif
### Stealth FIN scan (FIN,RST)
$cmd 00344 set 1 drop log tcp from any to any in tcpflags fin,rst recv $pif
### forced packet routing
$cmd 00345 set 1 drop log ip from any to any in ipoptions ssrr,lsrr,rr,ts recv $pif
### Ack scan
$cmd 00346 set 1 drop log tcp from any to any in tcpflags ack,rst recv $pif
# Deny ident
$cmd 00351 set 1 drop log tcp from any to any 113 in via $pif
# Deny all Netbios services.
$cmd 00361 set 1 drop log tcp from any to any 137 in via $pif
$cmd 00362 set 1 drop log tcp from any to any 138 in via $pif
$cmd 00363 set 1 drop log tcp from any to any 139 in via $pif
$cmd 00364 set 1 drop log tcp from any to any 81 in via $pif
# Deny fragments
$cmd 00370 set 1 drop log all from any to not me6 frag in via $pif
# Pass ICMP
# pass path-mtu in both directions
$cmd 00381 set 1 pass icmp from any to any icmptypes 3
# pass source quench in and out
$cmd 00383 set 1 pass icmp from any to any icmptypes 4
# pass outbound traceroutes
$cmd 00385 set 1 pass icmp from any to any icmptypes 11 in
# Allow DHCP
$cmd 00391 set 1 pass udp from any dhcpv6\\-server to me6 dhcpv6\\-client in via $pif keep-state
$cmd 00393 set 1 pass udp from any 67 to me 68 in recv $pif keep-state
# Local Rules (From 400 Rule)
## Note: For ssh protection use blacklistd(8)
# Pass ICMPv6 Ping
$cmd 00401 set 1 pass icmp6 from any to any icmp6types 128,129 keep-state
# Pass outbound pings and incoming ping responses
#$cmd 00403 set 1 pass icmp from any to any icmptypes 8 out
#$cmd 00404 set 1 pass icmp from any to any icmptypes 0 in
# SSH
$cmd 00405 set 1 pass tcp from any to { me or me6 } ssh in recv $pif
# Pass Outgoing Traffic
$cmd 09901 set 1 pass all from any to any out keep-state
# Deny ACK packets that did not match the dynamic rule table
$cmd 09903 set 1 drop log tcp from any to any established in via $pif
# Everything else is denied
$cmd 09999 set 1 drop log all from any to any
#$cmd 09999 set 1 pass log all from any to any in
# NAT Outbound (10000)
# Out (Maybe)
$cmd 10001 set 1 pass log all from any to any out keep-state
# Drop Outgoing Traffic after pass ipv4 NAT
$cmd 19999 set 1 drop log all from any to any out
# Replace config
ipfw set swap 0 1
ipfw delete set 1
Don’t forget to add execute permission:
chmod +x /etc/ipfw.rules
To limit the number of times a rule is logged per connection attempt, specify the number using this line in /etc/sysctl.conf:
net.inet.ip.fw.verbose_limit=5000
And make it persistent:
echo "net.inet.ip.fw.verbose_limit=5000" >> /etc/sysctl.conf
To keep sessions during flush:
echo net.inet.ip.fw.dyn_keep_states=1 >> /etc/sysctl.conf
echo net.inet.ip.fw.dyn_short_lifetime=30 >> /etc/sysctl.conf
echo net.inet.ip.fw.dyn_fin_lifetime=3 >> /etc/sysctl.conf
echo net.inet.ip.fw.dyn_rst_lifetime=3 >> /etc/sysctl.conf
echo net.inet.ip.fw.dyn_syn_lifetime=30 >> /etc/sysctl.conf
To make your life easier by not forcing every jails that using VNET for their network to have a configured firewall.
echo 'net.inet.ip.fw.default_to_accept=1' >> /boot/loader.conf.local
Do not forget to disable TSO.
echo net.inet.tcp.tso=0 >> /etc/sysctl.conf
Configure the NAT instance:
kldstat -q -m ipfw_nat || kldload ipfw_nat
ipfw nat 1 config if $pif same_ports unreg_only reset
The inbound NAT rule is inserted after the two rules which allow all traffic on the trusted and loopback interfaces and after the reassemble rule but before the check-state rule.
$cmd 00100 set 1 nat 1 ipv4 from any to any in recv $pif
This one for outbound:
$cmd 10000 set 1 nat 1 ipv4 from any to any out xmit $pif
Add an skipto 10000 before any pass outbound rule after inbount NAT rule.
For our base rules:
# Pass from me
$cmd 00160 set 1 skipto 10000 ipv4 from me to any proto tcp out xmit $pif
$cmd 00161 set 1 pass tcp from { me or me6 } to any established keep-state
$cmd 00162 set 1 skipto 10000 ipv4 from me to any proto tcp out xmit $pif
$cmd 00163 set 1 pass tcp from { me or me6 } to any setup keep-state
$cmd 00164 set 1 skipto 10000 ipv4 from me to any proto udp out xmit $pif
$cmd 00165 set 1 pass udp from { me or me6 } to any keep-state
$cmd 00166 set 1 skipto 10000 ipv4 from me to any proto icmp out xmit $pif
$cmd 00167 set 1 pass ipv4 from me to any proto icmp keep-state
$cmd 00168 set 1 pass icmp6 from me6 to any keep-state
# Pass ICMP
# pass path-mtu in both directions
$cmd 00380 set 1 skipto 10000 ipv4 from any to any proto icmp icmptypes 3 out xmit $pif
$cmd 00381 set 1 pass icmp from any to any icmptypes 3
# pass source quench in and out
$cmd 00382 set 1 skipto 10000 ipv4 from any to any proto icmp icmptypes 4 out xmit $pif
$cmd 00383 set 1 pass icmp from any to any icmptypes 4
# pass outbound traceroutes
$cmd 00384 set 1 skipto 10000 ipv4 from any to any proto icmp icmptypes 11 in
$cmd 00385 set 1 pass icmp from any to any icmptypes 11 in
# Pass ICMPv6 Ping
$cmd 00401 set 1 pass icmp6 from any to any icmp6types 128,129 keep-state
# Pass outbound pings and incoming ping responses
#$cmd 00402 set 1 skipto 10000 ipv4 from any to any proto icmp icmptypes 8 out xmit $pif
#$cmd 00403 set 1 pass icmp from any to any icmptypes 8 out
#$cmd 00404 set 1 pass icmp from any to any icmptypes 0 in
# Pass Outgoing Traffic
$cmd 09900 set 1 skipto 10000 ipv4 from any to any out xmit $pif
$cmd 09901 set 1 pass all from any to any out keep-state
Configure the NAT instance:
ipfw nat 123 config redirect_addr 10.0.0.1 10.0.0.66
redirect_port tcp 192.168.0.1:80 500
redirect_proto udp 192.168.1.43 192.168.1.1
redirect_addr 192.168.0.10,192.168.0.11
10.0.0.100 # LSNAT
redirect_port tcp 192.168.0.1:80,192.168.0.10:22
500 # LSNAT
or it could be split in:
ipfw nat 1 config redirect_addr 10.0.0.1 10.0.0.66
ipfw nat 2 config redirect_port tcp 192.168.0.1:80 500
ipfw nat 3 config redirect_proto udp 192.168.1.43 192.168.1.1
ipfw nat 4 config redirect_addr 192.168.0.10,192.168.0.11,192.168.0.12 10.0.0.100
ipfw nat 5 config redirect_port tcp 192.168.0.1:80,192.168.0.10:22,192.168.0.20:25 500
echo 'net.inet.ip.fw.nat64_direct_output=1' >> /etc/sysctl.conf
ipfw -q table T46 create type addr valtype ipv6
ipfw -q table T64 create type addr valtype ipv4
ipfw -q table T46 add 2.2.1.1 2a01:e140::53
ipfw -q table T64 add 2a01:e140::53 2.2.1.1
ipfw -q nat64stl NAT create table4 T46 table6 T64
$cmd 00100 set 1 nat64stl NAT ip from any to table(T46)
$cmd 00101 set 1 nat64stl NAT ip from table(T64) to 64:ff9b::/96
ipfw -q set 1 nat64lsn NAT64 create prefix4 A.B.C.D/32
$cmd 00110 set 1 nat64lsn NAT64 ipv4 from any to A.B.C.D/32 in
$cmd 00111 set 1 nat64lsn NAT64 ipv6 from any to 64:ff9b::/96 in
$cmd set 1 pass ip from any to any
For example
kldstat -q -m ipfw_nat64 || kldload ipfw_nat64
myip4="1.1.1.1/32"
$cmd -f flush
ipfw -q nat64lsn all destroy
# NAT64 Rules
$cmd 00050 set 1 nat64lsn NAT64 log ipv4 from any to $myip4 in recv $pif record-state
$cmd 00051 set 1 nat64lsn NAT64 log ipv6 from 2a01:e140:dead::/44 to 2a01:e140:cafe:ff::/96 in record-state
$cmd 00060 set 1 pass ipv4 from 172.30.10.2 to any in recv lo0 record-state
$cmd 00061 set 1 pass ipv4 from 172.30.10.2 to any out xmit $pif record-state
$cmd 00062 set 1 pass ipv6 from 2a01:e140:cafe:ff::/96 to 2a01:e140:dead::/44 in recv $pif record-state
$cmd 00063 set 1 pass ipv6 from 2a01:e140:cafe:ff::/96 to 2a01:e140:dead::/44 out xmit $pif record-state
$cmd 00064 set 1 pass ipv6 from 2a01:e140:cafe:ff::/96 to 2a01:e140:dead::/44 in recv lo0 record-state
By using sysctl variable below, you can separately manage/filter your inbound and outbound traffic:
sysctl net.inet.ip.fw.one_pass=0
Here is my recommended configuration:
#!/bin/sh
kldstat -q -m ipfw_nat || kldload ipfw_nat
kldstat -q -m ipfw_nat64 || kldload ipfw_nat64
### Documentations
## 5 Outgoing Jump
## 9-99 Before incoming NAT
## 100-199 Incoming NAT64
## 200-299 Incoming NAT
## 300-399 Check-state After incoming NAT
## 400-499 After incoming NAT
## 500-599 Before outgoing NAT
## 600-699 Outgoing NAT
## 700-799 Check-state After outgoing NAT
## 800-899 After NAT outgoing
### Skipto
## 1000-1999 IPv6 to me before incoming NAT
## 2000-2999 IPv4 to me before incoming NAT
## 3000-3999 IPv6 to me after incoming NAT
## 4000-4999 IPv4 to me after incoming NAT
## 5000-5999 IPv6 from me before outgoing NAT
## 6000-6999 IPv4 from me before outgoing NAT
## 7000-7999 IPv6 from me after outgoing NAT
## 8000-8999 IPv4 from me after outgoing NAT
## 9000-9099 IPv6 Link-Locals Only
## 9100-9199 Silent Drop (Incoming Drop Only)
## 9200-9999 Dangerous Packets (Incoming Drop Only)
## 10000-10999 IPv6 incoming forward
## 11000-11999 IPv4 incoming forward
## 12000-12999 IPv6 outgoing forward
## 13000-13999 IPv4 outgoing forward
# Flush out the list before we begin.
ipfw -q -f flush
# Delete tables and NATs
ipfw -q table all destroy
ipfw -q nat64lsn all destroy
ipfw -q nat64stl all destroy
# Replace Config
ipfw set disable 1
# Set rules command prefix (for test add -n)
cmd="/sbin/ipfw -q add"
# interface name of NIC attached to Internet
pif="vtnet0"
myip4="198.51.100.10/32"
# Networks
corp_pub="{ 192.0.2.0/24 or 203.0.113.0/24 }"
corp_pub6="{ 2001:2:1::/48 or 2001:2:2::/48 }"
# Non-routable IPv4 addresses
## Enable this for RFC1918
#martians="{ 127.0.0.0/8 or 192.168.0.0/16 or 172.16.0.0/12 or 10.0.0.0/8 \
## Note: 204.152.64.0/23 is for Sun cluster interconnect
martians="{ 169.254.0.0/16 or 192.0.2.0/24 or 0.0.0.0/8 or 240.0.0.0/4 \
or 204.152.64.0/23 or 224.0.0.0/3 }"
# NAT64
#ipfw -q table T46 create type addr valtype ipv6
#ipfw -q table T64 create type addr valtype ipv4
#ipfw -q set 1 nat64lsn NAT64 create prefix4 $myip4 prefix6 64:ff9b::/96
# NAT
ipfw nat 1 config if $pif same_ports unreg_only \
redirect_port tcp 172.29.0.1:80 80 \
redirect_port tcp 172.29.0.2:53 53 \
# IPv6 Link-locals
$cmd 00001 set 1 call 9000 ipv6 from any to any
# Outgoing Jump
$cmd 00005 set 1 skipto 00500 all from any to any out
#### Before incoming NAT
# Silent Drop to avoid logging
$cmd 00020 set 1 call 9100 all from any to any in
# Dangerous Packets
$cmd 00030 set 1 call 9200 all from any to any in
# IPv6 to me
$cmd 00040 set 1 call 1000 ipv6 from any to me6 in
# IPv4 to me
$cmd 00050 set 1 call 2000 ipv4 from any to me in
# Forward incoming IPv6
$cmd 00060 set 1 call 10000 ipv6 from any to any in
# Forward incoming IPv4
$cmd 00070 set 1 call 11000 ipv4 from any to any in
# Scrub (IPv4)
$cmd 00099 set 1 reass all from any to any in
#### Incoming NAT
# NAT64 Rules (100-199)
#$cmd 00110 set 1 nat64lsn NAT64 ipv4 from any to $myip4 in
#$cmd 00111 set 1 nat64lsn NAT64 ipv6 from any to 64:ff9b::/96 in
#$cmd 00120 set 1 pass ipv4 from $myip4 to any in recv lo0 record-state
#$cmd 00121 set 1 pass ipv4 from $myip4 to any out xmit $pif record-state
#$cmd 00130 set 1 pass ipv6 from 64:ff9b::/96 to 2a01:e140:dead::/44 in recv $pif record-state
#$cmd 00131 set 1 pass ipv6 from 64:ff9b::/96 to 2a01:e140:dead::/44 out xmit $pif record-state
#$cmd 00132 set 1 pass ipv6 from 64:ff9b::/96 to 2a01:e140:dead::/44 in recv lo0 record-state
# NAT Rules (200-299)
$cmd 00200 set 1 nat 1 ipv4 from any to $myip4 in recv $pif
### Check-state After incoming NAT
# Passes the packet through if it matches an existing entry in the dynamic rules table
$cmd 00300 set 1 check-state
### After incoming NAT
# Dangerous Packets
$cmd 00400 set 1 call 9200 all from any to any in
# IPv6 to me
$cmd 00410 set 1 call 3000 ipv6 from any to me6 in
# IPv4 to me
$cmd 00420 set 1 call 4000 ipv4 from any to me in
### Before outgoing NAT (500)
# IPv6 from me out
$cmd 00510 set 1 call 5000 ipv6 from me6 to any out
# IPv4 from me out
$cmd 00520 set 1 call 6000 ipv4 from me to any out
# Forward Outgoing IPv6
$cmd 00530 set 1 call 12000 ipv6 from any to any out
# Forward Outgoing IPv4
$cmd 00540 set 1 call 13000 ipv4 from any to any out
# NAT outbound
$cmd 00600 set 1 nat 1 ipv4 from any to any out xmit $pif
### Check-state After outgoing NAT
# Passes the packet through if it matches an existing entry in the dynamic rules table
$cmd 00700 set 1 check-state
### After outgoing NAT (800)
# IPv6 from me
$cmd 00810 set 1 call 7000 ipv6 from me6 to any out
# IPv4 from me
$cmd 00820 set 1 call 8000 ipv4 from any to any out
# Everything else is denied
$cmd 00999 set 1 drop log all from any to any
# IPv6 to me before incoming NAT (1000 -> 1999)
# Pass myself
$cmd 01000 set 1 pass ipv6 from me6 to me6 in
# Pass SSH
$cmd 01001 set 1 pass tcp from any to me6 ssh record-state defer-action in
# DNS
$cmd 01002 set 1 pass { tcp or udp } from any to { 2001:2::53 or 2001:2:10::53 } domain, domain\\-s record-state defer-action in
# Web
$cmd 01003 set 1 pass { tcp or udp } from any to { 2001:2::80 or 2001:2:10::80 } https,http record-state defer-action in
# Pass Internal Prefix
$cmd 01090 set 1 pass ipv6 from $corp_pub6 to any record-state defer-action in
# Pass ICMPv6
$cmd 01100 set 1 pass icmp6 from any to any in
# End of IPv6 to me6
$cmd 01999 set 1 return via any
# IPv4 to me before NAT (2000 -> 2999)
# Pass me
$cmd 02000 set 1 pass ipv4 from me to me in
# Pass SSH
$cmd 02001 set 1 pass tcp from any to $myip4 ssh record-state defer-action in recv $pif
# DNS
$cmd 02002 set 1 pass { tcp or udp } from any to $myip4 domain, domain\\-s record-state defer-action in recv $pif
# Web
$cmd 02003 set 1 pass tcp from any to $myip4 https,http record-state defer-action in recv $pif
# Pass Corp
$cmd 02090 set 1 pass ipv4 from $corp_pub to any record-state defer-action in recv $pif
# Pass ICMPv4
$cmd 02100 set 1 pass icmp from any to me record-state defer-action in recv $pif
# End of IPv4 to me
$cmd 02999 set 1 return via any
# IPv6 to me after incoming NAT (3000 -> 3999)
# End of IPv6 to me
$cmd 03999 set 1 return via any
# IPv4 to me after incoming NAT (4000 -> 4999)
# DNS
$cmd 04001 set 1 pass { tcp or udp } from any to 172.29.0.2 domain, domain\\-s keep-state in recv $pif
# Web
$cmd 04002 set 1 pass tcp from any to 172.29.0.1 https,http keep-state in recv $pif
# End of IPv4 to me
$cmd 04999 set 1 return via any
# IPv6 from me before outgoing nat (5000 -> 5999)
$cmd 05000 set 1 pass ipv6 from me6 to me6 out
$cmd 05001 set 1 pass ipv6 from me6 to any record-state defer-action out
# End of IPv6 from me6
$cmd 05999 set 1 return via any
# IPv4 from me before outgoing NAT (6000 -> 6999)
$cmd 06000 set 1 pass ipv4 from me to me out
# Allow any other out
$cmd 06101 set 1 pass ipv4 from me to any record-state defer-action out
# IPv4 xmit $pif Skipto outgoing NAT
$cmd 06900 set 1 skipto 00600 ipv4 from me to any out xmit $pif
# End of IPv4 from me before NAT
$cmd 06999 set 1 return via any
# IPv6 from me after outgoing nat (7000 -> 7999)
$cmd 07000 set 1 pass ipv6 from me6 to me6 out
$cmd 07001 set 1 pass ipv6 from me6 to any keep-state out
# End of IPv6 from me
$cmd 07999 set 1 return via any
# IPv4 from me after outgoing nat (8000 -> 8999)
$cmd 08000 set 1 pass ipv4 from me to me out
$cmd 08001 set 1 pass ipv4 from me to any keep-state out
# End of IPv4 from me
$cmd 08999 set 1 return via any
# Link-Locals Only (9000 -> 9099)
## Allow IPv6 DAD
$cmd 09000 set 1 pass icmp6 from :: to ff02::/16
## Allow IPv6 Link-locals specially for NDP (RS, RA, NS, NA, & redirect)
$cmd 09001 set 1 pass icmp6 from fe80::/10 to fe80::/10
$cmd 09002 set 1 pass icmp6 from fe80::/10 to ff02::/16
# Allow ICMPv6 NS, NA
$cmd 09010 set 1 pass icmp6 from fe80::/10 to 2000::/3 icmp6types 135,136
$cmd 09011 set 1 pass icmp6 from 2000::/3 to { ff02::/16 or fe80::/10 } icmp6types 135,136
# Link-Local Protocols
# Pass OSPF
$cmd 09020 set 1 pass ospf from fe80::/64 to any in recv $pif
# End of Link-Locals Only
$cmd 09099 set 1 return via any
# Silent Drop (9100 -> 9199)
# Drop Sooner to prevent being logged for nothing
$cmd 09100 set 1 drop udp from 0.0.0.0 68 to 255.255.255.255 67 in
$cmd 09101 set 1 drop udp from any 5678 to any 5678 in
$cmd 09102 set 1 drop udp from any to 255.255.255.255 5353 in
$cmd 09103 set 1 drop udp from any to 255.255.255.255 1900 in
# End of Silent Drop
$cmd 09199 set 1 return via any
# Dangerous Packets Drop Only (9200 -> 9999)
# Deny RH0 (RFC5095)
$cmd 09200 set 1 drop log ipv6 from any to any ext6hdr rthdr0 in
# uRPF (RFC)
$cmd 09201 set 1 drop log ip from not me to any not antispoof in
# Deny all inbound traffic from non-routable reserved address spaces
## IPv4
$cmd 09211 set 1 drop log ipv4 from $martians to any in recv $pif
$cmd 09212 set 1 drop log ipv4 from $martians to any in recv vtnet1
## IPv6
$cmd 09230 set 1 drop log ipv6 from ::1/128 to any in
### Site-local (RFC3879)
$cmd 09231 set 1 drop log ipv6 from fec0::/10 to any in
### IPv4-Compatible IPv6 (RFC4291)
$cmd 09232 set 1 drop log ipv6 from ::/96 to any in
### Deny Teredo IPv6 (RFC4380)
$cmd 09233 set 1 drop log ipv6 from 2001::/32 to any in
### Deny IPv6 Documentation (RFC3849)
$cmd 09234 set 1 drop log ipv6 from 2001:db8::/32 to any in
### Deny Discard-Only IPv6 Address (RFC6666)
$cmd 09235 set 1 drop log ipv6 from 100::/64 to any in
### Deny ORCHID (RFC5156)
$cmd 09236 set 1 drop log ipv6 from 2001:10::/28 to any in
### Deny Benchmarking IPv6 (RFC5180)
$cmd 09237 set 1 drop log ipv6 from 2001:2::/48 to any in
### Deny 6Bone (RFC3701)
$cmd 09238 set 1 drop log ipv6 from { 3ffe::/16 or 5f00::/8 } to any in
### XMAS tree
$cmd 09251 set 1 drop log tcp from any to any tcpflags fin,psh,urg in
### NULL scan (no flag set at all)
$cmd 09252 set 1 drop log tcp from any to any tcpflags !fin,!syn,!rst,!psh,!ack,!urg in
### SYN flood (SYN,FIN)
$cmd 09253 set 1 drop log tcp from any to any tcpflags syn,fin in
### Stealth FIN scan (FIN,RST)
$cmd 09254 set 1 drop log tcp from any to any tcpflags fin,rst in
### forced packet routing
$cmd 09255 set 1 drop log ip from any to any ipoptions ssrr,lsrr,rr,ts in
# Deny fragmented NDP packets (RFC6980)
$cmd 09271 set 1 drop log icmp6 from any to any ext6hdr frag icmp6types 130,131,132,133,134,135,136,143 in
# Deny ICMPv6 Redirect and Renumbering
$cmd 09272 set 1 drop log icmp6 from any to any icmp6types 137,138 in
# End of Dangerous packets
$cmd 09999 set 1 return via any
## 10000-10999 IPv6 incoming forward
$cmd 10000 set 1 pass ipv6 from 2a01:e140:dead::/48 to any in
$cmd 10001 set 1 pass ipv6 from any to 2a01:e140:dead::/48 in
# End of IPv6 Incoming Forward
$cmd 10999 set 1 return via any
## 11000-11999 IPv4 incoming forward
# End of IPv4 Incoming Forward
$cmd 11999 set 1 return via any
## 12000-12999 IPv6 outgoing forward
$cmd 12000 set 1 pass ipv6 from any to 2a01:e140:dead::/48 out
$cmd 12001 set 1 pass ipv6 from 2a01:e140:dead::/48 to any out
# End of IPv6 Incoming Forward
$cmd 12999 set 1 return via any
## 13000-13999 IPv4 outgoing forward
# End of IPv4 outgoing Forward
$cmd 13999 set 1 return via any
# Replace Config
ipfw set swap 0 1
ipfw delete set 1
Hope it helps!
]]>Enable (Persist) Services:
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 4"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"
Start Services:
service nfsd start
service lockd start
service statd start
Share tank/share dateset for 10.0.0.0/24 network:
zfs set mountpoint="maproot=root,sec=sys,alldirs,network=10.0.0.0/24" tank/share
zfs share tank/share
mkdir -p /home/user/.ssh
useradd user -m -d /home/user -s /bin/true
echo 'client-key' > /home/user/.ssh/authorized_keys
chown -R user:user /home/user
Replace x with your wanted remote port on the remote server.
Add to /etc/systemd/system/autossh-tunnel.service:
[Unit]
Description=AutoSSH tunnel service to server on local port 22
After=network.target
[Service]
Environment="AUTOSSH_GATETIME=0"
User=spmzt
Group=spmzt
ExecStart=/usr/bin/autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -NR x:localhost:22 user@spmzt.net -p 22
[Install]
WantedBy=multi-user.target
Replace x with your wanted remote port on the remote server.
Add to /usr/local/etc/rc.d/autossh_tunnel:
#!/bin/sh
#
# PROVIDE: autossh_tunnel
# REQUIRE: sshd
# KEYWORD: shutdown
. /etc/rc.subr
name=autossh_tunnel
desc="AutoSSH tunnel service to server on port 22"
rcvar=autossh_tunnel_enable
command="/usr/local/bin/sudo"
command_args='-u user /usr/local/bin/autossh -f -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -NR x:localhost:22 user@spmzt.net -p 22'
load_rc_config $name
#
# DO NOT CHANGE THESE DEFAULT VALUES HERE
# SET THEM IN THE /etc/rc.conf FILE
#
autossh_tunnel_enable=${autossh_tunnel_enable-"NO"}
pidfile=${autossh_tunnel_pidfile-"/var/run/autossh_tunnel.pid"}
run_rc_command "$1"
Here’s the problem: you may have access to a trusted network, but it doesn’t have direct access to the outside world. This means that you need to create an intermediate VPN that allows you to connect to the outside world from within the trusted network.
One way to set up this intermediate VPN is by using FreeBSD Jails with multiple FIBs. With this setup, you have full control over the traffic and can access multiple VPNs at the same time.
Here’s how it works: first, you establish a VPN connection from your laptop to the trusted network, which is your intermediate VPN. Then, using FreeBSD Jails with multiple FIBs, you can establish additional VPN connections from your intermediate VPN to multiple VPN endpoints. This allows you to bypass censorship and connect to VPNs that were previously inaccessible.
Using FreeBSD Jails with multiple FIBs is a powerful solution for those who need to access restricted networks. With this setup, you can have full control over your traffic and access multiple VPNs simultaneously.
In this blog post, I’m going to share my current network configuration. I’m going to break down my configuration step by step and explain each component in detail.
fib 0: without VPN
* vtnet0: 1.1.1.2/24
* e0a_overlay: 172.31.87.1/30
fib 1: with VPN
* e1a_overlay: 172.31.88.1/30
* tun257: openvpn-client 172.31.1.174 -> 173
jail:
* e0b_overlay: 172.31.87.2/30
* e1b_overlay: 172.31.88.2/30
* ocserv clients: 172.31.5.0/24
Firstly, my network consists of two FIBs: one without VPN and one with VPN. For the first FIB, I use the vtnet0 interface with IP address 1.1.1.2/24, while for the second FIB, I use the tun257 interface with IP address 172.31.1.174 -> 173. The VPN is established through OpenVPN, and the IP addresses for the VPN connections are 172.31.87.1/30 for e0a_overlay, and 172.31.88.1/30 for e1a_overlay.
Replace ip addresses and interface names to your own environment.
Add these to your /etc/rc.conf configurations:
ifconfig_vtnet0="inet 1.1.1.2/24 -tso -rxcsum"
defaultrouter="1.1.1.1"
cloned_interfaces="tun257 epair1 epair0"
ifconfig_epair0a="name e0a_overlay -tso -rxcsum"
ifconfig_epair0b="name e0b_overlay -tso -rxcsum"
ifconfig_epair1a="name e1a_overlay fib 1 -tso -rxcsum"
ifconfig_epair1b="name e1b_overlay fib 1 -tso -rxcsum"
ifconfig_tun257="fib 1 -tso -rxcsum"
pf_enable="YES"
pflog_enable="YES"
ezjail_enable="YES"
gateway_enable="YES"
static_routes="outsidevpn defaultfib1"
route_outsidevpn="-net 2.2.2.2 -gateway 1.1.1.1 -fib 1"
route_defaultfib1="-net default -gateway 1.1.1.1 -fib 1"
openvpn_outsidevpn_enable="YES"
openvpn_outsidevpn_fib="1"
openvpn_outsidevpn_configfile="/usr/local/etc/openvpn/outsidevpn.conf"
Next, let’s take a look at the rc.conf file. Here, we specify the configuration for our interfaces and the routing table. We enable the PF firewall and configure static routes for traffic to 2.2.2.2 which is our final vpn ip address. We also enable OpenVPN for outsidevpn, specifying that it uses FIB 1, and we provide the path to the OpenVPN configuration file.
/usr/jails/overlay/etc/rc.conf:
ifconfig_e0b_overlay="inet 172.31.87.2/30 fib 0"
ifconfig_e1b_overlay="inet 172.31.88.2/30 fib 0"
defaultrouter="172.31.88.1"
ocserv_enable="YES"
gateway_enable="YES"
Use the configuration below if you use ezjail for jail management.
export jail_overlay_hostname="overlay"
export jail_overlay_ip=""
export jail_overlay_rootdir="/usr/jails/overlay"
export jail_overlay_exec_start="/bin/sh /etc/rc"
export jail_overlay_exec_stop=""
export jail_overlay_mount_enable="YES"
export jail_overlay_devfs_enable="YES"
export jail_overlay_devfs_ruleset="6"
export jail_overlay_procfs_enable="YES"
export jail_overlay_fdescfs_enable="YES"
export jail_overlay_image=""
export jail_overlay_imagetype="zfs"
export jail_overlay_attachparams=""
export jail_overlay_attachblocking=""
export jail_overlay_forceblocking=""
export jail_overlay_zfs_datasets=""
export jail_overlay_cpuset=""
export jail_overlay_fib=""
export jail_overlay_parentzfs="zroot/jails"
export jail_overlay_parameters=""
export jail_overlay_post_start_script=""
export jail_overlay_retention_policy=""
export jail_overlay_vnet_enable="YES"
export jail_overlay_vnet_interface="e0b_overlay e1b_overlay"
export jail_overlay_exec_prestart0="/sbin/ifconfig e0a_overlay inet 172.31.87.1/30"
export jail_overlay_exec_prestart1="/sbin/ifconfig e1a_overlay inet 172.31.88.1/30"
export jail_overlay_exec_prestart2="/sbin/route add -net 172.31.5.0/24 -gateway 172.31.88.2 -fib 1"
export jail_overlay_exec_prestart3="/sbin/route add -net 172.31.5.0/24 -gateway 172.31.87.2"
export jail_overlay_exec_poststop0="/sbin/route del -net 172.31.5.0/24 -gateway 172.31.87.2"
export jail_overlay_exec_poststop1="/sbin/route del -net 172.31.5.0/24 -gateway 172.31.88.2 -fib 1"
export jail_overlay_exec_poststop2="/sbin/ifconfig e0a_overlay delete"
export jail_overlay_exec_poststop3="/sbin/ifconfig e1a_overlay delete"
Moving on to the jail configuration, we define several environment variables that specify the configuration for our jail. The jail_overlay_hostname specifies the hostname of the jail, while jail_overlay_ip is left empty because we use VNET to assign an IP address to the jail. We specify the root directory for the jail, as well as the startup and shutdown scripts. We enable various filesystems and attach the jail to a ZFS dataset. We also enable the VNET feature and specify the interfaces that the jail uses.
auth = "plain[passwd=/usr/local/etc/ocserv/auth.passwd]"
listen-host = 172.31.87.2
udp-listen-host = 172.31.87.2
tcp-port = 1435
udp-port = 1435
run-as-user = _ocserv
run-as-group = _ocserv
socket-file = /var/run/ocserv-socket
server-cert = /usr/local/etc/ocserv/fullchain.pem
server-key = /usr/local/etc/ocserv/privkey.pem
dh-params = /usr/local/etc/ocserv/dh.pem
ca-cert = /usr/local/etc/ocserv/ca.pem
banner = "It's Pouria, Welcome. All Clients Have 2 Concurrent Limitation."
max-clients = 32
max-same-clients = 2
rate-limit-ms = 100
server-stats-reset-time = 604800
keepalive = 32400
dpd = 90
mobile-dpd = 1800
switch-to-tcp-timeout = 25
try-mtu-discovery = true
cert-user-oid = 0.9.2342.19200300.100.1.1
tls-priorities = "NORMAL:%SERVER_PRECEDENCE:%COMPAT:-VERS-SSL3.0:-VERS-TLS1.0:-VERS-TLS1.1"
auth-timeout = 240
min-reauth-time = 300
max-ban-score = 80
ban-reset-time = 1200
cookie-timeout = 300
deny-roaming = false
rekey-time = 172800
rekey-method = ssl
use-occtl = true
pid-file = /var/run/ocserv.pid
log-level = 1
device = tun
predictable-ips = true
default-domain = overlay.spmzt.net
ipv4-network = 172.31.5.0
ipv4-netmask = 255.255.255.0
tunnel-all-dns = true
dns = 9.9.9.9
dns = 1.1.1.1
ping-leases = false
route = default
cisco-client-compat = true
dtls-legacy = true
client-bypass-protocol = false
I also have a jail, which has two interfaces, e0b_overlay and e1b_overlay, with IP addresses 172.31.87.2/30 and 172.31.88.2/30, respectively. This jail serves as an ocserv server, and all clients of ocserv have IP addresses in the range 172.31.5.0/24.
We take a look at the ocserv.conf file, which specifies the configuration for the ocserv server. Here, we define the authentication method and the port numbers for the TCP and UDP protocols. We also specify the server certificate and key, the Diffie-Hellman parameters, and the CA certificate.
set skip on { lo0 }
scrub in on vtnet0 all
nat pass on vtnet0 from 172.31.5.0/24 to any -> 1.1.1.2
nat pass on vtnet0 from 172.31.87.0/30 to any -> 1.1.1.2
nat pass on tun257 from 172.31.5.0/24 to any rtable 1 -> ( tun257 )
nat pass on tun257 from 172.31.88.0/30 to any rtable 1 -> ( tun257 )
rdr pass on vtnet0 inet proto tcp from any to 1.1.1.2 port = 1435 -> 172.31.87.2
rdr pass on vtnet0 inet proto udp from any to 1.1.1.2 port = 1435 -> 172.31.87.2
antispoof quick for vtnet0
block all
pass out on vtnet0 all
pass out on vtnet0 all rtable 1
pass log on e0a_overlay
pass log on e1a_overlay rtable 1
pass on tun257 all rtable 1
pass in on vtnet0 all
This is a configuration file for pf (Packet Filter), a firewall software for OpenBSD and other BSD-based operating systems.
The first line skips the firewall rules for the loopback interface.
The “scrub” line enables packet scrubbing on the “vtnet0” interface, which helps prevent certain types of attacks.
The next two “nat” lines perform network address translation (NAT) on traffic going out of the “vtnet0” interface, translating the source IP addresses from either the 172.31.5.0/24 or 172.31.87.0/30 network to 1.1.1.2.
The two “nat” lines that follow perform similar NAT functions for traffic going out of the “tun257” interface.
The two “rdr” lines redirect incoming TCP and UDP traffic destined for 1.1.1.2 on the “vtnet0” interface to 172.31.87.2.
The “antispoof” line enables antispoofing for the “vtnet0” interface, which helps prevent IP spoofing attacks.
The “block all” line blocks all traffic that is not specifically allowed by other rules.
The “pass out” lines allow all outgoing traffic on the “vtnet0” interface and also allow traffic from that interface with an associated routing table called “rtable 1”.
The two “pass log” lines allow traffic through two overlay interfaces called “e0a_overlay” and “e1a_overlay”, respectively.
The last two lines allow all traffic on the “tun257” interface and allow all incoming traffic on the “vtnet0” interface.
In the sysctl.conf file, we set net.add_addr_allfibs to 1.
net.add_addr_allfibs=1
net.fibs=2
sysctl -f /etc/sysctl.conf
[devfsrules_jail_vnet_tun=6]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_jail
add include $devfsrules_jail_vnet
add path 'tun*' unhide
The code you provided is a configuration file for devfs, which is the device file system used in FreeBSD operating system. The file is called devfs.conf and it contains a set of rules that specify how devices are accessed and managed.
The first line in the file sets the rule for a jail that has Virtual Network Interface Card (VNET) and a Tunnel interface (TUN) with the number 6. The following lines include other rules that hide or unhide various types of devices, such as basic devices, login devices, and jail devices. The final line in the file specifies that any device path that starts with “tun” should be unhidden, meaning it can be accessed by the jail.
Overall, this configuration allows me to maintain a secure and reliable network, with multiple FIBs and a jail that serves as an ocserv server. I hope this breakdown of my configuration has been helpful to you.
]]>In this case, I will move my entire operating system from ada0 to nvd0.
gpart create -s gpt nvd0
gpart add -a 1M -s 260M -t efi -l efiboot1 nvd0
gpart add -a 1M -s 4G -t freebsd-swap -l swap1 nvd0
gpart add -t freebsd-zfs -l zfsroot1 nvd0
The first command, gpart create -s gpt nvd0, creates a new partition table of type GPT (GUID Partition Table) on the disk identified as nvd0.
The next three commands add partitions to the newly created partition table on the nvd0 disk.
The command gpart add -a 1M -s 260M -t efi -l efiboot1 nvd0 adds a partition for EFI system, with a size of 260 megabytes and a label of efiboot1.
The command gpart add -a 1M -s 4G -t freebsd-swap -l swap1 nvd0 adds a partition for swap space, with a size of 4 gigabytes and a label of swap1.
The command gpart add -t freebsd-zfs -l zfsroot1 nvd0 adds a partition for ZFS (Zettabyte File System) and labels it zfsroot1.
gpart show -p nvd0
newfs_msdos -F32 -c 1 /dev/nvd0p1
mount -t msdosfs -o longnames /dev/nvd0p1 /mnt/
mkdir -p /mnt/EFI/BOOT
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.efi
umount /mnt/
The commands are used to format and mount an EFI System Partition (ESP) on a storage device /dev/nvd0 in the msdos file system format.
The first command newfs_msdos creates a new file system on the first partition /dev/nvd0p1 with the -F32 flag indicating to use the FAT32 file system, and the -c 1 flag indicates to set the cluster size to 1 sector.
The second command mount mounts the newly formatted partition on the mount point /mnt with the -t msdosfs flag indicating that the file system is msdosfs and the -o longnames flag enabling long file name support.
The third command creates the directory /mnt/EFI/BOOT with sudo mkdir -p to create the necessary folder structure for the EFI boot loader.
The fourth command copies the EFI boot loader loader.efi to /mnt/EFI/BOOT/BOOTX64.efi with sudo cp to make it available for booting.
The fifth and final command umount unmounts the partition at /mnt.
geli init -b -g -K /root/nvd0p3.key -s 4096 -l 256 /dev/nvd0p3
This is a command that initializes encryption using GELI (FreeBSD’s cryptographic disk encryption system) on the device /dev/nvd0p3.
The command uses several options:
-b creates a bootable GELI partition.
-g Enable booting from this encrypted root filesystem.
-K /root/nvd0p3.key specifies a key file to export for the encryption.
-s 4096 sets the sector size for the encrypted partition to 4096 bytes.
-l 256 sets the number of PBKDF2 iterations to use for key derivation.
Once the command is run, the system prompts the user to enter a passphrase to encrypt the partition. The encrypted partition can then be used like any other disk partition.
cp -R /var/backups/nvd0p3.eli /root/nvd0p3.key /media/da0p1/
geli configure -G /dev/ada0p3
geli attach -k /root/keys/nvd0p3.key /dev/nvd0p3
zpool attach zroot /dev/ada0p3.eli /dev/nvd0p3.eli
zpool status zroot
zpool detach zroot /dev/ada0p3.eli
sysctl net.fibs=2
setfib 1 route add default 172.16.1.1
echo 'net.fibs=2' >> /etc/sysctl.conf
add in jail.conf:
exec.fib=1;
ifconfig lo1 fib 1
The allfibs line is needed for FreeBSD 12+.
echo 'net.add_addr_allfibs=0' >> /boot/loader.conf
pkg install -y fusefs-exfat fusefs-ext2 fusefs-ntfs exfat-utils
echo 'vfs.usermount=1' | tee -a /etc/sysctl.conf
sysctl vfs.usermount=1
sysrc devfs_system_ruleset="localrules"
sysrc kld_list+="fusefs"
Add these lines to /etc/devfs.conf:
[localrules=5]
add path 'da[1-3]p*' mode 0660 group operator
Done:
mount -t msdosfs -o -m=644,-M=755 /dev/da1p1 /media
USB devices can be automatically mounted by uncommenting this line below in /etc/auto_master:
/media -media -nosuid
Edit `/etc/devd.conf
notify 100 {
match "system" "GEOM";
match "subsystem" "DEV";
action "/usr/sbin/automount -c";
};
And:
sysrc autofs_enable="YES"
service automount restart
service automountd restart
service autounmountd restart
service devd restart
pkg install -y dsbmc dsbmc-cli
sysrc dsbmd_enable=YES
service dsbmd start
Mount:
dsbmc-cli -m /dev/da0s1