21 Jan 2017

FreeBSD Server Guide

A guide to configuring your new FreeBSD server for performance and security.

FreeBSD is a secure, high-performance Unix-like operating system. It has been my server OS of choice since I started this self-hosting hobby in my college days. In this post, I'll describe how I set up my FreeBSD servers—installing packages, securing the firewall, tweaking network performance, and configuring daemons. This will be similar to those "first five minutes on a server" articles, but with a focus on FreeBSD 11. If you're not a BSD fan, you're misinformed, but much of the advice in here will apply to any Unix-like server that you connect to the internet.

  1. Why FreeBSD?
  2. Building Ports
  3. Hardware Configuration
  4. Network Settings
  5. Environment Setup
  6. Tweaking Network Performance
  7. LibreSSL and OpenSSH
  8. Clock Sync with OpenNTP
  9. Securing the PF Firewall
  10. Conclusion

Why FreeBSD?

Simply put, I use FreeBSD because it makes my life easier. Compared to Linux, it is a more integrated, stable, and well-documented operating system. And I don't mean stable in the sense that it has fewer bugs, but in the sense that it doesn't do things like switch init systems twice in the same decade. In addition, the BSD License provides far more freedom to developers and users than the convoluted, marxist GPLv3.

For an introduction to FreeBSD from a Linux perspective, this guide is usually cited as the best on the 'net. But for a shortened version: FreeBSD is an operating system, Linux is a kernel. FreeBSD is a cathedral, Linux is a bazaar. But subjectively, FreeBSD just feels better than Linux. Third party software is kept totally independent from the base OS, jails and ZFS are awesome, PF puts iptables to shame...I could go on and on. If you go with FreeBSD, you'll miss out on whatever the new Linux hotness of the day is. But in return, you'll get a solid, reliable Unix system that will quietly serve you well for years to come.

Building Ports

Before we dive into system configuration, we'll at least want vim installed. Third-party software packages in FreeBSD are called ports. You can either install binary packages using the pkg utility or build them from source. I prefer to build from source, as you get more fine-grained control over compile-time options and package dependencies. It doesn't really matter which method you choose, as long as you're consistent—mixing source and binary packages can sometimes cause odd behavior.

To start building ports, we'll need the latest version of the ports tree. Grab a cup of coffee while you run the following commands as root (it may take awhile).

portsnap fetch
portsnap extract

Next, we'll specify some build options in make.conf. Options that you put in this file will apply to every port you compile. It makes it easy to do things like globally disable X11 support—we won't need that on a headless server. Here is what I have in my make.conf:

# allow compiler optimizations specific to our CPU model CPUTYPE?=native # optimization level O2 is the highest supported by FreeBSD and most ports. CFLAGS=-O2 -pipe -fno-strict-aliasing # COPTFLAGS only apply when building the kernel COPTFLAGS=$CFLAGS # don't pull in X11, CUPS, etc OPTIONS_UNSET=DOCS NLS X11 EXAMPLES CUPS GUI DEBUG # disable profiling, unless you like 1hr compile times MK_PROFILE=no # default version to use when certain ports are pulled in as dependencies # ..notice LibreSSL :-) DEFAULT_VERSIONS+= ssl=libressl python=2.7 python2=2.7 python3=3.5 pgsql=9.6 php=7.0 ruby=2.3 perl=5.24 lua=5.1

Now we can install vim:

cd /usr/ports/editors/vim && make install clean

You will be prompted to select some compile-time options before the package is built and installed for you. You can search available packages by running make search name=$PACKAGENAME in /usr/ports. I usually have at least the following installed on my servers:


Hardware Configuration

There are a few easy modifications we can make to improve FreeBSD's performance on modern hardware. If you're using a solid state drive with the UFS file system, it's important to enable TRIM support. You should also set filesystem labels, so you won't have to worry about your disks getting renamed in between reboots (which often happens when you enable AHCI). We can't make these changes while the disks are mounted, so you'll need to reboot to single-user mode. Reboot your machine, and hit S at the bootloader prompt.

Enabling TRIM Support

Once you've booted into the single-user shell, you can get a list of your partitions using gpart show. Here is what I see on my machine:

# gpart show
=>        34  1953525101  ada0  GPT  (932G)
          34        2014        - free -  (1.0M)
        2048  1953521664     1  freebsd-ufs  (932G)
  1953523712        1423        - free -  (712K)

=>       34  500118125  ada1  GPT  (238G)
         34          6        - free -  (3.0K)
         40       1024     1  freebsd-boot  (512K)
       1064  500117088     2  freebsd-ufs  (238G)
  500118152          7        - free -  (3.5K)

So we've got two drives. ada0 is a 1 TB storage drive, and ada1 is an SSD for the OS. The first partition just holds the bootloader, but we'll want to make sure TRIM is enabled on the OS root partition.

# tunefs -p /dev/ada1p2
tunefs: POSIX.1e ACLs: (-a)                                disabled
tunefs: NFSv4 ACLs: (-N)                                   disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 enabled
tunefs: soft update journaling: (-j)                       enabled
tunefs: gjournal: (-J)                                     disabled
tunefs: trim: (-t)                                         disabled
tunefs: maximum blocks per file in a cylinder group: (-e)  4096
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)

Let's go ahead and enable TRIM on this partition.

tunefs -t enable /dev/ada1p2

Setting UFS Labels

While we have everything unmounted, we can set filesystem labels as well:

tunefs -L rootfs /dev/ada1p2
tunefs -L storagefs /dev/ada0p1

Type exit to leave single-user mode and continue the boot process. Once you're back into your system, you can edit /etc/fstab with your new filesystem labels.

/dev/ufs/rootfs / ufs rw 1 1 /dev/ufs/storagefs /storage ufs rw 1 2 # old disk names - replaced with labels above #/dev/ada1p2 / ufs rw 1 1 #/dev/ada0p1 /storage ufs rw 1 2

Loading Useful Kernel Modules

I usually put the following in /boot/loader.conf:

# bootloader prompt timeout (seconds) autoboot_delay="5" # enable temperature sensors coretemp_load="YES" # enable AHCI on modern hardware for better performance ahci_load="YES" # enable hardware accelerated AES (can speed up TLS) aesni_load="YES" # enable asynchronous I/O (big performance gains with NGINX) aio_load="YES" # in-memory file system tmpfs_load="YES" # load PF firewall and the Intel ethernet driver early at boot time pf_load="YES" pflog_load="YES" if_igb_load="YES"

Increasing Disk Read Ahead

Finally, increasing the UFS read ahead value almost always results in better performance. Add the following to /etc/sysctl.conf:


You should reboot your machine after making these changes to make sure you didn't break anything.

Network Settings

Open up /etc/rc.conf to configure your network interfaces. You will need an IP address (and hopefully an IPv6 address) for the machine, as well as a hostname of your choosing. I use dual NICs with a lagg failover interface, so I've included that in the snippet below. This example uses fake IP addresses, you will need real ones!

# choose a hostname for this machine. you did register a domain, right? hostname="beastie.c0ffee.net" # My machine has two interfaces in a failover configuration: # igb0 and igb1 are physical interfaces, lagg0 is a virtual aggregate. # You should disable LRO and TSO if this machine will route packets. ifconfig_igb0="up -lro -tso" ifconfig_igb1="up -lro -tso" cloned_interfaces="lagg0" # Your IPv4 address, netmask, and default gateway go here. # Your hosting provider should provide this info. ifconfig_lagg0="laggproto failover laggport igb0 laggport igb1" defaultrouter="" # IPv6 address and IPv6 gateway go here (if applicable). ifconfig_lagg0_ipv6="inet6 2000:f2a5:a440::2/64" ipv6_defaultrouter="2000:f2a5:a440::1" ipv6_activate_all_interfaces="YES" # If you only had one network interface, then the below would suffice: # ifconfig_igb0="inet -lro -tso" # defaultrouter="" # ifconfig_igb0_ipv6="inet6 2000:f2a5:a440::2/64" # ipv6_defaultrouter="2000:f2a5:a440::1" # ipv6_activate_all_interfaces="YES"

You will need to set your DNS servers in /etc/resolv.conf. For example, if you are using Google's DNS:

nameserver nameserver search c0ffee.net

Also, make sure to add your machine's IP addresses to /etc/hosts:

::1 localhost localhost.c0ffee.net localhost localhost.c0ffee.net 2000:f2a5:a440::2 beastie beastie.c0ffee.net beastie beastie.c0ffee.net

Environment Setup

It's the current year, so you should enable UTF-8 for your locale and charset everywhere. Add the following to /etc/profile:


Also, add the bolded lines below to your default login class in /etc/login.conf:

default:\ :passwd_format=sha512:\ :copyright=/etc/COPYRIGHT:\ :welcome=/etc/motd:\ :setenv=MAIL=/var/mail/$,BLOCKSIZE=K:\ :path=/sbin /bin /usr/sbin /usr/bin /usr/local/sbin /usr/local/bin ~/bin:\ :nologin=/var/run/nologin:\ :cputime=unlimited:\ :datasize=unlimited:\ :stacksize=unlimited:\ :memorylocked=64K:\ :memoryuse=unlimited:\ :filesize=unlimited:\ :coredumpsize=unlimited:\ :openfiles=unlimited:\ :maxproc=unlimited:\ :sbsize=unlimited:\ :vmemoryuse=unlimited:\ :swapuse=unlimited:\ :pseudoterminals=unlimited:\ :kqueues=unlimited:\ :umtxp=unlimited:\ :priority=0:\ :ignoretime@:\ :umask=022:\ :charset=UTF-8:\ :lang=en_US.UTF-8:

You'll need to rebuild the login database after you edit that file:

cap_mkdb /etc/login.conf

You should also set your timezone. I'm on the east coast, so I use America/New_York. Modify according to your location.

cp /usr/share/zoneinfo/America/New_York /etc/localtime

Tweaking Network Performance

In my experience, the default TCP settings of the FreeBSD kernel yielded very poor network performance. My server has a fairly fast 1 Gbps uplink, but the majority of my traffic must travel all the way across the country to the east coast (about a 100ms round-trip-time). My biggest problem involved TCP Slow Start, the algorithm that initially increases the throughput of TCP connections. I could eventually max out my server's network connection, but it would take 15 minutes or more for the transfer speed to ramp up.

To get decent throughput, I had to tweak a fair amount of various kernel options and sysctls. Most of my inspiration came from this awesome Calomel guide and a lot of trial and error. If you have a different network topology, you may have to modify some of these values and see what works best for you.

First, let's enable some boot-time kernel options in /boot/loader.conf.

# Load the H-TCP algorithm. It has a more aggressive ramp-up to max # bandwidth, and is optimized for high-speed, high-latency connections. cc_htcp_load="YES" # accept filters allow the kernel to buffer certain incoming connections # until a complete request is received (such as HTTP headers). This can # reduce the number of context switches required by the CPU. accf_http_load="YES" accf_data_load="YES" accf_dns_load="YES" # The hostcache is used to "grade" the throughput of previous connections. # Calomel says that disabling it can increase throughput on connections # incorrectly marked as slow. I didn't notice much difference either way. net.inet.tcp.hostcache.cachelimit="0" # This is the network interface queue length. According to this commit, # the default value of 50 is far too low. Calomel recommends 2x the value # of hw.igb.txd, which has worked well for me. net.link.ifqmaxlen="2048" # Enables a faster but possibly buggy implementation of soreceive. I # haven't had any problems with it. net.inet.tcp.soreceive_stream="1" # FreeBSD sets an artificial limit on the number of packets an Intel card # can process per interrupt cycle (default 100). This is almost totally # useless on modern hardware. -1 means no limit. hw.igb.rx_process_limit="-1"

You'll have to reboot your machine for these changes to take effect.

The rest of the network options can be tweaked on the fly using sysctl:

# soacceptqueue is the kernel's backlog queue depth for accepting new TCP # connections. A larger value should prevent clients from being dropped # during sudden bursty periods at the expense of more RAM and CPU load. kern.ipc.soacceptqueue=1024 # (default 128) # maxsockbuf is the max amount of memory that can be allocated to a socket. # In practice, this value determines the TCP window scaling factor - the # number of bytes that can be transmitted without requiring an ACK. If our # server is not under heavy load, we want a large scaling factor, because # we can transmit packets as fast as the receiver can process them. # # The default maxsockbuf value (2MB) will result in a scaling factor of 6, # which is ideal for a low-latency gigabit network. # # However, my server is on the west coast, on a gigabit connection with # 100ms of latency to my hometown on the east coast. A scaling factor of 8: # # 2^8 x 65,535-byte IP packet size = 16,776,960 bytes # # happens to saturates my connection: # # 16,776,960 bytes * 8 / .1 sec latency / 10^9 = 1.3421568 Gbps # # You remember Bandwidth Delay Product from networking class, right? :-) # # A maxsockbuf value of 8MB will yield a scaling factor of 8. To see # how this is derived, you can check /sys/netinet/tcp_syncache.c. # kern.ipc.maxsockbuf=8388608 # sendspace and recvspace are the network buffer sizes *initially* allocated # to each TCP connection. Bandwidth can be improved by increasing the buffer # size at the cost of using more kernel memory per connection. Saturating my # gigabit connection with 100ms latency would require: # # 1000 megabits * .1 sec / 8 bits = 12.5 MB # # However, just 80 simultaneous connections would immediately consume a # GIGABYTE of RAM! Most of the traffic to my server is for small, static # HTML pages and some SSH connections, so I've set a smaller default size: # # 256 KB = 256 * 1024 bytes = 262,144 bytes # # The kernel will allocate more memory as needed (at a very slight # performance hit) for the occasional full-throttle transfer of large # files. # net.inet.tcp.sendspace=262144 # (default 32768) net.inet.tcp.recvspace=262144 # (default 65536) # sendbuf_max and recvbuf_max control the maximum send and recv buffer sizes # the kernel will ever allocate for a single TCP connection. I set mine to # 16 MB, which is slightly higher than the 12.5 MB I calculated above. This # should let me to saturate my 100ms connection, as well as leave some # wiggle room to saturate even higher-latency clients. # # You should probably make sure these values are at least as large as # maxsockbuf. # net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 # sendbuf_inc and recvbuf_inc control the increments by which the kernel # increases sendspace and recvspace to sendbuf_max and recvbuf_max, # respectively. Higher values will cause fewer memory allocations, but may # result in wasted buffer space. # net.inet.tcp.sendbuf_inc=32768 # (default 8192) net.inet.tcp.recvbuf_inc=65536 # (default 16384) # increase the localhost buffer space, which may help localhost-only servers # more efficiently move data to network buffers. net.local.stream.sendspace=16384 # default (8192) net.local.stream.recvspace=16384 # default (8192) # increase the raw IP datagram buffers to the MTU for the localhost # interface (2^14 bytes = 16384). Thanks, Calomel! net.inet.raw.maxdgram=16384 net.inet.raw.recvspace=16384 # *** these two settings gave me the most drastic improvement: *** # TCP Slow Start gradually ramps up the data transmission rate until the # throughput on the network path has been determined. TCP Appropriate Byte # Counting (ABC) allows the kernel to increase the window size # exponentially. According to Calomel, with maxseg set to the default of # 1460 bytes, setting abc_l_var to 44 allows an increase of about 64 KB per # step, which happens to be the receive buffer size of most hosts that don't # support window scaling. # net.inet.tcp.abc_l_var=44 # (default 2) # The TCP initial congestion window determines the *initial* amount of data # that can be sent over the network before requiring an ACK from the other # side. The Slow Start algorithm will improve this value over time. A larger # initcwnd will speed up short, bursty connections. Google recommends 16, # but according to Calomel, you should also test 44. net.inet.tcp.initcwnd_segments=44 # (default 10) # Maximum Segment Size (MSS) specifies the largest amount of data that can # be placed in a single IPv4 TCP segment. This value is usually equal to: # # MTU (usually 1500) - 20 byte IPv4 header - 20 byte TCP header = 1460 # # If you have net.inet.tcp.rfc1323 enabled (it is by default on FreeBSD), # then TCP hosts can negotiate timestamps which increases the TCP headers # by 12 bytes. So in the case we'll use 1460 - 12 = 1448 bytes. # net.inet.tcp.mssdflt=1448 # (default 536) # The Minimum MSS specifies the smallest amount of data we will send in a # single IPv4 TCP segment. RFC 6691 requires a minimum value of 576 bytes. # Subtracting a 20 byte IP header and 20 (or 32, see above) byte TCP header # gives us: # # 576 (minimum MTU) - 20 byte IPv4 header - 32 byte TCP header = 524 bytes. # net.inet.tcp.minmss=524 # (default 216) # use the H-TCP algorithm that we enabled in /boot/loader.conf above. net.inet.tcp.cc.algorithm=htcp # Enable H-TCP's adaptive backoff optimization, which increases buffer # efficiency along the network path. net.inet.tcp.cc.htcp.adaptive_backoff=1 # Enable H-TCP's RTT scaling optimization, which increases fairness between # flows with different RTTs. net.inet.tcp.cc.htcp.rtt_scaling=1 # RFC 6675 improves TCP Fast Recovery when combined with SACK (which is # enabled by default on FreeBSD - net.inet.tcp.sack.enable) net.inet.tcp.rfc6675_pipe=1 # Disabling syncookies will give us more TCP features like window scale and # timestamps at the expense of making us more vulnerable to DoS attacks. net.inet.tcp.syncookies=0 # disable the TIME_WAIT state for the localhost interface, this should # result in localhost sockets being freed more quickly. net.inet.tcp.nolocaltimewait=1 # TSO (and LRO) should be disabled on machines that forward packets. I use # OpenVPN, sometimes, so I've disabled TSO here. These options can also be # disabled in /etc/rc.conf using the `-tso -lro` options. net.inet.tcp.tso=0 # these two values control the number of frames the NIC will accept before # firing an interrupt. If the queue fills up and the machine is overloaded, # packets will be dropped. Increasing this value should mitigate packet loss # in case of a storm of short, bursty packets, but if # net.inet.ip.intr_queue_drops remains greater than 0, you probably just # need better hardware. # net.inet.ip.intr_queue_maxlen=2048 # (default 256) net.route.netisr_maxqlen=2048 # (default 256) # Disable Intel's hardware-based ethernet flow control. Instead we will rely # on TCP which is peer-based and more fair to each flow. dev.igb.0.fc=0 # (default 3) dev.igb.1.fc=0 # (default 3)

Run the following command to update the kernel with the new values:

sysctl -f /etc/sysctl.conf

LibreSSL and OpenSSH

In the Building Ports section above, we set our default openssl implementation to LibreSSL. LibreSSL is a fork of OpenSSL initiated by the OpenBSD developers after the heartbleed bug was discovered. It aims to be a more secure, modern, and less crufty replacement for OpenSSL.

You can check the wiki page for details about running LibreSSL on FreeBSD. Currently, OpenSSL is still the default implementation in the base system, but you can build almost all ports using LibreSSL without any issues.

The base SSH daemon will continue to use the base OpenSSL. To use a more up-to-date, upstream build with LibreSSL, you can use the security/openssh-portable port.

cd /usr/ports/security/openssh-portable
make install clean

The default build options are fine. I usually enable LDNS so I can get DNS fingerprint verification.

Here are the options I set in my sshd_config. Many of them are taken from Mozilla's OpenSSH security guidelines. At the very least, you'll want to set a non-default port for SSH unless you want Chinese botnets bruteforcing logins 24/7.

# ANYTHING other than port 22! Port 15522 # explicitly disable the less-secure protocol 1 Protocol 2 # supported HostKey algorithms by order of preference. # I commented out the other ones. HostKey /usr/local/etc/ssh/ssh_host_ed25519_key HostKey /usr/local/etc/ssh/ssh_host_rsa_key HostKey /usr/local/etc/ssh/ssh_host_ecdsa_key KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256 Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com PermitRootLogin no StrictModes yes IgnoreRhosts yes # only accept pubkey-based authentication HostbasedAuthentication no PasswordAuthentication no PermitEmptyPasswords no ChallengeResponseAuthentication no AuthenticationMethods publickey X11Forwarding no UsePrivilegeSeparation sandbox Subsystem sftp /usr/local/libexec/sftp-server # only these whitelisted users may connect AllowUsers joeuser janeuser

Finally, you'll need to disable the base OpenSSH daemon and enable the new one we installed from ports:

#sshd_enable="YES" openssh_enable="YES"

Now start the new SSH server:

service openssh start

Since we have disabled all login mechanisms except for pubkey-based authentication, make sure you copy your public keys to your ~/.ssh/authorized_keys file on your server. I recommend ed25519 keys, as they are much faster and arguably more secure than RSA. You can generate ed25519 keys on your client machines with the following:

ssh-keygen -t ed25519

When you have successfully logged in over the new SSH port, you can stop the old SSH daemon.

service sshd onestop

Clock Sync with OpenNTP

There have been numerous security advisories related to the base NTP daemon, and I get the feeling that the code is written by scientists rather than sysadmins. I use OpenBSD's NTP daemon, available at net/openntpd.

cd /usr/ports/net/openntpd
make install clean

First, make sure the base NTP daemon isn't running:

service ntpd stop

Then enable the new OpenNTP daemon in /etc/rc.conf. Make sure the base NTP daemon is disabled:

#ntpd_enable="YES" openntpd_enable="YES" # the -s flag instructs OpenNTP to set the clock immediately at startup openntpd_flags="-s"

Start the new NTP service:

service openntpd start

Securing the PF Firewall

PF, the OpenBSD firewall, is included in the FreeBSD base install. People argue about performance between PF and IPFW, but I think PF's syntax is the easiest of any firewall in existence. We'll use a simple PF setup—just blocking all inbound connections except to specific services we allow.

The PF configuration lives at /etc/pf.conf:

# the external network interface to the internet ext_if="lagg0" # port on which sshd is running ssh_port = "15522" # allowed inbound ports (services hosted by this machine) inbound_tcp_services = "{auth, http, https, " $ssh_port " }" inbound_udp_services = "{dhcpv6-client,openvpn}" # politely send TCP RST for blocked packets. The alternative is # "set block-policy drop", which will cause clients to wait for a timeout # before giving up. set block-policy return # log only on the external interface set loginterface $ext_if # skip all filtering on localhost set skip on lo # reassemble all fragmented packets before filtering them scrub in on $ext_if all fragment reassemble # block forged client IPs (such as private addresses from WAN interface) antispoof for $ext_if # default behavior: block all traffic block all # allow all icmp traffic (like ping) pass quick on $ext_if proto icmp pass quick on $ext_if proto icmp6 # allow incoming traffic to services hosted by this machine pass in quick on $ext_if proto tcp to port $inbound_tcp_services pass in quick on $ext_if proto udp to port $inbound_udp_services # allow all outgoing traffic pass out quick on $ext_if

You should check the syntax of your PF configuration before enabling the firewall:

pfctl -vnf /etc/pf.conf

If all is well, enable the PF firewall daemon:


And then start the service. Your SSH session might get reset. (Note: it may be a good idea to have a serial console session open before you enable the firewall, in case you accidentally lock yourself out.)

service pf start

You should now have a basic firewall configuration to protect your server from unintended open ports. If you are feeling more paranoid, you can restrict outgoing traffic as well. Remember that PF processes rules from top to bottom—the last matching rule wins (with the exception of rules with the quick modifier: those rules match immediately, and no further matching is attempted).

If you are following my self-hosting guide, we will be coming back to this PF configuration frequently as we enable new services.


If you've followed everything in this guide, you should have a relatively modern and secure FreeBSD server that you can safely connect to the internet. Check back to this page as new FreeBSD versions get released. I will continue to update this post with what I consider to be Best Practices™️️ as the FreeBSD ecosystem evolves.