I was asked on twitter today to explain some recent work I have done in the multi-factor authentication department. I can write this article because I have recently released my work under the title “secure_thumb.”
While many 2FA and beyond methodologies implement SMS as a factor in authentication, I was uncomfortable sending factors over the wire. Over the years I have had one actual instance where a machine got rooted and the SSH private key (although passphrase protected) was stolen from said machine. At the time, I did not have PGP set up, so I had no way of getting a new key trusted by the FreeBSD community. We quickly blacklisted the stolen key and a review of recent commits was performed to check for any malicious activity. Another developer was sent to my house to verify government-issued ID before a new key was generated offline and injected into the cluster through a trusted channel.
From that incident many years ago, I vowed to never let a single factor stand between me and the FreeBSD cluster again (that factor being a passphrase typed at a command-prompt to load an SSH private key that gave access to the FreeBSD cluster and commit access to change any FreeBSD source code).
Since I was actively writing and maintaining the FreeBSD boot loader at the time, there was some mild concern that malicious code could have been injected into places that few others understood (the loader code that I write/maintain is in a language called Forth; a language which causes most to run the opposite direction and is known for being quite unreadable). I did a top-down review of every line of Forth at that time to put minds at-ease (added benefit, I then shortly after rewrote several of the inefficient structures and solved some inherited issues that were long-in-the-tooth as they say).
Post-incident, my SSH private key to the cluster remained secured on a thumb drive encrypted by GELI — an in-kernel FreeBSD system that allows for the encryption of entire filesystems with a myriad of options. We went from a single-factor to 3-factor in a single day, but we will see how a scare later resulted in the development of 4- and 5-factor authentication that is used today.
For many years I used 3FA implemented as a physical USB thumb drive kept in a secure location, offline, encrypted, holding SSH private keys protected with a unique passphrase. One day, it was believed that I lost this key. When it became obvious that we were not going to find it in any time soon, I had the cluster revoke my keys. This time, however, I had PGP properly set up and could quickly re-establish trust by sending a new public key in a PGP encrypted message to the cluster admin team.
Every time I regenerate my FreeBSD.org private keys (years apart) I take the opportunity to think about security. This time I had an entire framework built around my 3FA solution, and I envisioned a scenario where the person that may have found my thumb drive would not be able to decrypt the contents even if they had stolen the passphrase through keystroke logging. My 4FA solution was born.
Since FreeBSD GELI allows you to have multiple keys (which are essentially concatenated together to form one ginormous key), I built a new framework which creates a key (we’ll call this a trust certificate) for the host or hosts that you want to allow decryption. Thus, only if an attacker were able to purloin both my GELI passphrase using a keystroke logger and steal the GELI keyfile from trusted host will they be able to mount the volume. Those 2 factors plus the physical locality of the thumb drive (and being mostly offline), plus the passphrase required to use the SSH private key, altogether form 4FA.
Where does the fifth factor come into play?
Physically locking USB thumb drives that require a code to unlock before they will yield their plug to a computer. I have seen 3 such mechanical devices (e.g., the Cryptex Round Compass Lock by SDI) and a couple digital ones with a keypad/LCD on the thumb drive.
The secure_thumb framework for managing 4FA using securely-stored and encrypted SSH private keys, see https://FrauBSD.org/secure_thumb
As an added bonus, the framework creates one unencrypted partition to contain trust-management software (*cough* shell scripts that make sure the host is trusted and is not sniffing keystrokes ala DTrace for example) and scripts to automate the mounting of the GELI partition. But wait, there’s more …
The framework creates a FreeBSD slice to allow the creation of multiple encrypted partitions each with a different key and trust certificate. You can make a USB thumb drive that has 7 encrypted partitions, each with a different passphrase, and with the trust certificates for each partition sent to different people. The net effect would be a thumb drive that can be used by 7 people but each person can only open their partition.
There are other hidden features, such as the ability to dynamically resize the encrypted slice (both grow and shrink) but you can also add 2 additional partitions to it for making it look like it’s just a drive for another OS with some random unencrypted data on it for TSA to peruse.
At $work, we have been looking at Nginx Plus because unlike the free open-source version of Nginx, Plus has load-balancing features. We’re pretty excited about that because it means we could potentially replace our reliance on HAProxy. Below is a simple graphic showing the potential infrastructure change.
With an nginx plus license, you can eliminate the need for haproxy
The next thing we had to do was determine which platform we would be using for our test of Nginx Plus. That ended up being limited to just two choices based on our constraints — must be physical hardware and must support HTTP/2 and ALPN.
While the system requirements page doesn’t say this directly, we were informed that the only configuration currently supporting HTTP/2 and ALPN on physical hardware is FreeBSD 11.0 and Ubuntu 16.04 LTS. Others such as CentOS were disqualified because, for example, the OpenSSL version was too old to support ALPN.
With a little bit of work in massaging our PXE server (resulting in a complete rewrite of pxe-config for FreeBSD deployment automation), we had the FreeBSD systems deployed in a matter of days. The generation of a Debian Installer preseed for Ubuntu proved to be far more time-consuming and challenging, but after a week we had the Ubuntu systems deployed as well.
We then registered for a free 30-day trial of Nginx Plus and went through the installation instructions for each Operating System. The steps for FreeBSD and Ubuntu are similar wherein you configure your system to add their package server and use the standard package utilities (pkg on FreeBSD and apt-get on Ubuntu) to install a binary package.
With FreeBSD 11.0-RELEASE-p1 and Ubuntu 16.04 LTS on identical servers in the same rack/switch, we were ready to do some benchmarking to help us determine whether Nginx Plus can deliver equitable performance in a load-balanced array compared with an haproxy cluster sitting in front of the free and open-source version of Nginx.
To generate the right type of load for our performance benchmark, we are using hey by Jaana B. Dogan. Below is the usage statement from hey -h:
Usage: hey [options...] <url>
Options:
-n Number of requests to run. Default is 200.
-c Number of requests to run concurrently. Total number of requests cannot
be smaller than the concurrency level. Default is 50.
-q Rate limit, in seconds (QPS).
-o Output type. If none provided, a summary is printed.
"csv" is the only supported alternative. Dumps the response
metrics in comma-separated values format.
-m HTTP method, one of GET, POST, PUT, DELETE, HEAD, OPTIONS.
-H Custom HTTP header. You can specify as many as needed by repeating the flag.
For example, -H "Accept: text/html" -H "Content-Type: application/xml" .
-t Timeout for each request in seconds. Default is 20, use 0 for infinite.
-A HTTP Accept header.
-d HTTP request body.
-D HTTP request body from file. For example, /home/user/file.txt or ./file.txt.
-T Content-type, defaults to "text/html".
-a Basic authentication, username:password.
-x HTTP Proxy address as host:port.
-h2 Enable HTTP/2.
-host HTTP Host header.
-disable-compression Disable compression.
-disable-keepalive Disable keep-alive, prevents re-use of TCP
connections between different HTTP requests.
-cpus Number of used cpu cores.
(default for current machine is 8 cores)
-more Provides information on DNS lookup, dialup, request and
response timings.
NOTE: Although our requirements include HTTP/2 and hey has the -h2 flag to enable HTTP/2, our performance benchmarks will be using HTTP/1[.1] because our current edge infrastructure to which we can make comparisons does not yet support HTTP/2.
The command that we used to test the performance of our setup is as follows:
hey -n 3000 -c 300 -m GET -disable-keepalive <url>
This asks hey to perform a total of 3000 HTTP/1[.1] GET requests for <url> with up to 300 concurrent requests.
When <url> points to our vanilla Ubuntu 16.04 LTS test box running Nginx Plus, the results are as follows:
The hey command, hammering a vanilla Ubuntu 16.04 LTS server running Nginx Plus
When <url> points instead to our vanilla FreeBSD 11.0-RELEASE-p1 test box, the results are as follows:
The hey command, hammering a vanilla FreeBSD 11.0-RELEASE-p1 server running Nginx Plus
Focusing on the Summary Total, we can see that vanilla FreeBSD takes 3x longer than Ubuntu and something is causing high response times in FreeBSD.
SPOILER: After much digging, it was discovered that the nginx binary for Nginx Plus was linking against an OpenSSL that was compiled from ports without the ASM optimizers (highly optimized Assembly routines for speeding up calculations on supported CPU architectures).
The instructions for installing Nginx Plus on FreeBSD include “pkg install nginx-plus” and this brings in security/openssl from ports instead of using the OpenSSL that comes with the FreeBSD base Operating System. This is generally a good thing because ports are updated more frequently than base which helps keep Nginx Plus up-to-date with the latest OpenSSL.
The standard UNIX utility ldd shows us that /usr/local/sbin/nginx (as-installed by the nginx-plus package) links not against the system’s /usr/lib/libssl.so.8 but instead the non-base (read: ports) version located at /usr/local/lib/libssl.so.9
However, as we will see in the below photo, both the system OpenSSL in /usr/bin and the ports OpenSSL in /usr/local/bin are the same version compiled on the same calendar day (despite having different shared library suffixes).
Nginx Plus links to the ports OpenSSL
Though the two versions of OpenSSL appear to be the same, they are actually quite different. When OpenSSL is compiled with ASM optimizations, it can take full advantage of AES-NI and PCLMULQDQ, two important CPU instructions that increase the efficiency of cryptographic calculations.
Testing OpenSSL for AES-NI and PCLMULQDQ to ensure ASM optimizations in support of CPU based crypto
The OPENSSL_ia32cap environment variable is a bit-mask of OpenSSL capabilities which allows us to disable AES-NI and PCLMULQDQ. Combining the values of ~0x200000000000000 (disable AES-NI) and ~0x200000000 (disable PCLMULQDQ) to get ~0x200000200000000, we can disable both AES-NI and PCLMULQDQ for individual runs of “openssl speed“.
In the below two commands, if your CPU supports AES-NI and your OpenSSL has been compiled with ASM optimizations, the first command will be many times faster than the second (wherein optimizations are disabled if available).
As one might expect, the OpenSSL in /usr/bin shows a huge performance increase when AES-NI is not disabled. It was quite the shock to find that the ports OpenSSL in /usr/local/bin showed no differences in performance between the two commands.
We, the FreeBSD committers, took a look at the security/openssl port and discovered that it did not enable ASM optimizations by default at the time the binary packages were last compiled for FreeBSD 11. So I worked with the maintainer of port to fix that for the next time the packages get recompiled.
The next step is to determine the impact that an un-optimized OpenSSL has on our hey tests. The FreeBSD dynamic linker supports configurable dynamic object mapping through libmap.conf(5), so it is a fairly simple matter of telling /usr/local/sbin/nginx to use a different OpenSSL.
Creating /usr/local/etc/libmap.d/nginx.conf with the following contents will cause nginx to use the OpenSSL libraries that came with the base Operating System:
After creating this file and restarting nginx with “service nginx restart“, the hey performance tests now show FreeBSD ahead of Ubuntu in a head-to-head test.
To better illustrate the effects that the unoptimized OpenSSL had on the hey benchmarks, I wrote a utility that generates JSON from the output.
NOTE: While there are many ways to benchmark, this test focused on “time to completion” for 3000 requests with up-to 300 concurrent. The JSON generated depicts the non-linear approach toward completion.
Wrapper script for hey named hey_genlog for generating a log capable of being converted into JSON:
#!/bin/sh
############################################################ IDENT(1)
#
# $Title: Script to generate statistics from hey against a host $
# $Copyright: 2017 Devin Teske. All rights reserved. $
# $Smule$
#
############################################################ INFORMATION
#
# Statistics are logged to stdout. Use hey2graph to generate JSON.
# JSON is designed for highcharts/highstock API.
#
############################################################ CONFIGURATION
#
# hey utility from https://github.com/rakyll/hey
#
HEY=hey
#
# File to request
#
FILE=/aud1.m4a
#
# Total number of requests to perform
#
TOTAL=3000
#
# Maximum number of concurrent requests
#
CONCURRENT=300
#
# QoS rate limiting
# NB: Set to NULL to disable rate limiting
#
#RATE_LIMIT=1 # seconds
RATE_LIMIT= # seconds
#
# Should we use Secure-HTTP (https)?
# NB: Set to NULL to disable https
#
SECURE=1
############################################################ GLOBALS
pgm="${0##*/}" # Program basename
#
# Global exit status
#
SUCCESS=0
FAILURE=1
#
# Command-line arguments
#
HOST=$1
############################################################ FUNCTIONS
usage()
{
exec >&2
printf "Usage: %s HOST\n" "$pgm"
exit $FAILURE
}
############################################################ MAIN
case "$HOST" in
"") usage ;; # NOTREACHED
*:*) : fall through ;;
*)
if [ "$SECURE" ]; then
HOST="$HOST:443"
else
HOST="$HOST:80"
fi
esac
echo "Performing $TOTAL total requests"
echo "Maximum $CONCURRENT concurrent requests"
set -x
$HEY \
-n $TOTAL \
-c $CONCURRENT \
${RATE_LIMIT:+-q $RATE_LIMIT} \
-m GET \
-disable-keepalive \
http${SECURE:+s}://$HOST$FILE |
awk -v cmd="date +%s.%N" '
BEGIN {
cmd | getline start
close(cmd)
}
/requests done/ {
cmd | getline date
close(cmd)
date = sprintf("%0.4f", date - start)
sub(/^/, date " ")
}
1
' # END-QUOTE
################################################################################
# END
################################################################################
The process of generating JSON graph data was performed for Ubuntu, FreeBSD with ports OpenSSL, and FreeBSD with base OpenSSL. As you can see in the below graph, FreeBSD with base OpenSSL is the fastest with Ubuntu very close behind, and FreeBSD with an unoptimized ports OpenSSL coming in at 3x slower.
Satisfied that we had eliminated the performance issue causing FreeBSD to be 3x slower, we now asked why is Ubuntu slower than FreeBSD?
Intensive throughput benchmarks showed that FreeBSD is capable of reaching 87.1% line-rate while Ubuntu was only capable of 86.5%. Both systems given an Intel 10GE network interface, FreeBSD appears to be utilizing the hardware more efficiently. At the switch, we can see that FreeBSD is 99.1% efficient on 10GE, resulting in a measured 10.3% TCP overhead at the time of testing.
FreeBSD line rate test on Intel 10GE, switch-level throughput graph
The result of our testing is that FreeBSD running Nginx Plus is a suitable replacement for our HaProxy and Nginx topology. You and everyone reading this won’t have to worry about the documented issue with OpenSSL because I worked with Bernard Spil and Allan Jude to get it fixed in the FreeBSD ports tree. The security/openssl port has been updated to enable ASM optimizations by default and fairly soon the binary packages will be rebuilt — until then, you can use the above /usr/local/etc/libmap.d/nginx.conf file to temporarily use the base OpenSSL if you’re unable to update /usr/ports/security/openssl/Makefile and recompile it yourself.
… and I’m probably the last committer to work on sysinstall(8).
Now that introductions are out of the way, let’s meet our topic:
The FreeBSD Installer
If you’ve been living under a rock (or in a cave without Internet access) for the past 3+ years, here’s the highlights for what you’ve been missing in the FreeBSD Installer arena:
+ sysinstall(8) had a good run from 4.x to 8.x
+ 9.0-RELEASE: Introduced bsdinstall(8)[new default installer]
– 9.1-RELEASE: No noticeable changes
+ 9.2-RELEASE:
+ Improved bsdinstall(8) scripting
+ Introduced bsdconfig(8)[bsdinstall helper]
+ Introduced sysrc(8)
Before continuing, let’s take a brief moment to give credit where credit is due:
~23,000 Lines: Legacy sysinstall(8) by Jordan Hubbard NOTE: Including the hard work of countless others (including myself)
~6,000 Lines: New installer bsdinstall(8) by Nathan Whitehorn NOTE: Including many patches from others (including myself)
Meanwhile, I proudly take credit for the following:
~33,000 Lines: New installer companion bsdconfig(8)
~1,000 Lines: System management tool, sysrc(8)
However, exactly what is the future with respect to our Installer? It’s often tempting to sit down and look at where we came from and immediately [with ease] point out any number of short-comings in the replacements. While hindsight is paramount to preserving backward compatibility, we should be ever mindful to avoid simply reproducing a new generation of the same thing.
That being said, and with so much effort being put into a complete rewrite, I believe some new high-level goals and ideas can pave a path avoiding the dreaded bikeshed situation (see bikeshed.com; or perhaps teal.bikeshed.com is your cup of tea).
This is a nuclear reactor style look at the FreeBSD Installer from the top-down. Note however, that as we perform a top-down view, design was performed from the bottom-up. We’ll start our top-down view with pretty pictures and end with the very words and code that inspired this blog entry.
Starting at the top, let’s focus on the User Interface. The UI is still a fairly broad topic, so let’s further focus on a subset of the UI, realtime feedback. Our initial focal point will be the UI management for providing feedback during data input/output operations.
NOTE: To get proper box drawing characters on the system console under FreeBSD 9.0 or higher, you must change the default terminal type in ttys(5) from xterm to cons25. This is easy in FreeBSD 9.2 or higher — execute bsdconfig syscons_ttys, select IBM437 (VGA default), execute init q, then exit, and log back into the system console.
Meanwhile, here’s what legacy sysinstall(8) looks like:
The distExtractAll functionality of FreeBSD sysinstall(8)
As you can see from the above two pictures, bsdinstall(8) (top) is an improvement over sysinstall(8) (bottom). Perhaps the most noticeable change is that bsdinstall(8) presents a single widget to the user, providing feedback on all requested distributions within a single screen; meanwhile sysinstall(8) presents only a single progress bar widget, multiple times in a series.
While this is indeed an improvement, a few things come to mind:
However, perhaps the most important observation should be that both applications operate serially. Both bsdinstall(8) and sysinstall(8) do not proceed to the next distribution until finishing the previous one. Since the data in each distribution is unique, it is not a requirement to unpack them serially but rather, how it’s simply been done (for over 15 years).
When bsdinstall(8) improved the UI to provide a multi-progress widget, the obvious question of concurrent parallelism should be raised. From a UI standpoint, once you have a widget that can display progress for multiple items, the next logical step is to allow the “Pending” items to be processed without delay. In other words, the UI we have today for distfetch, is a great starting point but we can take it further.
For example, view below a prototype that I am developing:
A prototype replacement for FreeBSD bsdinstall(8) distfetch/distextract
NOTE: Both dialog(1) and dialog(3) render ANSI bold as gray on the system console in 9.0 or higher. Compare to the following image, executing the same program via SSH in which bold is exactly what you’d expect.
NOTE: The above was taken with a value of en_US.ISO8859-1 for the LC_ALL environment variable, nicely resulting in comma-separators in the KBytes/sec status line entry. See localeconv(3).
The prototype that I am developing can also replace other [less than informative] areas of the system. For example, the following picture shows how [deprecated] sysinstall(8) told the user that it was downloading the 24MB INDEX file (commonly via FTP):
The configPackages functionality of FreeBSD sysinstall(8)
As you can see above, sysinstall(8) is pretty spartan when it comes to providing any feedback in this area. The replacement, bsdconfig(8), does not yet bring any notable changes (pictured below):
The packages module of FreeBSD bsdconfig(8)
Neither deprecatedsysinstall(8) nor newbsdconfig(8) provide any information when [down]loading packages/INDEX. There’s no progress bar, no indicator that data is moving, and no data/rate info. My prototype application (named fdpv(1)) can improve the user experience here by providing those things (pictured below).
Prototype application for handling I/O streams
However, perhaps the best use for such a tool is to improve the way packages are installed. Pictured below is deprecatedsysinstall(8), installing a single package.
Legacy/deprecated sysinstall(8) installing vim-lite via FTP
Not quite as blunt as the fetching of packages/INDEX; this time at least sporting data/rate information in the status line. However, moving forward and away from sysinstall(8), let’s see what replacement bsdconfig(8) has for us:
bsdconfig(8) packageAdd installing vim-lite
Notice a slight regression in moving forward. Newbsdconfig(8) lacks data/rate information that deprecatedsysinstall(8) provided.
I plan to fix the regression by introducing fdpv(1) as a new back-end for package installation. Here’s what it could look like:
A prototype packageAdd that uses fdpv(1)
NOTE: The above is a Photoshop‘ed image. Concurrency has not yet been programmed.
To better understand the fdpv(1) prototype, here is a picture with annotations showing the layout of information and where it is drawn from (no pun intended).
Inormation layout of fdpv(1) display
As you can see, the fdpv(1) tool is just a conduit. As such, the tool can be used for both legacy package management as well as pkgng (new default package management tool in FreeBSD).
For the sake of discussion regarding package management, notice the re-introduction of the status line (at bottom) to provide data/rate info (enumerating total throughput) as well as concurrent parallel processing of multiple dependencies during the installation of xorg (pictured).
This should dramatically reduce the amount of time required to install packages because independent dependencies, whom already have their own dependencies satisfied, can be installed in-parallel rather than waiting.
Just how many concurrent workers are allowed? Well, that would be up to Grand Central Dispatch (GCD). Programmed to use GCD, the utility should allow transparent scaling, respective to hardware capability and machine load.
Of course, I have to admit to not knowing GCD yet, so I’ve cast a fellow FreeBSD committer to help with that end. I was originally planning on using POSIX Threads aka pthreads and fancy mutex signaling. But Grand Central Dispatch looks much more promising.
What I’ve accomplished thus far (as of tagged-version 0.5) on this project is the following:
Nearly complete manual (man(1) page) documenting syntax of command-line utility and all its flags.
Created test programs that exercise expected usage.
Prepared the code for clang (a requirement to use GCD).
What’s not done however, is the following:
For package management, a command-line utility is inadequate. The main() portion of the utility needs to be translated into a library (libfdpv) allowing the integration of libpkg.
Concurrent handling of file arguments via Grand Central Dispatch.
Actual handling of I/O (reads/writes on file paths).
Those things need to be finished before we can import this new tool to the FreeBSD base system. Once those things are finished and fdpv(1) is in the base system, we can then start rewriting bsdinstall(8) and bsdconfig(8) to use it, providing enhanced I/O data feedback where we need it most (the installation of the system and package maintenance, respectively).
Pure bonus is the ability to work in X11 (shown below).
Various dialogs showing fdpv(1) in X11 mode
This matches bsdconfig(8)‘s already built-in ability to work entirely in X11, meanwhile bsdinstall(8) has to slowly be reworked to support X11 (by introducing new tools such as fdpv(1) to replace the existing C components that currently support onlydialog(3)).
The code repository for fdpv(1) is at the below address:
NOTE: There is a “Download GNU Tarball” link at the bottom as well as a pull-down menu for selecting a “Sticky Tag”, allowing easy download of tagged revisions.
We’ve had an in-depth look at Data I/O Feedback, in the next blog post we will be switching over to another User Interface issue, Internationalization.
While FreeBSD-9.1 has recently been released, I find myself still working in FreeBSD-8 (and at the time of this writing, 8.4 is soon to be released). This article is based on work performed in FreeBSD-8.1 (only because we froze our development environment for several years to prove the technology that I’m going to talk about in this article).
In FreeBSD-8 there is a new feature for “jails” that is not enabled by default (for base information on standard FreeBSD jails, see the Jail Section of the FreeBSD Handbook).
The new optional feature is called “VNET” and it allows each jail to have a private networking stack. However useful and exciting this may sound, these VNET jails require you to “Bring Your Own Network Interface.”
The VNET feature allows you to move network interfaces in/out of the view of a jail. When a network interface is moved into the view of a jail, it is no longer visible to the host of said jail.
A VNET jail starts with an empty network stack.
For example, you can create an ad hoc persistent VNET jail using the following command (requires VIMAGE option enabled in kernel — more on that later):
NOTE: If your kernel is not compiled with the options VIMAGE line, then you’ll get an error of jail: unknown parameter: vnet).
This jail is now a special jail, called a VNET jail. In this jail, ifconfig shows a private network stack which is currently empty except for an unconfigured local-loopback interface (lo0).
You can now move network interfaces into that jail using the below ifconfig syntax from the host:
ifconfig <interface_name> vnet <jail_id>
Meanwhile, you can move the network interface back out-of that jail using the below syntax from the host:
ifconfig <interface_name> -vnet <jail_id>
Notice the -vnet to recover an interface versus vnet to relinquish an interface to a VNET jail.
Where <interface_name> is something like fxp0, em0, bge0, igb0, etc. and <jail_id> is the value shown in the “JID” column of the output produced by the jls utility (executed without arguments).
However, the usefulness of this is still very limiting. With the VNET feature alone, all one is afforded is the ability to move whole network interfaces into and out-of VNET jails. When one does this, the jail and host do not share the network interface but rather the host can no longer see (using ifconfig for example) nor use the network interface.
To alleviate this limitation, a bridging technology is required to allow the host and VNET jail to share a network interface.
Enter netgraph(4), another optional component in FreeBSD — graph based kernel networking subsystem — publicly available since FreeBSD-3.4. By adding a few NETGRAPH options to the kernel we can provide the VNET jails a solid bridging layer able to provide the jails fresh/functional network interfaces.
However, we’re still not to a point of usefulness because the following gaps still exist:
How to automate the bootup, shutdown, and management of VNET jails?
How to automate the creation, destruction, and bridging of netgraph devices?
How to make sure bridged interfaces have unique MAC addresses?
How to allow multiple VNET jails to have the same root directory? (considering each sets their own IP from rc.conf(5))
Enter vimage, software I’ve written to tie VNET jails together using netgraph.
NOTE: My “vimage” boot/management script (which lives in /etc/rc.d and oft-executed via “service”) should not be confused with the “vimage” utility from the VirtNet project (imunes.tel.fer.hr/virtnet/).
My vimage is a fork of the jail rc.d script — rewritten to work only with VNET jails and help you harness the most flexibility in network topology and jail structure.
HINT: If you are familiar with the IMUNES project (http://www.imunes.tel.fer.hr), both their work and my own performance-test results were factored into the ultimate choice to settle on netgraph for our bridging needs.
NOTE: In the future, when VIMAGE is compatible with SCTP, you will not need the nooptions SCTP line.
UPDATE: A known issue is that the non-defaultoptions IPFILTER conflicts with VIMAGE in stable/9 (and possibly HEAD too) in that a kernel panic occurs during boot. A backtrace shows the trap originating from fr_resolvenic() in sys/contrib/ipfilter/netinet/fil.c. My current recommendation for running a kernel with both Firewall abilities and VIMAGE is to use the IPFIREWALL family of options and ipfw(8).
Compile your kernel and boot it. Next, grab the FreeBSD package for vimage-1.4 and use the following command to install 2 files (/etc/rc.d/vimage and /etc/rc.conf.d/vimage):
pkg_add vimage-1.4.tbz
Now we’re ready to build a jail. Personally, I use my own “jail_build” script to build jails from binary releases. You can get jail_build from my SourceForge page:
NOTE: If you are planning on running i386 jails under an amd64 kernel, be advised that you may need a patch to your amd64 kernel to solve a problem with applying a default gateway using the 32-bit route command (as discussed here).
Currently jail_build works with any release, 8.x and older (sorry, no 9.x support yet).
# Download "base" directory from FTP to local "."
# Download "doc" directory from FTP to local "."
# Download "dict" directory from FTP to local "."
# Download "games" directory from FTP to local "."
# Download "info" directory from FTP to local "."
# Download "manpages" directory from FTP to local "."
# Download "proflibs" directory from FTP to local "."
# Download "kernels" directory from FTP to local "."
TIP:jail_build will automatically probe for /usr/repos/*-{RELEASE,STABLE,CURRENT}
NOTE: You can certainly run older releases as a jail, but generally speaking you should not run newer releases than what your host is running. So if you build an 8.3-RELEASE jail, you should have at least an 8.3 kernel or higher.
Executing jail_build brings up a menu of repositories to select from (living in /usr/repos) and allows you to select one before prompting you to enter the directory you wish to unpack the contents to (creating a new jail root directory). The default directory is /usr/jail/<jail_hostname> (change <jail_hostname> to the fully qualified hostname of the jail — this is suggested for easy tracking but feel free to make this whatever you like).
NOTE: You don’t have to use my jail_build technique to build jails, in fact there is a long-standing tradition of populating jails from source-code. For details on how to build a jail from source, see the Creating and Controlling Jails section of the FreeBSD Handbook.
After you’ve created a jail, it’s time to configure it as a vimage.
In the host’s /etc/rc.conf file, add the following:
vimage_enable="YES" # Set to NO to disable starting of any vimages
vimage_list="vj1" # Space separated list of names of vimages
vimage_vj1_rootdir="/usr/jail/vj1" # vimage's root directory
vimage_vj1_hostname="vj1" # vimage's hostname
vimage_vj1_devfs_enable="YES" # mount devfs in the vimage
This configures your basic vimage (we’ll cover how to give it a network interface in a moment). You can compare this directly to the procedures documented in the Creating and Controlling Jails section of the FreeBSD Handbook.
The same rules that apply to jail_list in the Handbook apply to vimage_list. The methodology is the same because again, /etc/rc.d/vimage is a fork of the documented /etc/rc.d/jail script.
Next, if you only want to move whole network interfaces into the vimage when it is started (and implied, automatically move the interface back out when the vimage is stopped), add the following to the same rc.conf file as above:
vimage_vj1_vnets="igb0" # list of interfaces to give to the vimage
However, the real flexibility comes from using the bridging option based on netgraph. To create a bridged interface for the vimage — leaving the original interface on the host unaffected — add the following instead of the above:
vimage_vj1_bridges="igb0 igb0" # list of interfaces to bridge
In the above example, a single port physical Intel Gigabit network adapter is listed twice — this will create two unique bridged interfaces from the same physical interface. This is not an error and is perfectly valid (imagine simulating a router that runs on two subnets but over the same physical wire).
If you start the vimage at this point (using service vimage start) you can get a list of the bridged interfaces from the jail’s point of view by executing:
jexec vj1 ifconfig
You’ll see (in the bridging example) two network interfaces, one named ng0_vj1 and the other ng1_vj1. The naming convention for bridged interfaces is ng#_NAME where # is a counter that starts at zero and increases by one for each bridge (regardless of the device being bridged) and NAME is the vimage name as seen in vimage_list.
NOTE: Due to an internal limitation, the name of any network interface in FreeBSD cannot exceed 15 characters. For long vimage names, be aware that ng#_NAME will be truncated to be less than 16 characters if necessary.
PRO-TIP: By placing the vimage name within the name of the bridged network interface, it makes it simple to configure multiple vimages to use the same root directory.
The network configuration for the above vimage is configured within the vimage’s own /etc/rc.conf file, example below:
Let’s say you had a second vimage named “vj2” pointed at the same root directory (and therefore the same /etc/rc.conf file). You would then add the following, for example:
Created a jail using either the source-method mentioned in the FreeBSD Handbook or using jail_build and a binary release. PRO-TIP: You can actually get by in testing without building a discrete jail but instead use “/” as the jail root path. This is perfectly valid and acceptable.
Configured the vj1 test vimage in /etc/rc.conf
Configured vj1 to bridge at least one physical interface using vimage_vj1_bridges (in the example above, we use igb0 — twice)
Configured ifconfig_ng0_vj1 and [optionally] defaultrouter in the /etc/rc.conf file within the vj1 root directory.
At this point, the vimage will boot with the rest of the machine and can be controlled after boot with the following syntaxes:
# Stop, start, or restart all vimages
service vimage stop
service vimage start
service vimage restart
# ... versus: Stop, start, or restart just the vj1 vimage
service vimage stop vj1
service vimage start vj1
service vimage restart vj1
When the vj1 vimage starts, the ng*_vj1 network interfaces will automatically be created with unique MAC addresses and moved into the VNET jail before kickstarting the FreeBSD boot process within the jail. As the jail boots, it will automatically configure the network interfaces through rc.conf(5). You can use jls(8) to see the running vimages and you can use jexec(8) to execute processes inside them (like tcsh(1), ps(1), and ifconfig(8)).
PRO-TIP:ps(1) can produce the JID of all running processes with the syntax: ps axopid,jid,command
However, what if you want to SSH into the vimage jail? SSH is not automatically started inside the vimage (and in-fact, only the network services required to get the vimage talking to the net are started). The answer is to add a new configuration line to the host machine’s /etc/rc.conf to configure sshd to be started in each/every jail:
vimage_afterstart_services="sshd" # set new default for all vimages
The default for vimage_after_services is NULL but can be set to a space separated list of services (names of scripts living in /etc/rc.d like sshd). If you wanted to leave this the default and only have the vj1 vimage start sshd, you can instead opt for the following line:
vimage_vj1_afterstart_services="sshd"
So now we have a usable framework for creating multiple vimages that can be SSH’d into as (let’s say) development environments. When you scale this out, netgraph starts to shine. Afterall, netgraph is based on graphs.
Without any additional work, simply by using vimage to produce VNET jails with netgraph bridged interfaces, we can produce graphs of the vimage network topology with the following syntax:
ngctl dot | dot -Tsvg -o vgraph.svg
PRO-TIP: The dot(1) utility can produce more than SVG (like PNG, JPEG, and GIF), but I find SVG to be the most scalable and informative (modern SVG viewers such as latent browsers display much information as tool-tips which I find helpful).
Here are some graphs of different topologies we’ve used over the years:
warden0.jbsd netgraph of bridges for vimage jails folsom netgraph of bridges for vimage jails
It’s worth noting that if an interface is not bridged, it is shown in the “disconnected” cluster. This does not imply that the network interface is unused within FreeBSD — just that it has not been connected to any nodes within the netgraph layer.
That’s it for now, thank you for reading. Comments welcome. Depending on comments, I may do another installment showing the more exotic things you can do with this configuration.
—
Cheers,
Devin
FreeBSD. It is Free (in the truest sense; with a capital-F) and is a descendant of Berkeley Systems Distribution (specifically BSD-Lite v4.2/4.4), which originated from the University of Berkeley, about 2.5 miles away from where I live (though I’m not a student, nor have I ever been). FreeBSD is an amazingly versatile and flexible UNIX operating system with ancestry reaching back decades.
This entry is for those persons already well versed in this well-edified system.
I would like to announce a completely new — written from scratch — boot menu for the FreeBSD operating system. It is today that I release the code that I’ve been developing for nearly 5 years. Code to enhance the level of options provided to the user at each and every boot of this venerable OS.
Historically, the original menu offered by FreeBSD featured our beloved helper-daemon mascot, named “Beastie.” But soon after, this menu was revamped to [a] lay-rest our triton-carrying/sneaker-wearing assistant in favor of the less-controversial textual “FreeBSD” banner and [b] imbue the menu with new options.
Currently, the default boot menu offers booting normally as well as options for the following:
Booting with ACPI support disabled (if you’re running an i386 compatible system)
Booting into “Safe Mode” (implies disabling ACPI)
Booting verbose
Booting into single-user mode
However, one of the things that is not currently possible is the ability to boot into any combination of the aforementioned options. That is to explain, you cannot boot into single-user mode with ACPI disabled but verbosity enabled. The reason for this is obvious as once a menu option has been selected, the system proceeds to boot with those options rather than allowing additional selections to be made. See below.
The Standard FreeBSD Boot Menu
Enter my new menu (pictured below in the following paragraphs), first created in 2006 internally for a proprietary custom installer, now generalized for the rest of the world and being released for all to enjoy. This new loader does not proceed to boot the system after selecting a menu option but rather updates the menu to display the option status, allowing the user to toggle any number of multiple options before pressing ENTER to finally boot with the displayed options.
And for those like myself that balked when Beastie exited stage-left, I have a nice surprise. Some time back, the FreeBSD community held a contest for artists to create a new logo for the FreeBSD project. The results of which can be viewed here: http://www.freebsd.org/logo.html
When the contest ended, I had my fun in running the graphics through some Image-to-ASCII conversion programs without much luck. Months later, I doubled-down my efforts still to no avail, attempting to create an ASCII-art version of the new orb to display at boot time. Six months later, I succeeded in producing a bi-color ASCII-art representation of the “horny orb” logo easily recognizable as the FreeBSD logo. This gem (as well as a B&W copy) have been added to the latest “beastie.4th” module for your viewing pleasure and is enabled by default upon installation.
Here you can lay your eyes on the final edition of the first-offering being made to the public today:
New FreeBSD Boot Loader Menu
Installing the new boot loader menu couldn’t be easier. After downloading the FreeBSD package from http://druidbsd.sf.net/, use the pkg_add(1) command to install the package, and finally reboot.
Other additions that you will notice are that the replacement menu does not load right away. Another enhancement is that there is a 2 second delay before launching the menu where the user can press Ctrl-C to escape the menu, allowing immediate booting or escape to the loader prompt. Something that the current menu does not allow for. See below. Loading the New Menu
The new menu is a system of ANS/FICL Forth (a reverse-polish stack-based language) modules that extend (but do not replace) the standard FreeBSD boot loader (/boot/loader). An interesting note about this package is that it will backup any files that will be replaced during installation to the file /var/backups/loader_menu.backup.tgz. Should you decide that you do not like the new and improved boot loader, you can uninstall it, returning your system back to the original boot loader by using the pkg_delete(1) utility (the backup will automatically be restored upon pkg_delete(1)).
With this initial offering, there are only two short-comings which should certainly be livable for most if not all:
1. The menu that we are replacing only conditionally displays the option to disable ACPI support (based on whether you are running an i386 compatible system), whereas the new loader always presents this option. This should be of no concern unless you’re running a non-i386/amd64 system (such as sparc, pc98, ppc, etc.).
2. The ability to boot into single-user mode by pressing “s” and boot verbose by pressing “v”, etc. is not enabed. The proponents for these options on the FreeBSD-Questions mailing list were clear in that they liked these options because they allowed “one-key” booting into a special mode. However — with the new menu — even if I enabled the use of those keys as shortcuts for toggling these options in the new stateful menu, the user would still have to press an additional key to boot (‘1’ or ENTER). So the primary principle use — one-key booting into a special mode — is no longer capable, unless I add these keys not as togglers for the visible menu items but invisible hot-keys for immediately booting into single/verbose mode. I’ll have to sleep on that (maybe as a concession for power-users as invisible options are obviously less than intuitive).
However, don’t let those misgivings dissuade you. All but the most ardent and stringent of surveyors will find those to be shortcomings and overall I predict your experience will be a rather pleasing one.
Some other nice additions are that if/when you select the option to escape to the loader prompt, you are given a way to get back to the menu (simply type “menu” and press ENTER), allowing you to perform work under the hood and then return to the menu as desired (without losing track of the status of the menu options even). As shown below:
Escaping to the Loader Prompt
Overall, the new stateful nature of this menu over the old makes it far less likely that a user will need to reboot if they hit the wrong key. The only mistake that could result in wasted cycles would be if they prematurely hit #1 to boot or hit #7 to reboot. All other options would simply toggle a menu item (except for #6 to escape to the loader prompt, to which they can immediately return to the menu if desired).
Last, but certainly not least, I’m including a completely rewritten password-checking module to replace the existing one (I couldn’t get the standard one to work). See below.
New Password Checking Module
I hope that you like what I’ve developed, and your feedback is certainly welcome. I’ll be cross-posting this to the FreeBSD mailing lists, announcing in RSS feeds, and updating http://druidbsd.sourceforge.net/ all simultaneously to coincide with this blog-post.
—
Cheers,
Devin
