FreeBSD Installer Enhancements

I work on FreeBSD as the current maintainer/author of the following [released] works:

Now that introductions are out of the way, let’s meet our topic:

The FreeBSD Installer

If you’ve been living under a rock (or in a cave without Internet access) for the past 3+ years, here’s the highlights for what you’ve been missing in the FreeBSD Installer arena:

+ sysinstall(8) had a good run from 4.x to 8.x
+ 9.0-RELEASE: Introduced bsdinstall(8) [new default installer]
9.1-RELEASE: No noticeable changes
+ 9.2-RELEASE:
        + Improved bsdinstall(8) scripting
        + Introduced bsdconfig(8) [bsdinstall helper]
        + Introduced sysrc(8)

Before continuing, let’s take a brief moment to give credit where credit is due:

~23,000 Lines: Legacy sysinstall(8) by Jordan Hubbard
NOTE: Including the hard work of countless others (including myself)

~6,000 Lines: New installer bsdinstall(8) by Nathan Whitehorn
NOTE: Including many patches from others (including myself)

Meanwhile, I proudly take credit for the following:

~33,000 Lines: New installer companion bsdconfig(8)
~1,000 Lines: System management tool, sysrc(8)

An amazing amount of work has gone into both bsdinstall(8) and bsdconfig(8) in an attempt to gain momentum toward a future away from sysinstall(8).

However, exactly what is the future with respect to our Installer? It’s often tempting to sit down and look at where we came from and immediately [with ease] point out any number of short-comings in the replacements. While hindsight is paramount to preserving backward compatibility, we should be ever mindful to avoid simply reproducing a new generation of the same thing.

That being said, and with so much effort being put into a complete rewrite, I believe some new high-level goals and ideas can pave a path avoiding the dreaded bikeshed situation (see bikeshed.com; or perhaps teal.bikeshed.com is your cup of tea).

This is a nuclear reactor style look at the FreeBSD Installer from the top-down. Note however, that as we perform a top-down view, design was performed from the bottom-up. We’ll start our top-down view with pretty pictures and end with the very words and code that inspired this blog entry.

Starting at the top, let’s focus on the User Interface. The UI is still a fairly broad topic, so let’s further focus on a subset of the UI, realtime feedback. Our initial focal point will be the UI management for providing feedback during data input/output operations.

User Interface :: Data I/O Feedback

View below, a picture of the distfetch module of bsdinstall(8):

bsdinstall(8) distfetch module
The distfetch module of FreeBSD bsdinstall(8)

NOTE: To get proper box drawing characters on the system console under FreeBSD 9.0 or higher, you must change the default terminal type in ttys(5) from xterm to cons25. This is easy in FreeBSD 9.2 or higher — execute bsdconfig syscons_ttys, select IBM437 (VGA default), execute init q, then exit, and log back into the system console.

Meanwhile, here’s what legacy sysinstall(8) looks like:

"sysinstall(8) distExtractAll"
The distExtractAll functionality of FreeBSD sysinstall(8)

As you can see from the above two pictures, bsdinstall(8) (top) is an improvement over sysinstall(8) (bottom). Perhaps the most noticeable change is that bsdinstall(8) presents a single widget to the user, providing feedback on all requested distributions within a single screen; meanwhile sysinstall(8) presents only a single progress bar widget, multiple times in a series.

While this is indeed an improvement, a few things come to mind:

  • bsdinstall(8) doesn’t display data/rate information
  • The progress bar shouldn’t be the only indicator that data is moving
  • A cursor block appears oddly pinned within the bsdinstall(8) widget
  • bsdinstall(8) separates fetching vs extraction (sysinstall(8) does not)

However, perhaps the most important observation should be that both applications operate serially. Both bsdinstall(8) and sysinstall(8) do not proceed to the next distribution until finishing the previous one. Since the data in each distribution is unique, it is not a requirement to unpack them serially but rather, how it’s simply been done (for over 15 years).

When bsdinstall(8) improved the UI to provide a multi-progress widget, the obvious question of concurrent parallelism should be raised. From a UI standpoint, once you have a widget that can display progress for multiple items, the next logical step is to allow the “Pending” items to be processed without delay. In other words, the UI we have today for distfetch, is a great starting point but we can take it further.

For example, view below a prototype that I am developing:

"FreeBSD Installer Prototype"
A prototype replacement for FreeBSD bsdinstall(8) distfetch/distextract

NOTE: Both dialog(1) and dialog(3) render ANSI bold as gray on the system console in 9.0 or higher. Compare to the following image, executing the same program via SSH in which bold is exactly what you’d expect.

NOTE: The above was taken with a value of en_US.ISO8859-1 for the LC_ALL environment variable, nicely resulting in comma-separators in the KBytes/sec status line entry. See localeconv(3).

The prototype that I am developing can also replace other [less than informative] areas of the system. For example, the following picture shows how [deprecated] sysinstall(8) told the user that it was downloading the 24MB INDEX file (commonly via FTP):

"sysinstall(8) configPackages"
The configPackages functionality of FreeBSD sysinstall(8)

As you can see above, sysinstall(8) is pretty spartan when it comes to providing any feedback in this area. The replacement, bsdconfig(8), does not yet bring any notable changes (pictured below):

"bsdconfig(8) packages"
The packages module of FreeBSD bsdconfig(8)

Neither deprecated sysinstall(8) nor new bsdconfig(8) provide any information when [down]loading packages/INDEX. There’s no progress bar, no indicator that data is moving, and no data/rate info. My prototype application (named fdpv(1)) can improve the user experience here by providing those things (pictured below).

"fdpv(1) INDEX fetch"
Prototype application for handling I/O streams

However, perhaps the best use for such a tool is to improve the way packages are installed. Pictured below is deprecated sysinstall(8), installing a single package.

"sysinstall(8) packageAdd"
Legacy/deprecated sysinstall(8) installing vim-lite via FTP

Not quite as blunt as the fetching of packages/INDEX; this time at least sporting data/rate information in the status line. However, moving forward and away from sysinstall(8), let’s see what replacement bsdconfig(8) has for us:

bsdconfig(8) packageAdd
bsdconfig(8) packageAdd installing vim-lite

Notice a slight regression in moving forward. New bsdconfig(8) lacks data/rate information that deprecated sysinstall(8) provided.

I plan to fix the regression by introducing fdpv(1) as a new back-end for package installation. Here’s what it could look like:

"fdpv(1) based packageAdd"
A prototype packageAdd that uses fdpv(1)

NOTE: The above is a Photoshop‘ed image. Concurrency has not yet been programmed.

To better understand the fdpv(1) prototype, here is a picture with annotations showing the layout of information and where it is drawn from (no pun intended).

"fdpv(1) based packageAdd with annotations"
Inormation layout of fdpv(1) display

As you can see, the fdpv(1) tool is just a conduit. As such, the tool can be used for both legacy package management as well as pkgng (new default package management tool in FreeBSD).

For the sake of discussion regarding package management, notice the re-introduction of the status line (at bottom) to provide data/rate info (enumerating total throughput) as well as concurrent parallel processing of multiple dependencies during the installation of xorg (pictured).

This should dramatically reduce the amount of time required to install packages because independent dependencies, whom already have their own dependencies satisfied, can be installed in-parallel rather than waiting.

Just how many concurrent workers are allowed? Well, that would be up to Grand Central Dispatch (GCD). Programmed to use GCD, the utility should allow transparent scaling, respective to hardware capability and machine load.

Of course, I have to admit to not knowing GCD yet, so I’ve cast a fellow FreeBSD committer to help with that end. I was originally planning on using POSIX Threads aka pthreads and fancy mutex signaling. But Grand Central Dispatch looks much more promising.

What I’ve accomplished thus far (as of tagged-version 0.5) on this project is the following:

  • Full UI integration with: dialog(3) [default], dialog(1) [-D], and Xdialog(1) [-X]
  • Nearly complete manual (man(1) page) documenting syntax of command-line utility and all its flags.
  • Created test programs that exercise expected usage.
  • Prepared the code for clang (a requirement to use GCD).

What’s not done however, is the following:

  • For package management, a command-line utility is inadequate. The main() portion of the utility needs to be translated into a library (libfdpv) allowing the integration of libpkg.
  • Concurrent handling of file arguments via Grand Central Dispatch.
  • Actual handling of I/O (reads/writes on file paths).

Those things need to be finished before we can import this new tool to the FreeBSD base system. Once those things are finished and fdpv(1) is in the base system, we can then start rewriting bsdinstall(8) and bsdconfig(8) to use it, providing enhanced I/O data feedback where we need it most (the installation of the system and package maintenance, respectively).

Pure bonus is the ability to work in X11 (shown below).

"fdpv(1) in X11 mode using x11/xdialog from FreeBSD ports"
Various dialogs showing fdpv(1) in X11 mode

This matches bsdconfig(8)‘s already built-in ability to work entirely in X11, meanwhile bsdinstall(8) has to slowly be reworked to support X11 (by introducing new tools such as fdpv(1) to replace the existing C components that currently support only dialog(3)).

The code repository for fdpv(1) is at the below address:

http://druidbsd.cvs.sf.net/druidbsd/fdpv/

NOTE: There is a “Download GNU Tarball” link at the bottom as well as a pull-down menu for selecting a “Sticky Tag”, allowing easy download of tagged revisions.

We’ve had an in-depth look at Data I/O Feedback, in the next blog post we will be switching over to another User Interface issue, Internationalization.

Thank you for reading.

Leave a Reply