FreeBSD Nginx Performance

At $work, we have been looking at Nginx Plus because unlike the free open-source version of Nginx, Plus has load-balancing features. We’re pretty excited about that because it means we could potentially replace our reliance on HAProxy. Below is a simple graphic showing the potential infrastructure change.

With an nginx plus license, you can eliminate the need for haproxy

The next thing we had to do was determine which platform we would be using for our test of Nginx Plus. That ended up being limited to just two choices based on our constraints — must be physical hardware and must support HTTP/2 and ALPN.

While the system requirements page doesn’t say this directly, we were informed that the only configuration currently supporting HTTP/2 and ALPN on physical hardware is FreeBSD 11.0 and Ubuntu 16.04 LTS. Others such as CentOS were disqualified because, for example, the OpenSSL version was too old to support ALPN.

With a little bit of work in massaging our PXE server (resulting in a complete rewrite of pxe-config for FreeBSD deployment automation), we had the FreeBSD systems deployed in a matter of days. The generation of a Debian Installer preseed for Ubuntu proved to be far more time-consuming and challenging, but after a week we had the Ubuntu systems deployed as well.

We then registered for a free 30-day trial of Nginx Plus and went through the installation instructions for each Operating System. The steps for FreeBSD and Ubuntu are similar wherein you configure your system to add their package server and use the standard package utilities (pkg on FreeBSD and apt-get on Ubuntu) to install a binary package.

With FreeBSD 11.0-RELEASE-p1 and Ubuntu 16.04 LTS on identical servers in the same rack/switch, we were ready to do some benchmarking to help us determine whether Nginx Plus can deliver equitable performance in a load-balanced array compared with an haproxy cluster sitting in front of the free and open-source version of Nginx.

To generate the right type of load for our performance benchmark, we are using hey by Jaana B. Dogan. Below is the usage statement from hey -h:

Usage: hey [options...] <url>

Options:
  -n  Number of requests to run. Default is 200.
  -c  Number of requests to run concurrently. Total number of requests cannot
      be smaller than the concurrency level. Default is 50.
  -q  Rate limit, in seconds (QPS).
  -o  Output type. If none provided, a summary is printed.
      "csv" is the only supported alternative. Dumps the response
      metrics in comma-separated values format.

  -m  HTTP method, one of GET, POST, PUT, DELETE, HEAD, OPTIONS.
  -H  Custom HTTP header. You can specify as many as needed by repeating the flag.
      For example, -H "Accept: text/html" -H "Content-Type: application/xml" .
  -t  Timeout for each request in seconds. Default is 20, use 0 for infinite.
  -A  HTTP Accept header.
  -d  HTTP request body.
  -D  HTTP request body from file. For example, /home/user/file.txt or ./file.txt.
  -T  Content-type, defaults to "text/html".
  -a  Basic authentication, username:password.
  -x  HTTP Proxy address as host:port.
  -h2 Enable HTTP/2.

  -host	HTTP Host header.

  -disable-compression  Disable compression.
  -disable-keepalive    Disable keep-alive, prevents re-use of TCP
                        connections between different HTTP requests.
  -cpus                 Number of used cpu cores.
                        (default for current machine is 8 cores)
  -more                 Provides information on DNS lookup, dialup, request and
                        response timings.

NOTE: Although our requirements include HTTP/2 and hey has the -h2 flag to enable HTTP/2, our performance benchmarks will be using HTTP/1[.1] because our current edge infrastructure to which we can make comparisons does not yet support HTTP/2.

The command that we used to test the performance of our setup is as follows:

hey -n 3000 -c 300 -m GET -disable-keepalive <url>

This asks hey to perform a total of 3000 HTTP/1[.1] GET requests for <url> with up to 300 concurrent requests.

When <url> points to our vanilla Ubuntu 16.04 LTS test box running Nginx Plus, the results are as follows:

Nginx Plus and Ubuntu
The hey command, hammering a vanilla Ubuntu 16.04 LTS server running Nginx Plus

When <url> points instead to our vanilla FreeBSD 11.0-RELEASE-p1 test box, the results are as follows:

Nginx Plus and FreeBSD
The hey command, hammering a vanilla FreeBSD 11.0-RELEASE-p1 server running Nginx Plus

Focusing on the Summary Total, we can see that vanilla FreeBSD takes 3x longer than Ubuntu and something is causing high response times in FreeBSD.

SPOILER: After much digging, it was discovered that the nginx binary for Nginx Plus was linking against an OpenSSL that was compiled from ports without the ASM optimizers (highly optimized Assembly routines for speeding up calculations on supported CPU architectures).

The instructions for installing Nginx Plus on FreeBSD include “pkg install nginx-plus” and this brings in security/openssl from ports instead of using the OpenSSL that comes with the FreeBSD base Operating System. This is generally a good thing because ports are updated more frequently than base which helps keep Nginx Plus up-to-date with the latest OpenSSL.

The standard UNIX utility ldd shows us that /usr/local/sbin/nginx (as-installed by the nginx-plus package) links not against the system’s /usr/lib/libssl.so.8 but instead the non-base (read: ports) version located at /usr/local/lib/libssl.so.9

However, as we will see in the below photo, both the system OpenSSL in /usr/bin and the ports OpenSSL in /usr/local/bin are the same version compiled on the same calendar day (despite having different shared library suffixes).

Command Line
Nginx Plus links to the ports OpenSSL

Though the two versions of OpenSSL appear to be the same, they are actually quite different. When OpenSSL is compiled with ASM optimizations, it can take full advantage of AES-NI and PCLMULQDQ, two important CPU instructions that increase the efficiency of cryptographic calculations.

Command Line
Testing OpenSSL for AES-NI and PCLMULQDQ to ensure ASM optimizations in support of CPU based crypto

The OPENSSL_ia32cap environment variable is a bit-mask of OpenSSL capabilities which allows us to disable AES-NI and PCLMULQDQ. Combining the values of ~0x200000000000000 (disable AES-NI) and ~0x200000000 (disable PCLMULQDQ) to get ~0x200000200000000, we can disable both AES-NI and PCLMULQDQ for individual runs of “openssl speed“.

In the below two commands, if your CPU supports AES-NI and your OpenSSL has been compiled with ASM optimizations, the first command will be many times faster than the second (wherein optimizations are disabled if available).

% openssl speed -elapsed -evp aes-256-cbc
% env OPENSSL_ia32cap="~0x2000002000000000" openssl speed -elapsed -evp aes-256-cbc

NOTE: See https://wiki.freebsd.org/SSHPerf for additional details.

As one might expect, the OpenSSL in /usr/bin shows a huge performance increase when AES-NI is not disabled. It was quite the shock to find that the ports OpenSSL in /usr/local/bin showed no differences in performance between the two commands.

We, the FreeBSD committers, took a look at the security/openssl port and discovered that it did not enable ASM optimizations by default at the time the binary packages were last compiled for FreeBSD 11. So I worked with the maintainer of port to fix that for the next time the packages get recompiled.

Review: D9480: security/openssl: Enable ASM by default
Submitted: rP433671

The next step is to determine the impact that an un-optimized OpenSSL has on our hey tests. The FreeBSD dynamic linker supports configurable dynamic object mapping through libmap.conf(5), so it is a fairly simple matter of telling /usr/local/sbin/nginx to use a different OpenSSL.

Creating /usr/local/etc/libmap.d/nginx.conf with the following contents will cause nginx to use the OpenSSL libraries that came with the base Operating System:

#
# origin		target
#
[/usr/local/sbin/nginx]
libssl.so.9		libssl.so.8
libcrypto.so.9		libcrypto.so.8

After creating this file and restarting nginx with “service nginx restart“, the hey performance tests now show FreeBSD ahead of Ubuntu in a head-to-head test.

To better illustrate the effects that the unoptimized OpenSSL had on the hey benchmarks, I wrote a utility that generates JSON from the output.

NOTE: While there are many ways to benchmark, this test focused on “time to completion” for 3000 requests with up-to 300 concurrent. The JSON generated depicts the non-linear approach toward completion.

Wrapper script for hey named hey_genlog for generating a log capable of being converted into JSON:

#!/bin/sh
############################################################ IDENT(1)
#
# $Title: Script to generate statistics from hey against a host $
# $Copyright: 2017 Devin Teske. All rights reserved. $
# $Smule$
#
############################################################ INFORMATION
#
# Statistics are logged to stdout. Use hey2graph to generate JSON.
# JSON is designed for highcharts/highstock API.
#
############################################################ CONFIGURATION

#
# hey utility from https://github.com/rakyll/hey
#
HEY=hey

#
# File to request
#
FILE=/aud1.m4a

#
# Total number of requests to perform
#
TOTAL=3000

#
# Maximum number of concurrent requests
#
CONCURRENT=300

#
# QoS rate limiting
# NB: Set to NULL to disable rate limiting
#
#RATE_LIMIT=1 # seconds
RATE_LIMIT= # seconds

#
# Should we use Secure-HTTP (https)?
# NB: Set to NULL to disable https
#
SECURE=1

############################################################ GLOBALS

pgm="${0##*/}" # Program basename

#
# Global exit status
#
SUCCESS=0
FAILURE=1

#
# Command-line arguments
#
HOST=$1

############################################################ FUNCTIONS

usage()
{
	exec >&2
	printf "Usage: %s HOST\n" "$pgm"
	exit $FAILURE
}

############################################################ MAIN

case "$HOST" in
"") usage ;; # NOTREACHED
*:*) : fall through ;;
*)
	if [ "$SECURE" ]; then
		HOST="$HOST:443"
	else
		HOST="$HOST:80"
	fi
esac
echo "Performing $TOTAL total requests"
echo "Maximum $CONCURRENT concurrent requests"
set -x
$HEY \
	-n $TOTAL \
	-c $CONCURRENT \
	${RATE_LIMIT:+-q $RATE_LIMIT} \
	-m GET \
	-disable-keepalive \
	http${SECURE:+s}://$HOST$FILE |
	awk -v cmd="date +%s.%N" '
		BEGIN {
			cmd | getline start
			close(cmd)
		}
		/requests done/ {
			cmd | getline date
			close(cmd)
			date = sprintf("%0.4f", date - start)
			sub(/^/, date " ")
		}
		1
	' # END-QUOTE

################################################################################
# END
################################################################################

Sample output:

[root@tc1.sf.smle.co ~]# ./hey_genlog a291.sf.smle.co | tee hey.freebsdbase.log
Performing 3000 total requests
Maximum 300 concurrent requests
0.5026 51 requests done.
1.0026 235 requests done.
1.5028 429 requests done.
2.0029 610 requests done.
2.5028 832 requests done.
3.0033 1025 requests done.
3.5031 1224 requests done.
4.0033 1427 requests done.
4.5038 1617 requests done.
5.0037 1816 requests done.
5.5039 2015 requests done.
6.0039 2240 requests done.
6.5041 2431 requests done.
7.0041 2639 requests done.
7.5041 2874 requests done.
7.7668 All requests done.

Summary:
  Total:	7.7650 secs
  Slowest:	4.9109 secs
  Fastest:	0.0157 secs
  Average:	0.6898 secs
  Requests/sec:	386.3489
  Total data:	8580021000 bytes
  Size/request:	2860007 bytes

Status code distribution:
  [200]	3000 responses

Response time histogram:
  0.016 [1]	|
  0.505 [1253]	|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  0.995 [1174]	|∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.484 [360]	|∎∎∎∎∎∎∎∎∎∎∎
  1.974 [140]	|∎∎∎∎
  2.463 [52]	|∎∎
  2.953 [13]	|
  3.442 [5]	|
  3.932 [0]	|
  4.421 [1]	|
  4.911 [1]	|

Latency distribution:
  10% in 0.2235 secs
  25% in 0.3667 secs
  50% in 0.5729 secs
  75% in 0.8668 secs
  90% in 1.3381 secs
  95% in 1.6279 secs
  99% in 2.3221 secs

Script for converting output from hey_genlog into JSON, hey2graph:

#!/usr/bin/awk -f
BEGIN { fmt = "[%0.4f, %u],\n"; printf fmt, 0, 0 }
/total requests/ { total = $2 }
/Total:/ { time = $2 }
/requests done/ && $2 ~ /^[0-9]+$/ { printf fmt, $1, $2 - y; y = $2 }
END { printf fmt, time, total - y }

Sample output:

[root@tc1.sf.smle.co ~]# ./hey2graph hey.freebsdbase.log
[0.0000, 0],
[0.5026, 51],
[1.0026, 184],
[1.5028, 194],
[2.0029, 181],
[2.5028, 222],
[3.0033, 193],
[3.5031, 199],
[4.0033, 203],
[4.5038, 190],
[5.0037, 199],
[5.5039, 199],
[6.0039, 225],
[6.5041, 191],
[7.0041, 208],
[7.5041, 235],
[7.7650, 126],

The process of generating JSON graph data was performed for Ubuntu, FreeBSD with ports OpenSSL, and FreeBSD with base OpenSSL. As you can see in the below graph, FreeBSD with base OpenSSL is the fastest with Ubuntu very close behind, and FreeBSD with an unoptimized ports OpenSSL coming in at 3x slower.

Satisfied that we had eliminated the performance issue causing FreeBSD to be 3x slower, we now asked why is Ubuntu slower than FreeBSD?

Intensive throughput benchmarks showed that FreeBSD is capable of reaching 87.1% line-rate while Ubuntu was only capable of 86.5%. Both systems given an Intel 10GE network interface, FreeBSD appears to be utilizing the hardware more efficiently. At the switch, we can see that FreeBSD is 99.1% efficient on 10GE, resulting in a measured 10.3% TCP overhead at the time of testing.

Switch Graph
FreeBSD line rate test on Intel 10GE, switch-level throughput graph

The result of our testing is that FreeBSD running Nginx Plus is a suitable replacement for our HaProxy and Nginx topology. You and everyone reading this won’t have to worry about the documented issue with OpenSSL because I worked with Bernard Spil and Allan Jude to get it fixed in the FreeBSD ports tree. The security/openssl port has been updated to enable ASM optimizations by default and fairly soon the binary packages will be rebuilt — until then, you can use the above /usr/local/etc/libmap.d/nginx.conf file to temporarily use the base OpenSSL if you’re unable to update /usr/ports/security/openssl/Makefile and recompile it yourself.

Cheers!

Mac OS X El Capitan and Native ssh-agent Notifications

For work, I have been given a new Macbook Pro 13″ running Mac OS X version 10.11.6, also known as El Capitan.

I was missing my ssh-agent notifications on this new mac (see links below), so it’s time to replay my recipe wherein I patch Apple’s OpenSSH to integrate support for Notification Center.

Original article:
http://devinteske.com/wp/ssh-agent-notifications-osx/
Update to original:
http://devinteske.com/wp/replay-mac-os-x-and-native-ssh-agent-notifications/

However, things have changed slightly in El Capitan, so we’ll explore a new set of instructions to simplify the process.

How to patch Apple’s OpenSSH on El Capitan to support native Notification Center in 16 steps (El Capitan specific changes highlighted in red):

REMINDER: You can browse Apple’s open source repository at https://opensource.apple.com/

  1. Disable System Integrity Protection using Apple SIP Guide
  2. Open Terminal.app
  3. curl -LO https://opensource.apple.com/tarballs/OpenSSL098/OpenSSL098-59.60.1.tar.gz
  4. tar zxf !$:t
  5. sudo cp -R !$:r:r/src/include/openssl /usr/local/include/
  6. curl -LO https://opensource.apple.com/tarballs/OpenSSH/OpenSSH-195.40.1.tar.gz
  7. tar zxf !$:t
  8. cd !$:r:r/openssh
  9. curl -L https://github.com/devinteske/apple/commit/f13a501678ed79bb9511828423e754df344e3bf0.patch -o patch.txt
  10. patch -N < !$
  11. ./configure --with-pam --with-audit=bsm
  12. make
  13. sudo cp -avf /usr/bin/ssh-agent{,.orig}
    Note: This step requires SIP to be disabled
  14. sudo cp ssh-agent /usr/bin/ssh-agent
    Note: This step requires SIP to be disabled
  15. killall ssh-agent
  16. ssh some-host
    Note: If you're just using ~/.ssh/id_rsa and haven't added the key to your keychain, no notification will appear and you will have to execute "ssh-add -K ~/.ssh/id_rsa" (without quotes; make sure to use capital-K) to add the key to your keychain.
  17. Optional: Re-enable SIP using Apple SIP Guide

Now every subsequent ssh request that uses a key stored in your keychain will cause a notification to appear in Apple's built-in Notification Center. Cheers!

Write Images to USB with AppleScript

As the only release engineer at $work, I have been tasked with creating the bootable USB devices of our FreeBSD-based product to send to the Contract Manufacturer. Every time a new release made it to General Availability (GA) status, Sabine would drop off a couple USB sticks and tell me which build to load onto them.

Wanting to empower her, I set out to create a better approach; one that did not involve dropping to the command-line and tapping-out a “dd” command. Meet a new AppleScript, named “WriteImage” written to take the guess-work out of the process.

http://druidbsd.cvs.sf.net/viewvc/druidbsd/WriteImage/

With WriteImage you can now take the USB image file (*.img or *.img.gz) and drop it on the AppleScript droplet. It automates the process of generating the necessary commands to image the file onto hardware USB. Tell the app through a series of clicks or drag/drop actions which image to write, it then prompts you to insert a USB device, and then asks for confirmation (twice) before launching Terminal to do the dirty work.

Let’s see it in action (photo tour):

Selecting an Image file for WriteImage
WriteImage initial dialog when double-clicked.
Prompt to insert USB device
WriteImage prompt, waiting for you to insert a USB device to image.
Last chance before proceeding
WriteImage displays a Last Chance dialog before proceeding to do anything.
Writing the Image to USB
WriteImage runs a shell script that does the dirty work.
Imaging the USB Drive
WriteImage ultimately results in an automatically crafted “dd” command run via Terminal
Get progress with Ctrl-T
When “dd” is running, press Ctrl-T on the keyboard to see how much data has been written and how fast
Success
As-is the case with FreeBSD images, when the process is complete, your Mac will give this error. Success in disguise.
Initializing a Drive
If you need to image a drive that is not currently readable by Mac OS X, click “Initialize…” and use Erase

Cheers!

REPLAY: Mac OS X and Native SSH-Agent Notifications

Having recently updated my 2014 Macbook Air 11″ from Mac OS X 10.10 to 10.10.4, I lost my customizations to Apple’s OpenSSH and no longer was I receiving anything in the Notification Center for SSH logins using my keychain. Looks like it’s time to rehash the steps necessary to reload support for the Notification Center for those going through the same ordeal.

For reference here’s the original blog post where I introduced the customizations required to get Apple Notification Center support for OpenSSH on Mac OS X:

http://devinteske.com/wp/ssh-agent-notifications-osx/

Step-by-step we’ll go through the motions of re-obtaining the latest copy of Apple’s modified OpenSSH, to be patched.

NOTE: Make sure you’ve updated Xcode in Apple Software Update, else you’ll get the error "configure: error: Your OpenSSL headers do not match your library" during compilation steps below.

ASIDE: You can browse Apple’s opensource repository at https://opensource.apple.com/ (obtaining the latest copy of OpenSSH with Apple’s customizations couldn’t be easier). Rather than documenting how to navigate said page, below steps will instead use direct links to the software we’re recompiling (for brevity).

How to patch Apple’s OpenSSH to support native Mac OS X Notification Center (in 13 easy steps):

  1. Open Terminal.app
  2. curl -LO https://opensource.apple.com/tarballs/OpenSSH/OpenSSH-189.tar.gz
  3. tar zxf !$:t
  4. cd !$:r:r/openssh
  5. curl -L https://github.com/devinteske/apple/commit/296d954851dbba2384797620c1b9a77e562917b8.patch -o patch.txt
  6. patch -N < !$
  7. ./configure --with-pam --with-audit=bsm
  8. make
  9. sudo cp -avf /usr/bin/ssh-agent{,.orig}
  10. sudo cp ssh-agent /usr/bin/ssh-agent
  11. killall ssh-agent
  12. ssh some-host
  13. Click "Always Allow" to confirm access from new ssh-agent to Keychain

Now every subsequent ssh request will give you a notification in Apple's built-in Notification Center. Cheers!

Mac OS X and Native ssh-agent Notifications

Apple Mac OS X makes working with SSH private keys easy and convenient. Things that you can safely forget and drop from your routine when using Mac OS X are:

  • How and when to launch ssh-agent(1)
  • Continuously loading your private key

Apple has integrated launchd(8) and Keychain support into ssh-agent(1) so it launches automatically on-demand and can interact with keys stored in Keychain Access.

First, a brief look at the above integration before we go improving things using new features introduced in Mac OS X 10.8 and higher.

EDIT: If you’re reading this thinking “Mac OS X has had Keychain and launchd integration for a while,” skip on ahead to “That being said…” below).

Traditionally when using an agent you would first manually launch an instance of ssh-agent(1) and then follow that up with:

ssh-add ~/.ssh/id_dsa

while on Mac OS X there is:

ssh-add -K ~/.ssh/id_dsa

which both loads your key into the running ssh-agent(1) and imports it into your Keychain. So you’ll find that quite untraditionally if you kill the running agent and then try to ssh(1) to a remote host, a new agent is launched on-demand and it loads the private key through the Keychain (login succeeds despite conventional logic about agents).

In the following picture, we can see previously loaded key(s) if we filter on the word “ssh” in Apple’s Keychain Access application (provided in the Utilities folder accessible via Command-Shift-U keyboard shortcut while in the Finder application):

View SSH private keys loaded with `ssh-add -K' in Keychain Access
View SSH private keys loaded with “ssh-add -K” in Keychain Access

If you delete this key from Keychain Access application then ssh(1) logins requiring that key no longer succeed as the running agent immediately loses access to said key. So the Keychain can be thought of as a persistent store for the agent; one that is tied to your Mac’s login account.

The automatic re-launching of ssh-agent(1) through launchd(8) means that if you kill the running agent it will be re-launched not immediately but only when access is attempted to a launchd(8)-provided default $SSH_AUTH_SOCK value as-configured in /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist.

Initially when Mac OS X boots, ssh-agent(1) is not started. It is only when you perform your first ssh(1) (or related) command that launchd(8) invokes the following command:

/usr/bin/ssh-agent -l

You can see launchd(8)‘s integration in-action by opening a Terminal immediately after booting your Mac and first executing:

ps axwww | grep ssh-agent

You’ll see there are no instances of ssh-agent(1), yet. Meanwhile, if you execute:

echo $SSH_AUTH_SOCK

You’ll see launchd(8) provided a default value (similar to the following):

/tmp/launch-oLGVUi/Listeners

When any ssh(1) command attempts to access this launchd(8) provided $SSH_AUTH_SOCK listener, launchd(8) invokes the previously mentioned “ssh-agent -l” instance which then accesses private keys stored in your keychain. For example, assuming Remote Login is enabled in the Sharing section of System Preferences (which enables sshd), execute:

ssh localhost

After which, re-execute:

ps axwww | grep ssh-agent

To find a running instance of “/usr/bin/ssh-agent -l” (the “-l” indicating it was launched on-demand by launchd(8)).

This is all very slick integration that quintessentially eases your SSH life. In a nut-shell, it’s private keys provided from your Keychain on-the-fly whenever you use an ssh utility.

 


 

That being said, we can certainly improve upon this integration by adding new code to Apple’s version of OpenSSH. Primarily, since ssh-agent(1) can now be loaded on-demand and access data in my Keychain, I would like to get a notification every time someone uses the running agent (for security reasons and also coolness factor).

In Mac OS X Mountain Lion (10.8) and Mavericks (10.9) we are blessed with the new Notification Center API (http://support.apple.com/kb/ht5362). This built-in functionality replaces the need for 3rd-party software such as Growl (http://growl.info) required to perform notifications in previous releases of Mac OS X (10.7 and older).

RESOURCE: Apple’s custom version of OpenSSH (available from http://opensource.apple.com — specifically for Mavericks http://opensource.apple.com/tarballs/OpenSSH/OpenSSH-186.tar.gz).

I first tested integration of the NSNotificationCenter API on a non-Apple version of OpenSSH after failing to get my friend’s Growl-based notification patches to work (I didn’t want to pay $3.99 for Growl in the Mac App Store). Prior to Mac OS X 10.8, my friend @kang had produced the following work for getting notifications out of ssh-agent(1):

https://www.insecure.ws/2013/09/25/ssh-agent-notification/

EDIT: My friend just made a patched version of Simon G. Tatham’s Pageant that provides native Windows notifications — http://www.twitpic.com/e2v52x/full — available in both Source (https://github.com/gdestuynder/putty-pagent-notification) and binary release (https://github.com/gdestuynder/putty-pagent-notification/releases).

However, my good friend’s work was on the non-Apple version of OpenSSH and thus my resulting agent binary lacked launchd(8) integration and Keychain support. This meant that if I were to replace the existing /usr/bin/ssh-agent binary in Mavericks with the patched non-Apple version, I would lose those features previously discussed above, let alone that his patches were for an external notification system versus the new built-in Notification Center.

So I set out to build a drop-in replacement for Mavericks’ /usr/bin/ssh-agent carrying all the integration of the original.

EDIT: Keep reading toward the end for GitHub source link and link to Mavericks binary.

After unpacking Apple’s OpenSSH-186.tar.gz (link above) and applying my patches to add support for NSNotificationCenter, I quickly learned that some additional non-standard configure arguments were required.

Within the top-level openssh directory (of the unpacked OpenSSH-186.tar.gz archive) I executed the following to produce a binary truly worthy of replacing the stock /usr/bin/ssh-agent:

./configure --with-pam --with-audit=bsm
make

After which I had a patched ssh-agent binary in the current working directory, suitable for doing the following:

sudo cp -avn /usr/bin/ssh-agent{,.orig}
sudo cp ssh-agent /usr/bin/ssh-agent
killall ssh-agent

The very next ssh(1) command I execute causes a new fresh instance of my patched ssh-agent(1) to be invoked. The first time my patched agent attempts to access my Keychain, I get the following audit request (image below):

Granting patched ssh-agent(1) access to Keychain for the first time.
Granting patched ssh-agent(1) access to Keychain for the first time.

I choose the “Always Allow” option, knowing that if I want to change it later I can locate the private key in Keychain Access (HINT: Filter on “ssh”) and press the Command-I keyboard shortcut to edit Access Controls for that key.

My patched agent is programmed to hook into the new Notification Center and send you a notification when signing requests (image below).

Notification Center message from ssh-agent(1)
Notification Center message from ssh-agent(1)

Now every time I authenticate by way of private key, regardless of whether that key was loaded once with “ssh-add” or from the Keychain imported via “ssh-add -K“, I get a message from the Notification Center.

EDIT: The primary concern that inspired this work is when you enable agent forwarding to remote hosts. Logging into a compromised host with agent forwarding, a hacker can use your agent to log into yet uncompromised systems to which you have access via your agent. While notifications won’t prevent the use of your agent to reach new machines should you enable forwarding through a compromised system, the notifications will let you know when to (a) shut down your agent (b) take inventory of active connections where agent forwarding was enabled and (c) cull the logs for machines you were forwarding through in an effort to find suspicious activity indicating either a malicious hacker or black-hat administrator on the system. For a corporate laptop that primarily only ever connects to corporate networks (where such activity is never to be expected), a notification is more than adequate to quell the problem (versus a public network where it may be more advantageous to use an ask-pass utility to confirm key usage rather than notify after open usage).

How those messages appear are configured in the System Preferences for Notifications (easily accessible via the gear icon at bottom-right of the Notification Center Drawer). Only after you’ve received your initial message from the patched ssh-agent(1) can you then configure the specifics of how its messages appear. The patched agent pretends to be Keychain Access when talking to the Notification Center, so configuring the style and attributes for Keychain Access notifications will allow you to customize how these agent notifications appear.

As notifications are delivered, they stack up in the pull-out drawer of the Notification Center. I like to clear the log of ssh-agent(1) notifications before I go to bed and check for any when I wake up (perhaps indicating that my agent signed a login request when I wasn’t looking).

By default, I’ve made the agent play the “Submarine” sound during notifications. This can be changed by making your own build from the source code I’ve branched into Git offered below:

https://github.com/devinteske/apple/tree/master/OpenSSH-186/openssh

But you don’t have to modify the code if all you want to do is disable the notification sound. Within the Notifications section of System Preferences you can state that you don’t want the Keychain Access notifications to play a sound.

Last, but not least, I have made a precompiled binary with my modifications so you can simply download and copy into place (using the below command-line instructions):

curl -OL http://druidbsd.sf.net/download/ssh-agent+notifications.osx-10.9.2.tbz
tar jxf !$:t
sudo cp -avn /usr/bin/ssh-agent{,.orig}
sudo cp -fv ssh-agent /usr/bin/
killall ssh-agent

Executing these 5 commands and then logging into a remote machine with ssh(1) using private keys should generate a notification from Keychain Access (pictured above). Cheers!

If you need to rollback to the original ssh-agent for any reason, the above steps made a backup that can be restored typing the following commands:

sudo mv -f /usr/bin/ssh-agent{.orig,}
killall ssh-agent

Wherein your next ssh(1) command will cause the original ssh-agent(1) to be loaded (which doesn’t support NSNotificationCenter API).

Debugging FreeNAS 9.2.0 System

I recently found my FreeNAS box rebooting itself in the middle of the night (every night) and I wanted to know what was going on. I first tried enabling the debug kernel (see below picture) but that didn’t work nor did it get me any closer to understanding the issue at-hand.

"Enabling the FreeNAS Debug Kernel from the WebUI"
Enabling the FreeNAS Debug Kernel from the WebUI

The idea behind enabling the debug kernel was that it would drop me into the kernel debugger rather than rebooting 15 seconds after an unattended fatal error (leaving zero logs to analyze). But much to my surprise, the system rebooted anyway.

The next step was to set up remote syslog to allow FreeNAS to log to an external box. The first thing I did was connect my FreeNAS server directly to a FreeBSD system. The FreeNAS system will be configured to log everything to the FreeBSD system; so when FreeNAS reboots I will have a log of what lead up to that event.

I configured the secondary network interface on each server to talk on the same subnet. On the FreeNAS side, I went into the WebUI and configured the second interface of em1 to have IPv4 address 192.168.1.2 with a /24 subnet (255.255.255.0). When that was done, it looked like the below picture.

"Configuring a local second network interface in the FreeNAS WebUI"
Configuring a local second network interface in the FreeNAS WebUI

After that, I logged into my FreeBSD server and prepared it to receive remote syslog data.

ifconfig em1 inet 192.168.1.1/24
echo 192.168.1.2 freenas >> /etc/hosts
sysrc syslogd_enable=YES
sysrc syslogd_flags="-a freenas -v -v"
echo "+freenas" >> /etc/syslog.conf
echo "*.* /var/log/freenas.log" >> /etc/syslog.conf
touch /var/log/freenas.log
service syslogd restart

Now SSH into your FreeNAS box and execute the following:

mount -uw /
echo 192.168.1.1 freebsd >> /conf/base/etc/hosts
echo 192.168.1.1 freebsd >> /etc/hosts
mount -ur /

Then perform the last final bit, which is to go into the FreeNAS WebUI and enter a value of “freebsd” for the syslog server (see below picture).

"Where to configure a remote syslog server fro within the FreeNAS WebUI"
Where to configure a remote syslog server fro within the FreeNAS WebUI

At this point, you should be able to execute via SSH on the FreeNAS system:

logger foo

And get the following entry in /var/log/freenas.log on your FreeBSD system:

Jan 26 04:42:37 freenas root: foo

Cheers!

Of course, the jury is still out on exactly why my system is crashing, but I suspect a failing hard disk (the logs being streamed to the FreeBSD system seam to indicate via smartd messages a prefail condition for the boot disk).

Recovering FreeNAS 9.2.0

If you should find that you have a FreeNAS system for which you have forgotten the root password, here’s an easy way to recover your system.

First, you’ll need to get shell access. If you’ve left the console menu enabled, then resetting the root password is as easy as 1, 2, 3. However if you have disabled the console menu for security reasons, keep reading for the prescribed recipe.

If you are lucky enough to have a user on the system that has sudo privileges, the first step is to log in as that user. If you do not have such a user, the first step is instead to boot into single-user mode.

TIP: Since FreeNAS has a 1-second timeout on the boot menu, you’ll have to repeatedly tap the space bar when you see it starting to load the kernel; with perseverance you will be able to abort the 1-second timeout to interact with the boot menu wherein single-user mode can be enabled.

Once you’re on the shell (either as root in single-user mode or as a secondary user with sudo access), the first task at-hand is to make the root filesystem read/write by executing:

mount -a         # Only required if running single-user
mount -uw /

Once this is completed, we can then perform the first of a two-step process to set a new password for the root account (restoring access to both SSH and the WebUI). Execute:

passwd root

This is only the first step on a FreeNAS system. The work done by the above command will be undone by the /etc/ix.rc.d/ix-passwd boot script every time the machine boots. We need to take the temporary work performed by the above command and make it permanent by copying the information into the FreeNAS SQLite accounting database.

This can be performed by executing:

sqlite3 /data/freenas-v1.db \
    "UPDATE account_bsdusers SET bsdusr_unixhash=\"$( \
        awk -F: '$1=="root"{print $2;exit}' /etc/master.passwd \
    )\" WHERE bsdusr_username=\"root\""

If you are running as a user with sudo privilege (instead of running as root in single-user mode), here is a sudo endowed version of the above command:

sudo sqlite3 /data/freenas-v1.db \
    "UPDATE account_bsdusers SET bsdusr_unixhash=\"$( \
        sudo awk -F: '$1=="root"{print $2;exit}' /etc/master.passwd \
    )\" WHERE bsdusr_username=\"root\""

To check your work, here’s a command to extract the hash from the SQLite database:

sqlite3 /data/freenas-v1.db "SELECT bsdusr_unixhash \
    FROM account_bsdusers WHERE bsdusr_username=\"root\""

At this point, if you are running in single-user mode, type…

exit

…to continue booting. If instead you used a sudo capable user to make the change, there is no need to reboot as the system accounting table in the FreeNAS SQLite database is in-sync with the FreeBSD system accounting files (/etc/master.passwd; /etc/spwd.db; etc.).

You should now be able to log into the FreeNAS WebUI with the updated password. Cheers!

FreeNAS 9.2.0 Development

I’ve been diving into FreeNAS development lately. In this blog post, I’ll talk a little bit about what I did to make development life on the FreeNAS appliance more livable from a FreeBSD developer’s perspective.

The first thing you notice after booting up FreeNAS is that it has a console menu. This was my first stop to getting the box online. A couple of keystrokes and we had an IP address. Load up the IP in your browser and you’re ready to start configuring FreeNAS. We set a root password (via the initial WebUI), and you’re off to the races.

Now comes time to customize FreeNAS for development purposes. Most people won’t want to (and shouldn’t) make these adjustments as they are strictly for the purpose of developing enhancements to FreeNAS (right on the box itself). These changes are not supported by the FreeNAS support team (I cannot emphasize this enough).

As a pre-requisite to getting any customizations in-place, we’ll need to be able to log into the UNIX command-line. In the WebUI, select “Services”, scroll down to “SSH” and hit the toggle to turn it “ON”. Then click the wrench logo just off to the right of the SSH toggle. To log in with the root password using SSH, we need to first enable the security feature which comes disabled by default. Check the configuration option labeled “Login as Root with password” (otherwise you can only login as root with a private key).

In my case, I want to develop enhancements before I configure my storage. So it is rather inconvenient that the FreeNAS WebUI only allows you to create users with home directories within configured storage. For your average consumer, this restriction is more than acceptable considering the boot media is often very small and has little free space. However, I need a place to store the patches that I will generate while working on the Volume Manager code and despite what it lacks in free space, the thumb drive or disk that FreeNAS boots from is ideal for this.

The trick to adding a user with non-standard home directory appears to be: first add your user from the WebUI, initially selecting “/nonexistent” as the home-directory (only valid choice when you have yet to configure a storage volume); and then afterward use the following command from the command-line of your FreeNAS appliance to change the home-directory:

sqlite3 /data/freenas-v1.db "UPDATE account_bsdusers \
    SET bsdusr_home=\"/data/home/USER\" WHERE bsdusr_username=\"USER\""

Note that every time you edit this user from the WebUI, you will need to change the home-directory to “/nonexistent” to force the WebUI to accept/save the changes. Afterward, repeat the above command to restore the desired home directory.

The next issue that I run into is perhaps less common. This issue is specific to my ISP. I require the ability to add the following line to the file /etc/dhclient.conf:

supersede domain-name "mydomain";

The reason for this is because I use my ISPs DHCP and the offer contains a domain of XYZ(dot)com, while I prefer ABC(dot)com. By adding the above line to /etc/dhclient.conf, the underlying FreeBSD system will “do the right thing” and supersede the domain provided by my ISP with my desired one. In the end, /etc/resolv.conf gets auto-populated with what I need while still using WAN-based DHCP (which itself is uncommon and seldom available).

The gotcha with simply modifying /etc/dhclient.conf is that you’ll lose your changes upon reboot. You really need to put the directive into:

/conf/base/etc/dhclient.conf

But before you can edit this file, you need to re-mount the root filesystem as read-write by executing:

mount -uw /

Then you can change the files in /conf and either reboot or execute the following to make the root filesystem read-only again:

mount -ur /

NOTE: This command may take a few moments as it needs to clear pending writes.

Next, I like to install a few packages like bash, vim-lite, pstree, iftop, and sudo (NB: root filesystem needs to be read-write before executing):

pkg_add -r bash
pkg_add -r vim-lite
pkg_add -r pstree
pkg_add -r iftop
pkg_add -r sudo

I’m sure that I install more than just the above, but since the FreeNAS appliance makes /var a memory filesystem, there’s no backup record (usually /var/db/pkg) of installed packages. Once you reboot, you’ll lose the record of what’s installed-versus-not.

ASIDE: It’s really convenient that FreeNAS allows you to specify that a group be allowed to execute sudo. Rather than checking the box for each individual user that you would like to allow sudo access, using a group means you can then store files as group-writable on the disk — simplifying team-based system administration.

After performing all these customizations, I think it’s time to make a backup. To make a backup of my FreeNAS thumb drive, I’ve had success in using version 0.9 of Win32DiskImager from Launchpad.net. Just stick the thumb drive into a Window system, launch Win32DiskImager, and use the “Read” option to make an image of your thumb drive. This image can then be stored for backup or imaged to another device.

I’m actually not a huge fan of running my OS from a thumb drive. So the first thing I do with my backup image is to write it to an internal Solid State Disk. I copied the backup image of my thumb drive to a FreeBSD server where the SSD was installed as an extra disk, and executed the following command to write the image to the drive:

dd if=backup.img of=/dev/ada3 bs=64k

NOTE: /dev/ada3 is the SSD installed as an extra disk.

After that has completed, I remove the SSD from my FreeBSD server and insert it into the 1U that used to boot FreeNAS from thumb drive.

All is dandy; the SSD boots up just like the thumb drive (except many times faster, shaving over 30 seconds from the boot process).

Next, I wish that the /data directory (mounted from /dev/ufs/FreeNASs4) was larger than the scant 20MB initially created by the installer. The SSD that I just cloned FreeNAS to is much larger than 2GB, leaving just over 80GB of unused free space. Being-one to never waste (and especially since I now have a user with a home directory of /data/home/user ready to utilize said space), I really want the /data partition to reach all the way to the end of the drive.

First, stick the drive into another FreeBSD or FreeNAS system (I used FreeBSD 9.2-STABLE @r260095M) where it is visible as an “extra” (unused) disk. Then execute the following steps to reclaim the unused free space (assigning it to the /data partition):

mount /dev/ufs/FreeNASs4 /mnt
mkdir /foo
rsync -avH /mnt/ /foo/
umount /mnt
gpart resize -i 4 ada3
newfs -b 32768 -f 4096 -L FreeNASs4 /dev/ada3s4
mount /dev/ufs/FreeNASs4 /mnt
rsync -avH --delete /foo/ /mnt/
umount /mnt
rm -Rf /foo

Now I remove the SSD, replace it into my FreeNAS chassis and boot it up. Voila, 82G of free space for /data.

FreeNAS is now ready for a collective team of multiple system administrators, able to treat the system less like an appliance and more like a sandbox for development. Cheers!

FreeBSD Installer Enhancements

I work on FreeBSD as the current maintainer/author of the following [released] works:

Now that introductions are out of the way, let’s meet our topic:

The FreeBSD Installer

If you’ve been living under a rock (or in a cave without Internet access) for the past 3+ years, here’s the highlights for what you’ve been missing in the FreeBSD Installer arena:

+ sysinstall(8) had a good run from 4.x to 8.x
+ 9.0-RELEASE: Introduced bsdinstall(8) [new default installer]
9.1-RELEASE: No noticeable changes
+ 9.2-RELEASE:
        + Improved bsdinstall(8) scripting
        + Introduced bsdconfig(8) [bsdinstall helper]
        + Introduced sysrc(8)

Before continuing, let’s take a brief moment to give credit where credit is due:

~23,000 Lines: Legacy sysinstall(8) by Jordan Hubbard
NOTE: Including the hard work of countless others (including myself)

~6,000 Lines: New installer bsdinstall(8) by Nathan Whitehorn
NOTE: Including many patches from others (including myself)

Meanwhile, I proudly take credit for the following:

~33,000 Lines: New installer companion bsdconfig(8)
~1,000 Lines: System management tool, sysrc(8)

An amazing amount of work has gone into both bsdinstall(8) and bsdconfig(8) in an attempt to gain momentum toward a future away from sysinstall(8).

However, exactly what is the future with respect to our Installer? It’s often tempting to sit down and look at where we came from and immediately [with ease] point out any number of short-comings in the replacements. While hindsight is paramount to preserving backward compatibility, we should be ever mindful to avoid simply reproducing a new generation of the same thing.

That being said, and with so much effort being put into a complete rewrite, I believe some new high-level goals and ideas can pave a path avoiding the dreaded bikeshed situation (see bikeshed.com; or perhaps teal.bikeshed.com is your cup of tea).

This is a nuclear reactor style look at the FreeBSD Installer from the top-down. Note however, that as we perform a top-down view, design was performed from the bottom-up. We’ll start our top-down view with pretty pictures and end with the very words and code that inspired this blog entry.

Starting at the top, let’s focus on the User Interface. The UI is still a fairly broad topic, so let’s further focus on a subset of the UI, realtime feedback. Our initial focal point will be the UI management for providing feedback during data input/output operations.

User Interface :: Data I/O Feedback

View below, a picture of the distfetch module of bsdinstall(8):

bsdinstall(8) distfetch module
The distfetch module of FreeBSD bsdinstall(8)

NOTE: To get proper box drawing characters on the system console under FreeBSD 9.0 or higher, you must change the default terminal type in ttys(5) from xterm to cons25. This is easy in FreeBSD 9.2 or higher — execute bsdconfig syscons_ttys, select IBM437 (VGA default), execute init q, then exit, and log back into the system console.

Meanwhile, here’s what legacy sysinstall(8) looks like:

"sysinstall(8) distExtractAll"
The distExtractAll functionality of FreeBSD sysinstall(8)

As you can see from the above two pictures, bsdinstall(8) (top) is an improvement over sysinstall(8) (bottom). Perhaps the most noticeable change is that bsdinstall(8) presents a single widget to the user, providing feedback on all requested distributions within a single screen; meanwhile sysinstall(8) presents only a single progress bar widget, multiple times in a series.

While this is indeed an improvement, a few things come to mind:

  • bsdinstall(8) doesn’t display data/rate information
  • The progress bar shouldn’t be the only indicator that data is moving
  • A cursor block appears oddly pinned within the bsdinstall(8) widget
  • bsdinstall(8) separates fetching vs extraction (sysinstall(8) does not)

However, perhaps the most important observation should be that both applications operate serially. Both bsdinstall(8) and sysinstall(8) do not proceed to the next distribution until finishing the previous one. Since the data in each distribution is unique, it is not a requirement to unpack them serially but rather, how it’s simply been done (for over 15 years).

When bsdinstall(8) improved the UI to provide a multi-progress widget, the obvious question of concurrent parallelism should be raised. From a UI standpoint, once you have a widget that can display progress for multiple items, the next logical step is to allow the “Pending” items to be processed without delay. In other words, the UI we have today for distfetch, is a great starting point but we can take it further.

For example, view below a prototype that I am developing:

"FreeBSD Installer Prototype"
A prototype replacement for FreeBSD bsdinstall(8) distfetch/distextract

NOTE: Both dialog(1) and dialog(3) render ANSI bold as gray on the system console in 9.0 or higher. Compare to the following image, executing the same program via SSH in which bold is exactly what you’d expect.

NOTE: The above was taken with a value of en_US.ISO8859-1 for the LC_ALL environment variable, nicely resulting in comma-separators in the KBytes/sec status line entry. See localeconv(3).

The prototype that I am developing can also replace other [less than informative] areas of the system. For example, the following picture shows how [deprecated] sysinstall(8) told the user that it was downloading the 24MB INDEX file (commonly via FTP):

"sysinstall(8) configPackages"
The configPackages functionality of FreeBSD sysinstall(8)

As you can see above, sysinstall(8) is pretty spartan when it comes to providing any feedback in this area. The replacement, bsdconfig(8), does not yet bring any notable changes (pictured below):

"bsdconfig(8) packages"
The packages module of FreeBSD bsdconfig(8)

Neither deprecated sysinstall(8) nor new bsdconfig(8) provide any information when [down]loading packages/INDEX. There’s no progress bar, no indicator that data is moving, and no data/rate info. My prototype application (named fdpv(1)) can improve the user experience here by providing those things (pictured below).

"fdpv(1) INDEX fetch"
Prototype application for handling I/O streams

However, perhaps the best use for such a tool is to improve the way packages are installed. Pictured below is deprecated sysinstall(8), installing a single package.

"sysinstall(8) packageAdd"
Legacy/deprecated sysinstall(8) installing vim-lite via FTP

Not quite as blunt as the fetching of packages/INDEX; this time at least sporting data/rate information in the status line. However, moving forward and away from sysinstall(8), let’s see what replacement bsdconfig(8) has for us:

bsdconfig(8) packageAdd
bsdconfig(8) packageAdd installing vim-lite

Notice a slight regression in moving forward. New bsdconfig(8) lacks data/rate information that deprecated sysinstall(8) provided.

I plan to fix the regression by introducing fdpv(1) as a new back-end for package installation. Here’s what it could look like:

"fdpv(1) based packageAdd"
A prototype packageAdd that uses fdpv(1)

NOTE: The above is a Photoshop‘ed image. Concurrency has not yet been programmed.

To better understand the fdpv(1) prototype, here is a picture with annotations showing the layout of information and where it is drawn from (no pun intended).

"fdpv(1) based packageAdd with annotations"
Inormation layout of fdpv(1) display

As you can see, the fdpv(1) tool is just a conduit. As such, the tool can be used for both legacy package management as well as pkgng (new default package management tool in FreeBSD).

For the sake of discussion regarding package management, notice the re-introduction of the status line (at bottom) to provide data/rate info (enumerating total throughput) as well as concurrent parallel processing of multiple dependencies during the installation of xorg (pictured).

This should dramatically reduce the amount of time required to install packages because independent dependencies, whom already have their own dependencies satisfied, can be installed in-parallel rather than waiting.

Just how many concurrent workers are allowed? Well, that would be up to Grand Central Dispatch (GCD). Programmed to use GCD, the utility should allow transparent scaling, respective to hardware capability and machine load.

Of course, I have to admit to not knowing GCD yet, so I’ve cast a fellow FreeBSD committer to help with that end. I was originally planning on using POSIX Threads aka pthreads and fancy mutex signaling. But Grand Central Dispatch looks much more promising.

What I’ve accomplished thus far (as of tagged-version 0.5) on this project is the following:

  • Full UI integration with: dialog(3) [default], dialog(1) [-D], and Xdialog(1) [-X]
  • Nearly complete manual (man(1) page) documenting syntax of command-line utility and all its flags.
  • Created test programs that exercise expected usage.
  • Prepared the code for clang (a requirement to use GCD).

What’s not done however, is the following:

  • For package management, a command-line utility is inadequate. The main() portion of the utility needs to be translated into a library (libfdpv) allowing the integration of libpkg.
  • Concurrent handling of file arguments via Grand Central Dispatch.
  • Actual handling of I/O (reads/writes on file paths).

Those things need to be finished before we can import this new tool to the FreeBSD base system. Once those things are finished and fdpv(1) is in the base system, we can then start rewriting bsdinstall(8) and bsdconfig(8) to use it, providing enhanced I/O data feedback where we need it most (the installation of the system and package maintenance, respectively).

Pure bonus is the ability to work in X11 (shown below).

"fdpv(1) in X11 mode using x11/xdialog from FreeBSD ports"
Various dialogs showing fdpv(1) in X11 mode

This matches bsdconfig(8)‘s already built-in ability to work entirely in X11, meanwhile bsdinstall(8) has to slowly be reworked to support X11 (by introducing new tools such as fdpv(1) to replace the existing C components that currently support only dialog(3)).

The code repository for fdpv(1) is at the below address:

http://druidbsd.cvs.sf.net/druidbsd/fdpv/

NOTE: There is a “Download GNU Tarball” link at the bottom as well as a pull-down menu for selecting a “Sticky Tag”, allowing easy download of tagged revisions.

We’ve had an in-depth look at Data I/O Feedback, in the next blog post we will be switching over to another User Interface issue, Internationalization.

Thank you for reading.