Mac OS X and Native ssh-agent Notifications

Apple Mac OS X makes working with SSH private keys easy and convenient. Things that you can safely forget and drop from your routine when using Mac OS X are:

  • How and when to launch ssh-agent(1)
  • Continuously loading your private key

Apple has integrated launchd(8) and Keychain support into ssh-agent(1) so it launches automatically on-demand and can interact with keys stored in Keychain Access.

First, a brief look at the above integration before we go improving things using new features introduced in Mac OS X 10.8 and higher.

EDIT: If you’re reading this thinking “Mac OS X has had Keychain and launchd integration for a while,” skip on ahead to “That being said…” below).

Traditionally when using an agent you would first manually launch an instance of ssh-agent(1) and then follow that up with:

ssh-add ~/.ssh/id_dsa

while on Mac OS X there is:

ssh-add -K ~/.ssh/id_dsa

which both loads your key into the running ssh-agent(1) and imports it into your Keychain. So you’ll find that quite untraditionally if you kill the running agent and then try to ssh(1) to a remote host, a new agent is launched on-demand and it loads the private key through the Keychain (login succeeds despite conventional logic about agents).

In the following picture, we can see previously loaded key(s) if we filter on the word “ssh” in Apple’s Keychain Access application (provided in the Utilities folder accessible via Command-Shift-U keyboard shortcut while in the Finder application):

View SSH private keys loaded with `ssh-add -K' in Keychain Access

View SSH private keys loaded with “ssh-add -K” in Keychain Access

If you delete this key from Keychain Access application then ssh(1) logins requiring that key no longer succeed as the running agent immediately loses access to said key. So the Keychain can be thought of as a persistent store for the agent; one that is tied to your Mac’s login account.

The automatic re-launching of ssh-agent(1) through launchd(8) means that if you kill the running agent it will be re-launched not immediately but only when access is attempted to a launchd(8)-provided default $SSH_AUTH_SOCK value as-configured in /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist.

Initially when Mac OS X boots, ssh-agent(1) is not started. It is only when you perform your first ssh(1) (or related) command that launchd(8) invokes the following command:

/usr/bin/ssh-agent -l

You can see launchd(8)‘s integration in-action by opening a Terminal immediately after booting your Mac and first executing:

ps axwww | grep ssh-agent

You’ll see there are no instances of ssh-agent(1), yet. Meanwhile, if you execute:

echo $SSH_AUTH_SOCK

You’ll see launchd(8) provided a default value (similar to the following):

/tmp/launch-oLGVUi/Listeners

When any ssh(1) command attempts to access this launchd(8) provided $SSH_AUTH_SOCK listener, launchd(8) invokes the previously mentioned “ssh-agent -l” instance which then accesses private keys stored in your keychain. For example, assuming Remote Login is enabled in the Sharing section of System Preferences (which enables sshd), execute:

ssh localhost

After which, re-execute:

ps axwww | grep ssh-agent

To find a running instance of “/usr/bin/ssh-agent -l” (the “-l” indicating it was launched on-demand by launchd(8)).

This is all very slick integration that quintessentially eases your SSH life. In a nut-shell, it’s private keys provided from your Keychain on-the-fly whenever you use an ssh utility.

 


 

That being said, we can certainly improve upon this integration by adding new code to Apple’s version of OpenSSH. Primarily, since ssh-agent(1) can now be loaded on-demand and access data in my Keychain, I would like to get a notification every time someone uses the running agent (for security reasons and also coolness factor).

In Mac OS X Mountain Lion (10.8) and Mavericks (10.9) we are blessed with the new Notification Center API (http://support.apple.com/kb/ht5362). This built-in functionality replaces the need for 3rd-party software such as Growl (http://growl.info) required to perform notifications in previous releases of Mac OS X (10.7 and older).

RESOURCE: Apple’s custom version of OpenSSH (available from http://opensource.apple.com — specifically for Mavericks http://opensource.apple.com/tarballs/OpenSSH/OpenSSH-186.tar.gz).

I first tested integration of the NSNotificationCenter API on a non-Apple version of OpenSSH after failing to get my friend’s Growl-based notification patches to work (I didn’t want to pay $3.99 for Growl in the Mac App Store). Prior to Mac OS X 10.8, my friend @kang had produced the following work for getting notifications out of ssh-agent(1):

https://www.insecure.ws/2013/09/25/ssh-agent-notification/

EDIT: My friend just made a patched version of Simon G. Tatham’s Pageant that provides native Windows notifications — http://www.twitpic.com/e2v52x/full — available in both Source (https://github.com/gdestuynder/putty-pagent-notification) and binary release (https://github.com/gdestuynder/putty-pagent-notification/releases).

However, my good friend’s work was on the non-Apple version of OpenSSH and thus my resulting agent binary lacked launchd(8) integration and Keychain support. This meant that if I were to replace the existing /usr/bin/ssh-agent binary in Mavericks with the patched non-Apple version, I would lose those features previously discussed above, let alone that his patches were for an external notification system versus the new built-in Notification Center.

So I set out to build a drop-in replacement for Mavericks’ /usr/bin/ssh-agent carrying all the integration of the original.

EDIT: Keep reading toward the end for GitHub source link and link to Mavericks binary.

After unpacking Apple’s OpenSSH-186.tar.gz (link above) and applying my patches to add support for NSNotificationCenter, I quickly learned that some additional non-standard configure arguments were required.

Within the top-level openssh directory (of the unpacked OpenSSH-186.tar.gz archive) I executed the following to produce a binary truly worthy of replacing the stock /usr/bin/ssh-agent:

./configure --with-pam --with-audit=bsm
make

After which I had a patched ssh-agent binary in the current working directory, suitable for doing the following:

sudo cp -avn /usr/bin/ssh-agent{,.orig}
sudo cp ssh-agent /usr/bin/ssh-agent
killall ssh-agent

The very next ssh(1) command I execute causes a new fresh instance of my patched ssh-agent(1) to be invoked. The first time my patched agent attempts to access my Keychain, I get the following audit request (image below):

Granting patched ssh-agent(1) access to Keychain for the first time.

Granting patched ssh-agent(1) access to Keychain for the first time.

I choose the “Always Allow” option, knowing that if I want to change it later I can locate the private key in Keychain Access (HINT: Filter on “ssh”) and press the Command-I keyboard shortcut to edit Access Controls for that key.

My patched agent is programmed to hook into the new Notification Center and send you a notification when signing requests (image below).

Notification Center message from ssh-agent(1)

Notification Center message from ssh-agent(1)

Now every time I authenticate by way of private key, regardless of whether that key was loaded once with “ssh-add” or from the Keychain imported via “ssh-add -K“, I get a message from the Notification Center.

EDIT: The primary concern that inspired this work is when you enable agent forwarding to remote hosts. Logging into a compromised host with agent forwarding, a hacker can use your agent to log into yet uncompromised systems to which you have access via your agent. While notifications won’t prevent the use of your agent to reach new machines should you enable forwarding through a compromised system, the notifications will let you know when to (a) shut down your agent (b) take inventory of active connections where agent forwarding was enabled and (c) cull the logs for machines you were forwarding through in an effort to find suspicious activity indicating either a malicious hacker or black-hat administrator on the system. For a corporate laptop that primarily only ever connects to corporate networks (where such activity is never to be expected), a notification is more than adequate to quell the problem (versus a public network where it may be more advantageous to use an ask-pass utility to confirm key usage rather than notify after open usage).

How those messages appear are configured in the System Preferences for Notifications (easily accessible via the gear icon at bottom-right of the Notification Center Drawer). Only after you’ve received your initial message from the patched ssh-agent(1) can you then configure the specifics of how its messages appear. The patched agent pretends to be Keychain Access when talking to the Notification Center, so configuring the style and attributes for Keychain Access notifications will allow you to customize how these agent notifications appear.

As notifications are delivered, they stack up in the pull-out drawer of the Notification Center. I like to clear the log of ssh-agent(1) notifications before I go to bed and check for any when I wake up (perhaps indicating that my agent signed a login request when I wasn’t looking).

By default, I’ve made the agent play the “Submarine” sound during notifications. This can be changed by making your own build from the source code I’ve branched into Git offered below:

https://github.com/devinteske/apple/tree/master/OpenSSH-186/openssh

But you don’t have to modify the code if all you want to do is disable the notification sound. Within the Notifications section of System Preferences you can state that you don’t want the Keychain Access notifications to play a sound.

Last, but not least, I have made a precompiled binary with my modifications so you can simply download and copy into place (using the below command-line instructions):

curl -OL http://druidbsd.sf.net/download/ssh-agent+notifications.osx-10.9.2.tbz
tar jxf !$:t
sudo cp -avn /usr/bin/ssh-agent{,.orig}
sudo cp -fv ssh-agent /usr/bin/
killall ssh-agent

Executing these 5 commands and then logging into a remote machine with ssh(1) using private keys should generate a notification from Keychain Access (pictured above). Cheers!

If you need to rollback to the original ssh-agent for any reason, the above steps made a backup that can be restored typing the following commands:

sudo mv -f /usr/bin/ssh-agent{.orig,}
killall ssh-agent

Wherein your next ssh(1) command will cause the original ssh-agent(1) to be loaded (which doesn’t support NSNotificationCenter API).

Posted in Apple | Tagged , , , , , , | Comments Off

Debugging FreeNAS 9.2.0 System

I recently found my FreeNAS box rebooting itself in the middle of the night (every night) and I wanted to know what was going on. I first tried enabling the debug kernel (see below picture) but that didn’t work nor did it get me any closer to understanding the issue at-hand.

"Enabling the FreeNAS Debug Kernel from the WebUI"

Enabling the FreeNAS Debug Kernel from the WebUI

The idea behind enabling the debug kernel was that it would drop me into the kernel debugger rather than rebooting 15 seconds after an unattended fatal error (leaving zero logs to analyze). But much to my surprise, the system rebooted anyway.

The next step was to set up remote syslog to allow FreeNAS to log to an external box. The first thing I did was connect my FreeNAS server directly to a FreeBSD system. The FreeNAS system will be configured to log everything to the FreeBSD system; so when FreeNAS reboots I will have a log of what lead up to that event.

I configured the secondary network interface on each server to talk on the same subnet. On the FreeNAS side, I went into the WebUI and configured the second interface of em1 to have IPv4 address 192.168.1.2 with a /24 subnet (255.255.255.0). When that was done, it looked like the below picture.

"Configuring a local second network interface in the FreeNAS WebUI"

Configuring a local second network interface in the FreeNAS WebUI

After that, I logged into my FreeBSD server and prepared it to receive remote syslog data.

ifconfig em1 inet 192.168.1.1/24
echo 192.168.1.2 freenas >> /etc/hosts
sysrc syslogd_enable=YES
sysrc syslogd_flags="-a freenas -v -v"
echo "+freenas" >> /etc/syslog.conf
echo "*.* /var/log/freenas.log" >> /etc/syslog.conf
touch /var/log/freenas.log
service syslogd restart

Now SSH into your FreeNAS box and execute the following:

mount -uw /
echo 192.168.1.1 freebsd >> /conf/base/etc/hosts
echo 192.168.1.1 freebsd >> /etc/hosts
mount -ur /

Then perform the last final bit, which is to go into the FreeNAS WebUI and enter a value of “freebsd” for the syslog server (see below picture).

"Where to configure a remote syslog server fro within the FreeNAS WebUI"

Where to configure a remote syslog server fro within the FreeNAS WebUI

At this point, you should be able to execute via SSH on the FreeNAS system:

logger foo

And get the following entry in /var/log/freenas.log on your FreeBSD system:

Jan 26 04:42:37 freenas root: foo

Cheers!

Of course, the jury is still out on exactly why my system is crashing, but I suspect a failing hard disk (the logs being streamed to the FreeBSD system seam to indicate via smartd messages a prefail condition for the boot disk).

Posted in FreeNAS | Comments Off

Recovering FreeNAS 9.2.0

If you should find that you have a FreeNAS system for which you have forgotten the root password, here’s an easy way to recover your system.

First, you’ll need to get shell access. If you’ve left the console menu enabled, then resetting the root password is as easy as 1, 2, 3. However if you have disabled the console menu for security reasons, keep reading for the prescribed recipe.

If you are lucky enough to have a user on the system that has sudo privileges, the first step is to log in as that user. If you do not have such a user, the first step is instead to boot into single-user mode.

TIP: Since FreeNAS has a 1-second timeout on the boot menu, you’ll have to repeatedly tap the space bar when you see it starting to load the kernel; with perseverance you will be able to abort the 1-second timeout to interact with the boot menu wherein single-user mode can be enabled.

Once you’re on the shell (either as root in single-user mode or as a secondary user with sudo access), the first task at-hand is to make the root filesystem read/write by executing:

mount -a         # Only required if running single-user
mount -uw /

Once this is completed, we can then perform the first of a two-step process to set a new password for the root account (restoring access to both SSH and the WebUI). Execute:

passwd root

This is only the first step on a FreeNAS system. The work done by the above command will be undone by the /etc/ix.rc.d/ix-passwd boot script every time the machine boots. We need to take the temporary work performed by the above command and make it permanent by copying the information into the FreeNAS SQLite accounting database.

This can be performed by executing:

sqlite3 /data/freenas-v1.db \
    "UPDATE account_bsdusers SET bsdusr_unixhash=\"$( \
        awk -F: '$1=="root"{print $2;exit}' /etc/master.passwd \
    )\" WHERE bsdusr_username=\"root\""

If you are running as a user with sudo privilege (instead of running as root in single-user mode), here is a sudo endowed version of the above command:

sudo sqlite3 /data/freenas-v1.db \
    "UPDATE account_bsdusers SET bsdusr_unixhash=\"$( \
        sudo awk -F: '$1=="root"{print $2;exit}' /etc/master.passwd \
    )\" WHERE bsdusr_username=\"root\""

To check your work, here’s a command to extract the hash from the SQLite database:

sqlite3 /data/freenas-v1.db "SELECT bsdusr_unixhash \
    FROM account_bsdusers WHERE bsdusr_username=\"root\""

At this point, if you are running in single-user mode, type…

exit

…to continue booting. If instead you used a sudo capable user to make the change, there is no need to reboot as the system accounting table in the FreeNAS SQLite database is in-sync with the FreeBSD system accounting files (/etc/master.passwd; /etc/spwd.db; etc.).

You should now be able to log into the FreeNAS WebUI with the updated password. Cheers!

Posted in FreeNAS | Comments Off

FreeNAS 9.2.0 Development

I’ve been diving into FreeNAS development lately. In this blog post, I’ll talk a little bit about what I did to make development life on the FreeNAS appliance more livable from a FreeBSD developer’s perspective.

The first thing you notice after booting up FreeNAS is that it has a console menu. This was my first stop to getting the box online. A couple of keystrokes and we had an IP address. Load up the IP in your browser and you’re ready to start configuring FreeNAS. We set a root password (via the initial WebUI), and you’re off to the races.

Now comes time to customize FreeNAS for development purposes. Most people won’t want to (and shouldn’t) make these adjustments as they are strictly for the purpose of developing enhancements to FreeNAS (right on the box itself). These changes are not supported by the FreeNAS support team (I cannot emphasize this enough).

As a pre-requisite to getting any customizations in-place, we’ll need to be able to log into the UNIX command-line. In the WebUI, select “Services”, scroll down to “SSH” and hit the toggle to turn it “ON”. Then click the wrench logo just off to the right of the SSH toggle. To log in with the root password using SSH, we need to first enable the security feature which comes disabled by default. Check the configuration option labeled “Login as Root with password” (otherwise you can only login as root with a private key).

In my case, I want to develop enhancements before I configure my storage. So it is rather inconvenient that the FreeNAS WebUI only allows you to create users with home directories within configured storage. For your average consumer, this restriction is more than acceptable considering the boot media is often very small and has little free space. However, I need a place to store the patches that I will generate while working on the Volume Manager code and despite what it lacks in free space, the thumb drive or disk that FreeNAS boots from is ideal for this.

The trick to adding a user with non-standard home directory appears to be: first add your user from the WebUI, initially selecting “/nonexistent” as the home-directory (only valid choice when you have yet to configure a storage volume); and then afterward use the following command from the command-line of your FreeNAS appliance to change the home-directory:

sqlite3 /data/freenas-v1.db "UPDATE account_bsdusers \
    SET bsdusr_home=\"/data/home/USER\" WHERE bsdusr_username=\"USER\""

Note that every time you edit this user from the WebUI, you will need to change the home-directory to “/nonexistent” to force the WebUI to accept/save the changes. Afterward, repeat the above command to restore the desired home directory.

The next issue that I run into is perhaps less common. This issue is specific to my ISP. I require the ability to add the following line to the file /etc/dhclient.conf:

supersede domain-name "mydomain";

The reason for this is because I use my ISPs DHCP and the offer contains a domain of XYZ(dot)com, while I prefer ABC(dot)com. By adding the above line to /etc/dhclient.conf, the underlying FreeBSD system will “do the right thing” and supersede the domain provided by my ISP with my desired one. In the end, /etc/resolv.conf gets auto-populated with what I need while still using WAN-based DHCP (which itself is uncommon and seldom available).

The gotcha with simply modifying /etc/dhclient.conf is that you’ll lose your changes upon reboot. You really need to put the directive into:

/conf/base/etc/dhclient.conf

But before you can edit this file, you need to re-mount the root filesystem as read-write by executing:

mount -uw /

Then you can change the files in /conf and either reboot or execute the following to make the root filesystem read-only again:

mount -ur /

NOTE: This command may take a few moments as it needs to clear pending writes.

Next, I like to install a few packages like bash, vim-lite, pstree, iftop, and sudo (NB: root filesystem needs to be read-write before executing):

pkg_add -r bash
pkg_add -r vim-lite
pkg_add -r pstree
pkg_add -r iftop
pkg_add -r sudo

I’m sure that I install more than just the above, but since the FreeNAS appliance makes /var a memory filesystem, there’s no backup record (usually /var/db/pkg) of installed packages. Once you reboot, you’ll lose the record of what’s installed-versus-not.

ASIDE: It’s really convenient that FreeNAS allows you to specify that a group be allowed to execute sudo. Rather than checking the box for each individual user that you would like to allow sudo access, using a group means you can then store files as group-writable on the disk — simplifying team-based system administration.

After performing all these customizations, I think it’s time to make a backup. To make a backup of my FreeNAS thumb drive, I’ve had success in using version 0.9 of Win32DiskImager from Launchpad.net. Just stick the thumb drive into a Window system, launch Win32DiskImager, and use the “Read” option to make an image of your thumb drive. This image can then be stored for backup or imaged to another device.

I’m actually not a huge fan of running my OS from a thumb drive. So the first thing I do with my backup image is to write it to an internal Solid State Disk. I copied the backup image of my thumb drive to a FreeBSD server where the SSD was installed as an extra disk, and executed the following command to write the image to the drive:

dd if=backup.img of=/dev/ada3 bs=64k

NOTE: /dev/ada3 is the SSD installed as an extra disk.

After that has completed, I remove the SSD from my FreeBSD server and insert it into the 1U that used to boot FreeNAS from thumb drive.

All is dandy; the SSD boots up just like the thumb drive (except many times faster, shaving over 30 seconds from the boot process).

Next, I wish that the /data directory (mounted from /dev/ufs/FreeNASs4) was larger than the scant 20MB initially created by the installer. The SSD that I just cloned FreeNAS to is much larger than 2GB, leaving just over 80GB of unused free space. Being-one to never waste (and especially since I now have a user with a home directory of /data/home/user ready to utilize said space), I really want the /data partition to reach all the way to the end of the drive.

First, stick the drive into another FreeBSD or FreeNAS system (I used FreeBSD 9.2-STABLE @r260095M) where it is visible as an “extra” (unused) disk. Then execute the following steps to reclaim the unused free space (assigning it to the /data partition):

mount /dev/ufs/FreeNASs4 /mnt
mkdir /foo
rsync -avH /mnt/ /foo/
umount /mnt
gpart resize -i 4 ada3
newfs -b 32768 -f 4096 -L FreeNASs4 /dev/ada3s4
mount /dev/ufs/FreeNASs4 /mnt
rsync -avH --delete /foo/ /mnt/
umount /mnt
rm -Rf /foo

Now I remove the SSD, replace it into my FreeNAS chassis and boot it up. Voila, 82G of free space for /data.

FreeNAS is now ready for a collective team of multiple system administrators, able to treat the system less like an appliance and more like a sandbox for development. Cheers!

Posted in FreeNAS | Comments Off

FreeBSD Installer Enhancements

I work on FreeBSD as the current maintainer/author of the following [released] works:

Now that introductions are out of the way, let’s meet our topic:

The FreeBSD Installer

If you’ve been living under a rock (or in a cave without Internet access) for the past 3+ years, here’s the highlights for what you’ve been missing in the FreeBSD Installer arena:

+ sysinstall(8) had a good run from 4.x to 8.x
+ 9.0-RELEASE: Introduced bsdinstall(8) [new default installer]
9.1-RELEASE: No noticeable changes
+ 9.2-RELEASE:
        + Improved bsdinstall(8) scripting
        + Introduced bsdconfig(8) [bsdinstall helper]
        + Introduced sysrc(8)

Before continuing, let’s take a brief moment to give credit where credit is due:

~23,000 Lines: Legacy sysinstall(8) by Jordan Hubbard
NOTE: Including the hard work of countless others (including myself)

~6,000 Lines: New installer bsdinstall(8) by Nathan Whitehorn
NOTE: Including many patches from others (including myself)

Meanwhile, I proudly take credit for the following:

~33,000 Lines: New installer companion bsdconfig(8)
~1,000 Lines: System management tool, sysrc(8)

An amazing amount of work has gone into both bsdinstall(8) and bsdconfig(8) in an attempt to gain momentum toward a future away from sysinstall(8).

However, exactly what is the future with respect to our Installer? It’s often tempting to sit down and look at where we came from and immediately [with ease] point out any number of short-comings in the replacements. While hindsight is paramount to preserving backward compatibility, we should be ever mindful to avoid simply reproducing a new generation of the same thing.

That being said, and with so much effort being put into a complete rewrite, I believe some new high-level goals and ideas can pave a path avoiding the dreaded bikeshed situation (see bikeshed.com; or perhaps teal.bikeshed.com is your cup of tea).

This is a nuclear reactor style look at the FreeBSD Installer from the top-down. Note however, that as we perform a top-down view, design was performed from the bottom-up. We’ll start our top-down view with pretty pictures and end with the very words and code that inspired this blog entry.

Starting at the top, let’s focus on the User Interface. The UI is still a fairly broad topic, so let’s further focus on a subset of the UI, realtime feedback. Our initial focal point will be the UI management for providing feedback during data input/output operations.

User Interface :: Data I/O Feedback

View below, a picture of the distfetch module of bsdinstall(8):

bsdinstall(8) distfetch module

The distfetch module of FreeBSD bsdinstall(8)

NOTE: To get proper box drawing characters on the system console under FreeBSD 9.0 or higher, you must change the default terminal type in ttys(5) from xterm to cons25. This is easy in FreeBSD 9.2 or higher — execute bsdconfig syscons_ttys, select IBM437 (VGA default), execute init q, then exit, and log back into the system console.

Meanwhile, here’s what legacy sysinstall(8) looks like:

"sysinstall(8) distExtractAll"

The distExtractAll functionality of FreeBSD sysinstall(8)

As you can see from the above two pictures, bsdinstall(8) (top) is an improvement over sysinstall(8) (bottom). Perhaps the most noticeable change is that bsdinstall(8) presents a single widget to the user, providing feedback on all requested distributions within a single screen; meanwhile sysinstall(8) presents only a single progress bar widget, multiple times in a series.

While this is indeed an improvement, a few things come to mind:

  • bsdinstall(8) doesn’t display data/rate information
  • The progress bar shouldn’t be the only indicator that data is moving
  • A cursor block appears oddly pinned within the bsdinstall(8) widget
  • bsdinstall(8) separates fetching vs extraction (sysinstall(8) does not)

However, perhaps the most important observation should be that both applications operate serially. Both bsdinstall(8) and sysinstall(8) do not proceed to the next distribution until finishing the previous one. Since the data in each distribution is unique, it is not a requirement to unpack them serially but rather, how it’s simply been done (for over 15 years).

When bsdinstall(8) improved the UI to provide a multi-progress widget, the obvious question of concurrent parallelism should be raised. From a UI standpoint, once you have a widget that can display progress for multiple items, the next logical step is to allow the “Pending” items to be processed without delay. In other words, the UI we have today for distfetch, is a great starting point but we can take it further.

For example, view below a prototype that I am developing:

"FreeBSD Installer Prototype"

A prototype replacement for FreeBSD bsdinstall(8) distfetch/distextract

NOTE: Both dialog(1) and dialog(3) render ANSI bold as gray on the system console in 9.0 or higher. Compare to the following image, executing the same program via SSH in which bold is exactly what you’d expect.

NOTE: The above was taken with a value of en_US.ISO8859-1 for the LC_ALL environment variable, nicely resulting in comma-separators in the KBytes/sec status line entry. See localeconv(3).

The prototype that I am developing can also replace other [less than informative] areas of the system. For example, the following picture shows how [deprecated] sysinstall(8) told the user that it was downloading the 24MB INDEX file (commonly via FTP):

"sysinstall(8) configPackages"

The configPackages functionality of FreeBSD sysinstall(8)

As you can see above, sysinstall(8) is pretty spartan when it comes to providing any feedback in this area. The replacement, bsdconfig(8), does not yet bring any notable changes (pictured below):

"bsdconfig(8) packages"

The packages module of FreeBSD bsdconfig(8)

Neither deprecated sysinstall(8) nor new bsdconfig(8) provide any information when [down]loading packages/INDEX. There’s no progress bar, no indicator that data is moving, and no data/rate info. My prototype application (named fdpv(1)) can improve the user experience here by providing those things (pictured below).

"fdpv(1) INDEX fetch"

Prototype application for handling I/O streams

However, perhaps the best use for such a tool is to improve the way packages are installed. Pictured below is deprecated sysinstall(8), installing a single package.

"sysinstall(8) packageAdd"

Legacy/deprecated sysinstall(8) installing vim-lite via FTP

Not quite as blunt as the fetching of packages/INDEX; this time at least sporting data/rate information in the status line. However, moving forward and away from sysinstall(8), let’s see what replacement bsdconfig(8) has for us:

bsdconfig(8) packageAdd

bsdconfig(8) packageAdd installing vim-lite

Notice a slight regression in moving forward. New bsdconfig(8) lacks data/rate information that deprecated sysinstall(8) provided.

I plan to fix the regression by introducing fdpv(1) as a new back-end for package installation. Here’s what it could look like:

"fdpv(1) based packageAdd"

A prototype packageAdd that uses fdpv(1)

NOTE: The above is a Photoshop‘ed image. Concurrency has not yet been programmed.

To better understand the fdpv(1) prototype, here is a picture with annotations showing the layout of information and where it is drawn from (no pun intended).

"fdpv(1) based packageAdd with annotations"

Inormation layout of fdpv(1) display

As you can see, the fdpv(1) tool is just a conduit. As such, the tool can be used for both legacy package management as well as pkgng (new default package management tool in FreeBSD).

For the sake of discussion regarding package management, notice the re-introduction of the status line (at bottom) to provide data/rate info (enumerating total throughput) as well as concurrent parallel processing of multiple dependencies during the installation of xorg (pictured).

This should dramatically reduce the amount of time required to install packages because independent dependencies, whom already have their own dependencies satisfied, can be installed in-parallel rather than waiting.

Just how many concurrent workers are allowed? Well, that would be up to Grand Central Dispatch (GCD). Programmed to use GCD, the utility should allow transparent scaling, respective to hardware capability and machine load.

Of course, I have to admit to not knowing GCD yet, so I’ve cast a fellow FreeBSD committer to help with that end. I was originally planning on using POSIX Threads aka pthreads and fancy mutex signaling. But Grand Central Dispatch looks much more promising.

What I’ve accomplished thus far (as of tagged-version 0.5) on this project is the following:

  • Full UI integration with: dialog(3) [default], dialog(1) [-D], and Xdialog(1) [-X]
  • Nearly complete manual (man(1) page) documenting syntax of command-line utility and all its flags.
  • Created test programs that exercise expected usage.
  • Prepared the code for clang (a requirement to use GCD).

What’s not done however, is the following:

  • For package management, a command-line utility is inadequate. The main() portion of the utility needs to be translated into a library (libfdpv) allowing the integration of libpkg.
  • Concurrent handling of file arguments via Grand Central Dispatch.
  • Actual handling of I/O (reads/writes on file paths).

Those things need to be finished before we can import this new tool to the FreeBSD base system. Once those things are finished and fdpv(1) is in the base system, we can then start rewriting bsdinstall(8) and bsdconfig(8) to use it, providing enhanced I/O data feedback where we need it most (the installation of the system and package maintenance, respectively).

Pure bonus is the ability to work in X11 (shown below).

"fdpv(1) in X11 mode using x11/xdialog from FreeBSD ports"

Various dialogs showing fdpv(1) in X11 mode

This matches bsdconfig(8)‘s already built-in ability to work entirely in X11, meanwhile bsdinstall(8) has to slowly be reworked to support X11 (by introducing new tools such as fdpv(1) to replace the existing C components that currently support only dialog(3)).

The code repository for fdpv(1) is at the below address:

http://druidbsd.cvs.sf.net/druidbsd/fdpv/

NOTE: There is a “Download GNU Tarball” link at the bottom as well as a pull-down menu for selecting a “Sticky Tag”, allowing easy download of tagged revisions.

We’ve had an in-depth look at Data I/O Feedback, in the next blog post we will be switching over to another User Interface issue, Internationalization.

Thank you for reading.

Posted in FreeBSD | Leave a comment

VIMAGE Jails on FreeBSD-8

While FreeBSD-9.1 has recently been released, I find myself still working in FreeBSD-8 (and at the time of this writing, 8.4 is soon to be released). This article is based on work performed in FreeBSD-8.1 (only because we froze our development environment for several years to prove the technology that I’m going to talk about in this article).

In FreeBSD-8 there is a new feature for “jails” that is not enabled by default (for base information on standard FreeBSD jails, see the Jail Section of the FreeBSD Handbook).

The new optional feature is called “VNET” and it allows each jail to have a private networking stack. However useful and exciting this may sound, these VNET jails require you to “Bring Your Own Network Interface.”

The VNET feature allows you to move network interfaces in/out of the view of a jail. When a network interface is moved into the view of a jail, it is no longer visible to the host of said jail.

A VNET jail starts with an empty network stack.

For example, you can create an ad hoc persistent VNET jail using the following command (requires VIMAGE option enabled in kernel — more on that later):

jail -c vnet name=vj1 host.hostname=vj1 path=/ persist

NOTE: If your kernel is not compiled with the options VIMAGE line, then you’ll get an error of jail: unknown parameter: vnet).

This jail is now a special jail, called a VNET jail. In this jail, ifconfig shows a private network stack which is currently empty except for an unconfigured local-loopback interface (lo0).

You can now move network interfaces into that jail using the below ifconfig syntax from the host:

ifconfig <interface_name> vnet <jail_id>

Meanwhile, you can move the network interface back out-of that jail using the below syntax from the host:

ifconfig <interface_name> -vnet <jail_id>

Notice the -vnet to recover an interface versus vnet to relinquish an interface to a VNET jail.

Where <interface_name> is something like fxp0, em0, bge0, igb0, etc. and <jail_id> is the value shown in the “JID” column of the output produced by the jls utility (executed without arguments).

However, the usefulness of this is still very limiting. With the VNET feature alone, all one is afforded is the ability to move whole network interfaces into and out-of VNET jails. When one does this, the jail and host do not share the network interface but rather the host can no longer see (using ifconfig for example) nor use the network interface.

To alleviate this limitation, a bridging technology is required to allow the host and VNET jail to share a network interface.

Enter netgraph(4), another optional component in FreeBSD — graph based kernel networking subsystem — publicly available since FreeBSD-3.4. By adding a few NETGRAPH options to the kernel we can provide the VNET jails a solid bridging layer able to provide the jails fresh/functional network interfaces.

However, we’re still not to a point of usefulness because the following gaps still exist:

  • How to automate the bootup, shutdown, and management of VNET jails?
  • How to automate the creation, destruction, and bridging of netgraph devices?
  • How to make sure bridged interfaces have unique MAC addresses?
  • How to allow multiple VNET jails to have the same root directory? (considering each sets their own IP from rc.conf(5))

Enter vimage, software I’ve written to tie VNET jails together using netgraph.

NOTE: My “vimage” boot/management script (which lives in /etc/rc.d and oft-executed via “service”) should not be confused with the “vimage” utility from the VirtNet project (imunes.tel.fer.hr/virtnet/).

My vimage is a fork of the jail rc.d script — rewritten to work only with VNET jails and help you harness the most flexibility in network topology and jail structure.

HINT: If you are familiar with the IMUNES project (http://www.imunes.tel.fer.hr), both their work and my own performance-test results were factored into the ultimate choice to settle on netgraph for our bridging needs.

My vimage boot/mgmt script is available here:
http://druidbsd.sourceforge.net/download.shtml#vimage

Let’s get started…

First, you’ll need a custom kernel for your FreeBSD host before you can run any VNET jails, let alone vimage jails (VNET jails with netgraph).

Add the following to a GENERIC FreeBSD kernel config as a minimum:

makeoptions NO_MODULES=yes
options VIMAGE
options NETGRAPH
options NETGRAPH_ETHER
options NETGRAPH_BRIDGE
options NETGRAPH_EIFACE
options NETGRAPH_SOCKET
nooptions SCTP

NOTE: In the future, when VIMAGE is compatible with SCTP, you will not need the nooptions SCTP line.

UPDATE: A known issue is that the non-default options IPFILTER conflicts with VIMAGE in stable/9 (and possibly HEAD too) in that a kernel panic occurs during boot. A backtrace shows the trap originating from fr_resolvenic() in sys/contrib/ipfilter/netinet/fil.c. My current recommendation for running a kernel with both Firewall abilities and VIMAGE is to use the IPFIREWALL family of options and ipfw(8).

Compile your kernel and boot it. Next, grab the FreeBSD package for vimage-1.4 and use the following command to install 2 files (/etc/rc.d/vimage and /etc/rc.conf.d/vimage):

pkg_add vimage-1.4.tbz

Now we’re ready to build a jail. Personally, I use my own “jail_build” script to build jails from binary releases. You can get jail_build from my SourceForge page:

http://druidbsd.sourceforge.net/download.shtml#jail_build

NOTE: If you are planning on running i386 jails under an amd64 kernel, be advised that you may need a patch to your amd64 kernel to solve a problem with applying a default gateway using the 32-bit route command (as discussed here).

Currently jail_build works with any release, 8.x and older (sorry, no 9.x support yet).

How you use jail_build in a nut-shell:

fetch http://druidbsd.sourceforge.net/download/jail_build.txt
mv jail_build.txt jail_build
chmod +x jail_build
mkdir -p /usr/repos
cd /usr/repos
# For example
mkdir 8.3-RELEASE
cd 8.3-RELEASE

# Now pick a single base-URL from below:
# ftp://ftp.freebsd.org/pub/FreeBSD/releases/i386/8.3-RELEASE/
# ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/8.3-RELEASE/
# ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/i386/8.3-RELEASE/
# ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/amd64/8.3-RELEASE/

# Download "base" directory from FTP to local "."
# Download "doc" directory from FTP to local "."
# Download "dict" directory from FTP to local "."
# Download "games" directory from FTP to local "."
# Download "info" directory from FTP to local "."
# Download "manpages" directory from FTP to local "."
# Download "proflibs" directory from FTP to local "."
# Download "kernels" directory from FTP to local "."

You should now have:

/usr/repos/8.3-RELEASE/{base,doc,dict,games,info,manpages,proflibs,kernels}/

TIP: jail_build will automatically probe for /usr/repos/*-{RELEASE,STABLE,CURRENT}

NOTE: You can certainly run older releases as a jail, but generally speaking you should not run newer releases than what your host is running. So if you build an 8.3-RELEASE jail, you should have at least an 8.3 kernel or higher.

Executing jail_build brings up a menu of repositories to select from (living in /usr/repos) and allows you to select one before prompting you to enter the directory you wish to unpack the contents to (creating a new jail root directory). The default directory is /usr/jail/<jail_hostname> (change <jail_hostname> to the fully qualified hostname of the jail — this is suggested for easy tracking but feel free to make this whatever you like).

NOTE: You don’t have to use my jail_build technique to build jails, in fact there is a long-standing tradition of populating jails from source-code. For details on how to build a jail from source, see the Creating and Controlling Jails section of the FreeBSD Handbook.

After you’ve created a jail, it’s time to configure it as a vimage.

In the host’s /etc/rc.conf file, add the following:

vimage_enable="YES" # Set to NO to disable starting of any vimages
vimage_list="vj1" # Space separated list of names of vimages

vimage_vj1_rootdir="/usr/jail/vj1" # vimage's root directory
vimage_vj1_hostname="vj1" # vimage's hostname
vimage_vj1_devfs_enable="YES" # mount devfs in the vimage

This configures your basic vimage (we’ll cover how to give it a network interface in a moment). You can compare this directly to the procedures documented in the Creating and Controlling Jails section of the FreeBSD Handbook.

The same rules that apply to jail_list in the Handbook apply to vimage_list. The methodology is the same because again, /etc/rc.d/vimage is a fork of the documented /etc/rc.d/jail script.

Next, if you only want to move whole network interfaces into the vimage when it is started (and implied, automatically move the interface back out when the vimage is stopped), add the following to the same rc.conf file as above:

vimage_vj1_vnets="igb0" # list of interfaces to give to the vimage

However, the real flexibility comes from using the bridging option based on netgraph. To create a bridged interface for the vimage — leaving the original interface on the host unaffected — add the following instead of the above:

vimage_vj1_bridges="igb0 igb0" # list of interfaces to bridge

In the above example, a single port physical Intel Gigabit network adapter is listed twice — this will create two unique bridged interfaces from the same physical interface. This is not an error and is perfectly valid (imagine simulating a router that runs on two subnets but over the same physical wire).

If you start the vimage at this point (using service vimage start) you can get a list of the bridged interfaces from the jail’s point of view by executing:

jexec vj1 ifconfig

You’ll see (in the bridging example) two network interfaces, one named ng0_vj1 and the other ng1_vj1. The naming convention for bridged interfaces is ng#_NAME where # is a counter that starts at zero and increases by one for each bridge (regardless of the device being bridged) and NAME is the vimage name as seen in vimage_list.

NOTE: Due to an internal limitation, the name of any network interface in FreeBSD cannot exceed 15 characters. For long vimage names, be aware that ng#_NAME will be truncated to be less than 16 characters if necessary.

PRO-TIP: By placing the vimage name within the name of the bridged network interface, it makes it simple to configure multiple vimages to use the same root directory.

The network configuration for the above vimage is configured within the vimage’s own /etc/rc.conf file, example below:

ifconfig_ng0_vj1="inet 192.168.1.100 netmask 255.255.255.0"
ifconfig_ng1_vj1="inet 10.0.0.1 netmask 255.255.255.0"
defaultrouter="192.168.1.1"

Let’s say you had a second vimage named “vj2″ pointed at the same root directory (and therefore the same /etc/rc.conf file). You would then add the following, for example:

ifconfig_ng0_vj2="inet 10.0.0.2 netmask 255.255.255.0"

The vj1 vimage jail would ignore the vj2 entries and vice-versa.

At this point, let’s recap — you should have done the following:

  1. Compiled, installed, and executed a custom kernel that enables the VNET and netgraph(4) requirements.
  2. Installed my vimage package from druidbsd.sf.net.
  3. Created a jail using either the source-method mentioned in the FreeBSD Handbook or using jail_build and a binary release.
    PRO-TIP: You can actually get by in testing without building a discrete jail but instead use “/” as the jail root path. This is perfectly valid and acceptable.
  4. Configured the vj1 test vimage in /etc/rc.conf
  5. Configured vj1 to bridge at least one physical interface using vimage_vj1_bridges (in the example above, we use igb0 — twice)
  6. Configured ifconfig_ng0_vj1 and [optionally] defaultrouter in the /etc/rc.conf file within the vj1 root directory.

At this point, the vimage will boot with the rest of the machine and can be controlled after boot with the following syntaxes:

# Stop, start, or restart all vimages
service vimage stop
service vimage start
service vimage restart

# ... versus: Stop, start, or restart just the vj1 vimage
service vimage stop vj1
service vimage start vj1
service vimage restart vj1

When the vj1 vimage starts, the ng*_vj1 network interfaces will automatically be created with unique MAC addresses and moved into the VNET jail before kickstarting the FreeBSD boot process within the jail. As the jail boots, it will automatically configure the network interfaces through rc.conf(5). You can use jls(8) to see the running vimages and you can use jexec(8) to execute processes inside them (like tcsh(1), ps(1), and ifconfig(8)).

PRO-TIP: ps(1) can produce the JID of all running processes with the syntax:
ps axopid,jid,command

However, what if you want to SSH into the vimage jail? SSH is not automatically started inside the vimage (and in-fact, only the network services required to get the vimage talking to the net are started). The answer is to add a new configuration line to the host machine’s /etc/rc.conf to configure sshd to be started in each/every jail:

vimage_afterstart_services="sshd" # set new default for all vimages

The default for vimage_after_services is NULL but can be set to a space separated list of services (names of scripts living in /etc/rc.d like sshd). If you wanted to leave this the default and only have the vj1 vimage start sshd, you can instead opt for the following line:

vimage_vj1_afterstart_services="sshd"

So now we have a usable framework for creating multiple vimages that can be SSH’d into as (let’s say) development environments. When you scale this out, netgraph starts to shine. Afterall, netgraph is based on graphs.

Without any additional work, simply by using vimage to produce VNET jails with netgraph bridged interfaces, we can produce graphs of the vimage network topology with the following syntax:

ngctl dot | dot -Tsvg -o vgraph.svg

PRO-TIP: The dot(1) utility can produce more than SVG (like PNG, JPEG, and GIF), but I find SVG to be the most scalable and informative (modern SVG viewers such as latent browsers display much information as tool-tips which I find helpful).

Here are some graphs of different topologies we’ve used over the years:

netgraph

warden0.jbsd netgraph of bridges for vimage jails


netgraph

folsom netgraph of bridges for vimage jails

It’s worth noting that if an interface is not bridged, it is shown in the “disconnected” cluster. This does not imply that the network interface is unused within FreeBSD — just that it has not been connected to any nodes within the netgraph layer.

That’s it for now, thank you for reading. Comments welcome. Depending on comments, I may do another installment showing the more exotic things you can do with this configuration.

Cheers,
Devin

Posted in FreeBSD | Leave a comment

New FreeBSD Boot Loader Menu

FreeBSD. It is Free (in the truest sense; with a capital-F) and is a descendant of Berkeley Systems Distribution (specifically BSD-Lite v4.2/4.4), which originated from the University of Berkeley, about 2.5 miles away from where I live (though I’m not a student, nor have I ever been). FreeBSD is an amazingly versatile and flexible UNIX operating system with ancestry reaching back decades.

This entry is for those persons already well versed in this well-edified system.

I would like to announce a completely new — written from scratch — boot menu for the FreeBSD operating system. It is today that I release the code that I’ve been developing for nearly 5 years. Code to enhance the level of options provided to the user at each and every boot of this venerable OS.

Historically, the original menu offered by FreeBSD featured our beloved helper-daemon mascot, named “Beastie.” But soon after, this menu was revamped to [a] lay-rest our triton-carrying/sneaker-wearing assistant in favor of the less-controversial textual “FreeBSD” banner and [b] imbue the menu with new options.

Currently, the default boot menu offers booting normally as well as options for the following:

  • Booting with ACPI support disabled (if you’re running an i386 compatible system)
  • Booting into “Safe Mode” (implies disabling ACPI)
  • Booting verbose
  • Booting into single-user mode

However, one of the things that is not currently possible is the ability to boot into any combination of the aforementioned options. That is to explain, you cannot boot into single-user mode with ACPI disabled but verbosity enabled. The reason for this is obvious as once a menu option has been selected, the system proceeds to boot with those options rather than allowing additional selections to be made. See below.

standard-menu

The Standard FreeBSD Boot Menu

Enter my new menu (pictured below in the following paragraphs), first created in 2006 internally for a proprietary custom installer, now generalized for the rest of the world and being released for all to enjoy. This new loader does not proceed to boot the system after selecting a menu option but rather updates the menu to display the option status, allowing the user to toggle any number of multiple options before pressing ENTER to finally boot with the displayed options.

And for those like myself that balked when Beastie exited stage-left, I have a nice surprise. Some time back, the FreeBSD community held a contest for artists to create a new logo for the FreeBSD project. The results of which can be viewed here: http://www.freebsd.org/logo.html

When the contest ended, I had my fun in running the graphics through some Image-to-ASCII conversion programs without much luck. Months later, I doubled-down my efforts still to no avail, attempting to create an ASCII-art version of the new orb to display at boot time. Six months later, I succeeded in producing a bi-color ASCII-art representation of the “horny orb” logo easily recognizable as the FreeBSD logo. This gem (as well as a B&W copy) have been added to the latest “beastie.4th” module for your viewing pleasure and is enabled by default upon installation.

Here you can lay your eyes on the final edition of the first-offering being made to the public today:

loader_menu-1.0

New FreeBSD Boot Loader Menu

Installing the new boot loader menu couldn’t be easier. After downloading the FreeBSD package from http://druidbsd.sf.net/, use the pkg_add(1) command to install the package, and finally reboot.

Other additions that you will notice are that the replacement menu does not load right away. Another enhancement is that there is a 2 second delay before launching the menu where the user can press Ctrl-C to escape the menu, allowing immediate booting or escape to the loader prompt. Something that the current menu does not allow for. See below.

loading-the-new-menu

Loading the New Menu

The new menu is a system of ANS/FICL Forth (a reverse-polish stack-based language) modules that extend (but do not replace) the standard FreeBSD boot loader (/boot/loader). An interesting note about this package is that it will backup any files that will be replaced during installation to the file /var/backups/loader_menu.backup.tgz. Should you decide that you do not like the new and improved boot loader, you can uninstall it, returning your system back to the original boot loader by using the pkg_delete(1) utility (the backup will automatically be restored upon pkg_delete(1)).

With this initial offering, there are only two short-comings which should certainly be livable for most if not all:

1. The menu that we are replacing only conditionally displays the option to disable ACPI support (based on whether you are running an i386 compatible system), whereas the new loader always presents this option. This should be of no concern unless you’re running a non-i386/amd64 system (such as sparc, pc98, ppc, etc.).

2. The ability to boot into single-user mode by pressing “s” and boot verbose by pressing “v”, etc. is not enabed. The proponents for these options on the FreeBSD-Questions mailing list were clear in that they liked these options because they allowed “one-key” booting into a special mode. However — with the new menu — even if I enabled the use of those keys as shortcuts for toggling these options in the new stateful menu, the user would still have to press an additional key to boot (’1′ or ENTER). So the primary principle use — one-key booting into a special mode — is no longer capable, unless I add these keys not as togglers for the visible menu items but invisible hot-keys for immediately booting into single/verbose mode. I’ll have to sleep on that (maybe as a concession for power-users as invisible options are obviously less than intuitive).

However, don’t let those misgivings dissuade you. All but the most ardent and stringent of surveyors will find those to be shortcomings and overall I predict your experience will be a rather pleasing one.

Some other nice additions are that if/when you select the option to escape to the loader prompt, you are given a way to get back to the menu (simply type “menu” and press ENTER), allowing you to perform work under the hood and then return to the menu as desired (without losing track of the status of the menu options even). As shown below:

escape-to-loader-prompt

Escaping to the Loader Prompt

Overall, the new stateful nature of this menu over the old makes it far less likely that a user will need to reboot if they hit the wrong key. The only mistake that could result in wasted cycles would be if they prematurely hit #1 to boot or hit #7 to reboot. All other options would simply toggle a menu item (except for #6 to escape to the loader prompt, to which they can immediately return to the menu if desired).

Last, but certainly not least, I’m including a completely rewritten password-checking module to replace the existing one (I couldn’t get the standard one to work). See below.

password-prompt

New Password Checking Module

I hope that you like what I’ve developed, and your feedback is certainly welcome. I’ll be cross-posting this to the FreeBSD mailing lists, announcing in RSS feeds, and updating http://druidbsd.sourceforge.net/ all simultaneously to coincide with this blog-post.

Cheers,
Devin

Posted in FreeBSD | Leave a comment

XCode Update Woes – With Solution!

I’ve found that when you upgrade versions of XCode, that you lose the ability to compile against older SDK’s (and therefore, the ability to make a binary targeted for the older platforms). This is further annoying by the fact that every sample project that you download from Apple that is slightly older (by that I mean 6 months), you have to go in and “Edit Project Settings” to change the SDK from “4.0 Missing” to “iOS 4.3″ before you can build/compile, install to device, or even run the App (I’m not complaining — a missing SDK is surely something worthy in commanding such manual intervention but — If you’re loading up over 200 sample apps from Apple, it gets quite tedious having to modify each project before being able to install it to the device).

I found the solution to all this, and you’ll be surprised at how much work Apple makes us do in order to maintain backward compatibility. This problem had me on the ropes for 2 weeks while I tried to figure out how to — with *either* the new XCode 4 or the still-offered legacy XCode 3.2.6 (both released last month) — compile ANY code for my iPhone 3G which — due to Apple’s decisioning — cannot update to the latest iOS. Without the below solution, I would not be able to compile code for the iPhone 3G without downgrading XCode. However, with the below solution, I can use the latest and greatest XCode with the older SDKs to compile code for the older devices.

Step 1: Stay up to date, but make backups!

Whenever Apple issues a new XCode installer, that installer only comes with the SDKs that they want to provide you. Currently, that’s only iOS 4.3 (note: the Device SDK is different than the Simulator SDK). Running said installer will wipe-out any existing old SDKs, leaving you only with what they have provided you in the latest installer. This is certainly cause for alarm, because if your device doesn’t use one of the provided SDKs, you cannot publish for that device. However, there are no rules for the AppStore that says you must baseline your app to a specific SDK, so whatever baseline you want to choose is fine with Apple — as long as it runs smoothly on the Device SDKs that Apple will test your App against.

Naturally, you’ll want to prevent the loss of any existing older SDKs before you install. Go to the Finder (by clicking the “Finder” icon in the Dock, or clicking the desktop background image, for example), and then select from the menubar Go->Go To Folder… (or simply press Apple+Shift+G). This will bring up a dialog prompt allowing you to go to a directory based on UNIX pathname. Enter the text “/Developer/Platforms/iPhoneOS.platform/Developer/SDKs” (without quotes) and press ENTER (aka return). This will open the directory where the SDKs are installed. Before installing the latest XCode Developer Tools from Apple, backup everything in this directory.

Step 2: If you want to install additional SDKs (one’s that you don’t have).

Visit this webpage to find links to the old Developer Tools.

NOTE: We’re not going to be installing these developer tools, but rather we’ll rip-open the images and pluck out only the SDK information (not XCode and associated developer tools such as gcc and friends).

http://iphonesdkdev.blogspot.com/2010/04/old-versions-of-iphone-sdk.html

Once you’ve downloaded the SDK, it should automatically mount on your desktop (if you’re using Safari anyway) as a disk image named “Xcode and iPhone SDK”. Within this disk image is a hidden directory named “Packages”. Since it is hidden, the simplest way to access this directory is to once-again use the Go->Go To Folder… menubar item (or press Apple+Shift+G), entering “/Volumes/Xcode and iPhone SDK/Packages” (without quotes) and pressing ENTER. This will allow you to browse the hidden “Packages” directory on the read-only disk image containing the Developer Tools. Within this directory are files named similar to:

iPhoneSDK3_0.pkg
iPhoneSDK3_1_2.pkg
iPhoneSDK3_1_3.pkg
iPhoneSDK3_1.pkg
iPhoneSDK3_2.pkg
iPhoneSDK4_0.pkg

…ad nauseum. Go ahead and double-click any of these “*.pkg” files to launch the SDK installer (which will install ONLY to the directory “/Developer/Platforms/iPhoneOS.platform/Developer/SDKs”.

As you launch the Installer, be aware that we’ll have to make some adjustments to allow the installation to occur. When prompted to select the target installation disk (immediately after clicking “Continue” on the initial dialog) click the hard disk that you want to install to (even if it’s already highlighted) to cause a “Choose Folder…” button to appear. Click the “Choose Folder…” button and pick somewhere that is writable as you (if you attempt to install to the default location, the installation will fail — because the installer did not pre-authenticate with root privileges as is done in the main-installer opposed to these sub-installers). I chose to make a new directory on my Desktop. After the installation is complete, you should have a “Developer/Platforms/iPhoneOS.platform/Developer/SDKs/{version}” directory in the target installation directory that you chose. This “{version}” directory will need to be plucked from where you installed it and dropped into the “/Developer/Platforms/iPhoneOS.platform/SDKs/” directory.

Once you’ve installed a new SDK, you’ll need to restart XCode.

Immediately after installing the device SDK required by any Apple Sample code and restarting XCode, you’ll then be able to simply select “Device” from the arch menu (at the top-left of the project window), and press Apple+R to run the app on your connected device. No more need to go into the project settings to change the default SDK (sometimes needs to be done for each sub-target) to one that exists.

Happy coding,
Devin Teske

Posted in Apple | Leave a comment

Apple XCode 4 loses CVS support — D’Oh!

I felt that this was worth a blog-post.

So here we are. XCode (Apple’s interactive development system for building applications) version 4 has been released to the general public. I am wonderfully pleased with the simplification/unification efforts but XCode4 has me scratching my head why Apple would remove CVS support.

I look around me and all I see is CVS — sourceforge.net still firmly embraces CVS. Is this a sign that Apple doesn’t like open-source ventures such as sourceforge.net? Was maintaining CVS support too difficult? Were there some capabilities that were lacking in CVS?

As a person well entrenched in the ways of CVS, I find it ineffable that I might have to migrate sizable iOS projects to some other SCM system such as Subversion (SVN) or Git just to maintain GUI integration within XCode. Mind you, I have no qualms with doing all SCM from the command-line and leaving XCode as a debugger/profiler/analyzer/compiler/editor/builder (though I have to admit, having CVS SCM built-in to XCode 3 was oh-so nice — in-fact I use to pitch it to co-workers as a “does everything” editor that could even be used for unix projects too, optionally replacing a system of Makefiles).

I can only hope and pray that in the coming weeks/months, there is an overwhelming public outcry over the removal of CVS from XCode 4 and that Apple — hopefully being the kind-hearted and in-touch kind of people I could only imagine them to be — might resurrect this industry-standard (truly a fallen comrade). Until then, back to the command-line.

admin -ko,
Devin

P.S. If Apple decides to subsequently remove the command-line utility from the Developer Tools installation, I’ll be turning to MacPorts, Fink, or compile it from source (yeah, I’m that determined to use CVS).

P.P.S. With tools like gource [ http://code.google.com/p/gource/ ], nobody in their right mind would switch from their current SCM to any other unless their 15+ year commit history was guaranteed to be preserved in the conversion process.

Posted in Apple | Leave a comment

D-String, an answer to P-String and C-String limitations

C-String’s are delimited by a NULL byte. P-Strings are preceded by a length identifier. Both have their downsides and I’ve developed the solution (it’s called the D-String; D for Data). The C-String’s downfall is that it cannot contain a NULL (else the interpreting language — C — will prematurely terminate the data). The P-String’s downfall is that it cannot represent more than 255 bytes (unless of course you use a wider length identifier in which case you’ve also increased the overhead). The D-String overcomes both of these limitations with minimal overhead. Let’s have a look at the specifics (free of charge).

NOTE: These are my own internal notes. They will be translated into a full technical explanation in another blog posting. However… the fact is that I’ve sat on this technology for 10 years and want to finally make it public. This is the first step in doing so. Last, this will serve as a backup should my iPhone crash (currently the only machine in the world with a documented example of the methodology). This is not meant to be digested by mere mortals (but if you can, all the more power to you — you’ll have a leg-up on the rest of those waiting on the technical discussion).

shxd.rfc04 — dStr data object

Len
0×00 => 0×00
0×01 => 0×01 0×00 0×00 DATA
0xFF => 0xFF 0×00 0×00 DATA
0×0100 => 0×0101 0×00 0×01 DATA
0×0101 => 0×0101 0×00 0×00 DATA
0xFFFF => 0xFFFF 0×00 0×00 DATA
0×010000 => 0×010101 0×00 0×03 DATA
0×010100 => 0×010101 0×00 0×02 DATA
0×010100 => 0×010101 0×00 0×01 DATA
0xFFFFFF => 0xFFFFFF 0×00 0×00 DATA
0×01000000 => 0×01010101 0×00 0×07 DATA
0×01000001 => 0×01010101 0×00 0×06 DATA
0×01000100 => 0×01010101 0×00 0×05 DATA
0×01000101 => 0×01010101 0×00 0×04 DATA
0×01010000 => 0×01010101 0×00 0×03 DATA
0×01010001 => 0×01010101 0×00 0×02 DATA
0×01010100 => 0×01010101 0×00 0×01 DATA
0xFFFFFFFF => 0xFFFFFFFF 0×00 0×00 DATA
0×0100000000 => 0×0101010101 0×00 0x0F DATA
0×0100000001 => 0×0101010101 0×00 0x0E DATA
0×0100000100 => 0×0101010101 0×00 0x0D DATA
0×0100000101 => 0×0101010101 0×00 0x0C DATA
0×0100010000 => 0×0101010101 0×00 0x0B DATA
0×0100010001 => 0×0101010101 0×00 0x0A DATA
0×0100010100 => 0×0101010101 0×00 0×09 DATA
0×0100010101 => 0×0101010101 0×00 0×08 DATA
0×0101000000 => 0×0101010101 0×00 0×07 DATA
0×0101000001 => 0×0101010101 0×00 0×06 DATA
0×0101000100 => 0×0101010101 0×00 0×05 DATA
0×0101000101 => 0×0101010101 0×00 0×04 DATA
0×0101010000 => 0×0101010101 0×00 0×03 DATA
0×0101010001 => 0×0101010101 0×00 0×02 DATA
0×0101010100 => 0×0101010101 0×00 0×01 DATA
0xFFFFFFFFFF => 0xFFFFFFFFFF 0×00 0×00 DATA
0×010000000000 => 0×010101010101 0×00 0x1F DATA
0×010000000001 => 0×010101010101 0×00 0x1E DATA
0×010000000100 => 0×010101010101 0×00 0x1D DATA
0×010000000101 => 0×010101010101 0×00 0x1C DATA
0×010000010000 => 0×010101010101 0×00 0x1B DATA
0×010000010001 => 0×010101010101 0×00 0x1A DATA
0×010000010100 => 0×010101010101 0×00 0×19 DATA
0×010000010101 => 0×010101010101 0×00 0×18 DATA
0×010001000000 => 0×010101010101 0×00 0×17 DATA
0×010001000001 => 0×010101010101 0×00 0×16 DATA
0×010001000100 => 0×010101010101 0×00 0×15 DATA
0×010001000101 => 0×010101010101 0×00 0×14 DATA
0×010001010000 => 0×010101010101 0×00 0×13 DATA
0×010001010001 => 0×010101010101 0×00 0×12 DATA
0×010001010100 => 0×010101010101 0×00 0×11 DATA
0×010001010101 => 0×010101010101 0×00 0×10 DATA
0×010100000000 => 0×010101010101 0×00 0x0F DATA
0×010100000001 => 0×010101010101 0×00 0x0E DATA
0×010100000100 => 0×010101010101 0×00 0x0D DATA
0×010100000101 => 0×010101010101 0×00 0x0C DATA
0×010100010000 => 0×010101010101 0×00 0x0B DATA
0×010100010001 => 0×010101010101 0×00 0x0A DATA
0×010100010100 => 0×010101010101 0×00 0×09 DATA
0×010100010101 => 0×010101010101 0×00 0×08 DATA
0×010101000000 => 0×010101010101 0×00 0×07 DATA
0×010101000001 => 0×010101010101 0×00 0×06 DATA
0×010101000100 => 0×010101010101 0×00 0×05 DATA
0×010101000101 => 0×010101010101 0×00 0×04 DATA
0×010101010000 => 0×010101010101 0×00 0×03 DATA
0×010101010001 => 0×010101010101 0×00 0×02 DATA
0×010101010100 => 0×010101010101 0×00 0×01 DATA
0xFFFFFFFFFFFF => 0xFFFFFFFFFFFF 0×00 0×00 DATA
.
.
.
0×0100000000000000 => 0×0101010101010101 0×00 0x7F DATA
0xFFFFFFFFFFFFFFFF => 0xFFFFFFFFFFFFFFFF 0×00 0×00 DATA
.
.
.
0×01000000000000000000000000000000 => 0×01010101010101010101010101010101 0×00 0x7FFFF DATA
0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF => 0xFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF 0×00 0×0000 DATA

That’s a length identifier of 2^(8*16) or 2^128 or 3.4028236692094e+38 or 340,282 thousand Decillion bytes long. The length identifier is valid with only 3 bytes of overhead preceding the actual DATA (compared to 16 bytes for the length identifier).

Scaling this higher (to 512 bit integers — 64-bytes wide), the overhead would be 9 bytes.

The overhead is always the length of the length-identifier (in bytes) divided by 8 plus one (with the minimum overhead being two bytes at the low end).

If a dStr contains 0 bytes of data, the dStr will be 0×00.

If a dStr contains 1-15 bytes of data, the dStr will be 0xLL 0×00 0xNN DATA (header is 3 bytes). LL is the length of DATA. NN is the encode register.

If a dStr contains 65536-16777215 bytes of data, the dStr will be 0xLLLLLL 0×00 0xNN DATA (header is 5 bytes).

If a dStr contains 16777216-4294967295 bytes of data, the dStr will be 0xLLLLLLLL 0×00 0xNN DATA (header of 6 bytes).

If a dStr contains 4294967296-1099511627775 bytes of data, the dStr will be 0xLLLLLLLLLL 0×00 0xNN DATA (header of 7 bytes).

If a dStr contains 1099511627776-281474976710655 bytes of data, the dStr will be 0xLLLLLLLLLLLL 0×00 0xNN DATA (header of 8 bytes).

If a dStr contains 281474976710655-7.2057594037928e+16 bytes of data, the dStr will be 0xLLLLLLLLLLLLLL 0×00 0xNN DATA (header of 9 bytes).

If a dStr contains 7.2057594037928e+16-1.844674407371e+19 bytes of data, the dStr will be 0xLLLLLLLLLLLLLLLL 0×00 0xNN DATA (header of 10 bytes).

If a dStr contains 1.844674407371e+19-4.7223664828696e+21 bytes of data, the dStr will be 0xLLLLLLLLLLLLLLLLLL 0×00 0xNNNN DATA (header of 12 bytes).

If the dStr contains 4.7223664828696e+21-1.2089258196146e+24 bytes of data, the dStr will be 0xLLLLLLLLLLLLLLLLLLLL 0×00 0xNNNN DATA (header of 13 bytes).

Ad nausea to infinitum.

Posted in Data | Leave a comment