Apple Mac OS X makes working with SSH private keys easy and convenient. Things that you can safely forget and drop from your routine when using Mac OS X are:
- How and when to launch ssh-agent(1)
- Continuously loading your private key
Apple has integrated launchd(8) and Keychain support into ssh-agent(1) so it launches automatically on-demand and can interact with keys stored in Keychain Access.
First, a brief look at the above integration before we go improving things using new features introduced in Mac OS X 10.8 and higher.
EDIT: If you’re reading this thinking “Mac OS X has had Keychain and launchd integration for a while,” skip on ahead to “That being said…” below).
Traditionally when using an agent you would first manually launch an instance of ssh-agent(1) and then follow that up with:
ssh-add ~/.ssh/id_dsa
while on Mac OS X there is:
ssh-add -K ~/.ssh/id_dsa
which both loads your key into the running ssh-agent(1) and imports it into your Keychain. So you’ll find that quite untraditionally if you kill the running agent and then try to ssh(1) to a remote host, a new agent is launched on-demand and it loads the private key through the Keychain (login succeeds despite conventional logic about agents).
In the following picture, we can see previously loaded key(s) if we filter on the word “ssh” in Apple’s Keychain Access application (provided in the Utilities folder accessible via Command-Shift-U keyboard shortcut while in the Finder application):
If you delete this key from Keychain Access application then ssh(1) logins requiring that key no longer succeed as the running agent immediately loses access to said key. So the Keychain can be thought of as a persistent store for the agent; one that is tied to your Mac’s login account.
The automatic re-launching of ssh-agent(1) through launchd(8) means that if you kill the running agent it will be re-launched not immediately but only when access is attempted to a launchd(8)-provided default $SSH_AUTH_SOCK value as-configured in /System/Library/LaunchAgents/org.openbsd.ssh-agent.plist.
Initially when Mac OS X boots, ssh-agent(1) is not started. It is only when you perform your first ssh(1) (or related) command that launchd(8) invokes the following command:
/usr/bin/ssh-agent -l
You can see launchd(8)‘s integration in-action by opening a Terminal immediately after booting your Mac and first executing:
ps axwww | grep ssh-agent
You’ll see there are no instances of ssh-agent(1), yet. Meanwhile, if you execute:
echo $SSH_AUTH_SOCK
You’ll see launchd(8) provided a default value (similar to the following):
/tmp/launch-oLGVUi/Listeners
When any ssh(1) command attempts to access this launchd(8) provided $SSH_AUTH_SOCK listener, launchd(8) invokes the previously mentioned “ssh-agent -l” instance which then accesses private keys stored in your keychain. For example, assuming Remote Login is enabled in the Sharing section of System Preferences (which enables sshd), execute:
ssh localhost
After which, re-execute:
ps axwww | grep ssh-agent
To find a running instance of “/usr/bin/ssh-agent -l” (the “-l” indicating it was launched on-demand by launchd(8)).
This is all very slick integration that quintessentially eases your SSH life. In a nut-shell, it’s private keys provided from your Keychain on-the-fly whenever you use an ssh utility.
That being said, we can certainly improve upon this integration by adding new code to Apple’s version of OpenSSH. Primarily, since ssh-agent(1) can now be loaded on-demand and access data in my Keychain, I would like to get a notification every time someone uses the running agent (for security reasons and also coolness factor).
In Mac OS X Mountain Lion (10.8) and Mavericks (10.9) we are blessed with the new Notification Center API (http://support.apple.com/kb/ht5362). This built-in functionality replaces the need for 3rd-party software such as Growl (http://growl.info) required to perform notifications in previous releases of Mac OS X (10.7 and older).
RESOURCE: Apple’s custom version of OpenSSH (available from http://opensource.apple.com — specifically for Mavericks http://opensource.apple.com/tarballs/OpenSSH/OpenSSH-186.tar.gz).
I first tested integration of the NSNotificationCenter API on a non-Apple version of OpenSSH after failing to get my friend’s Growl-based notification patches to work (I didn’t want to pay $3.99 for Growl in the Mac App Store). Prior to Mac OS X 10.8, my friend @kang had produced the following work for getting notifications out of ssh-agent(1):
https://www.insecure.ws/2013/09/25/ssh-agent-notification/
EDIT: My friend just made a patched version of Simon G. Tatham’s Pageant that provides native Windows notifications — http://www.twitpic.com/e2v52x/full — available in both Source (https://github.com/gdestuynder/putty-pagent-notification) and binary release (https://github.com/gdestuynder/putty-pagent-notification/releases).
However, my good friend’s work was on the non-Apple version of OpenSSH and thus my resulting agent binary lacked launchd(8) integration and Keychain support. This meant that if I were to replace the existing /usr/bin/ssh-agent binary in Mavericks with the patched non-Apple version, I would lose those features previously discussed above, let alone that his patches were for an external notification system versus the new built-in Notification Center.
So I set out to build a drop-in replacement for Mavericks’ /usr/bin/ssh-agent carrying all the integration of the original.
EDIT: Keep reading toward the end for GitHub source link and link to Mavericks binary.
After unpacking Apple’s OpenSSH-186.tar.gz (link above) and applying my patches to add support for NSNotificationCenter, I quickly learned that some additional non-standard configure arguments were required.
Within the top-level openssh directory (of the unpacked OpenSSH-186.tar.gz archive) I executed the following to produce a binary truly worthy of replacing the stock /usr/bin/ssh-agent:
./configure --with-pam --with-audit=bsm
make
After which I had a patched ssh-agent binary in the current working directory, suitable for doing the following:
sudo cp -avn /usr/bin/ssh-agent{,.orig}
sudo cp ssh-agent /usr/bin/ssh-agent
killall ssh-agent
The very next ssh(1) command I execute causes a new fresh instance of my patched ssh-agent(1) to be invoked. The first time my patched agent attempts to access my Keychain, I get the following audit request (image below):
I choose the “Always Allow” option, knowing that if I want to change it later I can locate the private key in Keychain Access (HINT: Filter on “ssh”) and press the Command-I keyboard shortcut to edit Access Controls for that key.
My patched agent is programmed to hook into the new Notification Center and send you a notification when signing requests (image below).
Now every time I authenticate by way of private key, regardless of whether that key was loaded once with “ssh-add” or from the Keychain imported via “ssh-add -K“, I get a message from the Notification Center.
EDIT: The primary concern that inspired this work is when you enable agent forwarding to remote hosts. Logging into a compromised host with agent forwarding, a hacker can use your agent to log into yet uncompromised systems to which you have access via your agent. While notifications won’t prevent the use of your agent to reach new machines should you enable forwarding through a compromised system, the notifications will let you know when to (a) shut down your agent (b) take inventory of active connections where agent forwarding was enabled and (c) cull the logs for machines you were forwarding through in an effort to find suspicious activity indicating either a malicious hacker or black-hat administrator on the system. For a corporate laptop that primarily only ever connects to corporate networks (where such activity is never to be expected), a notification is more than adequate to quell the problem (versus a public network where it may be more advantageous to use an ask-pass utility to confirm key usage rather than notify after open usage).
How those messages appear are configured in the System Preferences for Notifications (easily accessible via the gear icon at bottom-right of the Notification Center Drawer). Only after you’ve received your initial message from the patched ssh-agent(1) can you then configure the specifics of how its messages appear. The patched agent pretends to be Keychain Access when talking to the Notification Center, so configuring the style and attributes for Keychain Access notifications will allow you to customize how these agent notifications appear.
As notifications are delivered, they stack up in the pull-out drawer of the Notification Center. I like to clear the log of ssh-agent(1) notifications before I go to bed and check for any when I wake up (perhaps indicating that my agent signed a login request when I wasn’t looking).
By default, I’ve made the agent play the “Submarine” sound during notifications. This can be changed by making your own build from the source code I’ve branched into Git offered below:
https://github.com/devinteske/apple/tree/master/OpenSSH-186/openssh
But you don’t have to modify the code if all you want to do is disable the notification sound. Within the Notifications section of System Preferences you can state that you don’t want the Keychain Access notifications to play a sound.
Last, but not least, I have made a precompiled binary with my modifications so you can simply download and copy into place (using the below command-line instructions):
curl -OL http://druidbsd.sf.net/download/ssh-agent+notifications.osx-10.9.2.tbz
tar jxf !$:t
sudo cp -avn /usr/bin/ssh-agent{,.orig}
sudo cp -fv ssh-agent /usr/bin/
killall ssh-agent
Executing these 5 commands and then logging into a remote machine with ssh(1) using private keys should generate a notification from Keychain Access (pictured above). Cheers!
If you need to rollback to the original ssh-agent for any reason, the above steps made a backup that can be restored typing the following commands:
sudo mv -f /usr/bin/ssh-agent{.orig,}
killall ssh-agent
Wherein your next ssh(1) command will cause the original ssh-agent(1) to be loaded (which doesn’t support NSNotificationCenter API).