I recently found my FreeNAS box rebooting itself in the middle of the night (every night) and I wanted to know what was going on. I first tried enabling the debug kernel (see below picture) but that didn’t work nor did it get me any closer to understanding the issue at-hand.

The idea behind enabling the debug kernel was that it would drop me into the kernel debugger rather than rebooting 15 seconds after an unattended fatal error (leaving zero logs to analyze). But much to my surprise, the system rebooted anyway.

The next step was to set up remote syslog to allow FreeNAS to log to an external box. The first thing I did was connect my FreeNAS server directly to a FreeBSD system. The FreeNAS system will be configured to log everything to the FreeBSD system; so when FreeNAS reboots I will have a log of what lead up to that event.

I configured the secondary network interface on each server to talk on the same subnet. On the FreeNAS side, I went into the WebUI and configured the second interface of em1 to have IPv4 address 192.168.1.2 with a /24 subnet (255.255.255.0). When that was done, it looked like the below picture.

After that, I logged into my FreeBSD server and prepared it to receive remote syslog data.

ifconfig em1 inet 192.168.1.1/24
echo 192.168.1.2 freenas >> /etc/hosts
sysrc syslogd_enable=YES
sysrc syslogd_flags="-a freenas -v -v"
echo "+freenas" >> /etc/syslog.conf
echo "*.* /var/log/freenas.log" >> /etc/syslog.conf
touch /var/log/freenas.log
service syslogd restart

Now SSH into your FreeNAS box and execute the following:

mount -uw /
echo 192.168.1.1 freebsd >> /conf/base/etc/hosts
echo 192.168.1.1 freebsd >> /etc/hosts
mount -ur /

Then perform the last final bit, which is to go into the FreeNAS WebUI and enter a value of “freebsd” for the syslog server (see below picture).

At this point, you should be able to execute via SSH on the FreeNAS system:

logger foo

And get the following entry in /var/log/freenas.log on your FreeBSD system:

Jan 26 04:42:37 freenas root: foo

Cheers!

Of course, the jury is still out on exactly why my system is crashing, but I suspect a failing hard disk (the logs being streamed to the FreeBSD system seam to indicate via smartd messages a prefail condition for the boot disk).

If you should find that you have a FreeNAS system for which you have forgotten the root password, here’s an easy way to recover your system.

First, you’ll need to get shell access. If you’ve left the console menu enabled, then resetting the root password is as easy as 1, 2, 3. However if you have disabled the console menu for security reasons, keep reading for the prescribed recipe.

If you are lucky enough to have a user on the system that has sudo privileges, the first step is to log in as that user. If you do not have such a user, the first step is instead to boot into single-user mode.

TIP: Since FreeNAS has a 1-second timeout on the boot menu, you’ll have to repeatedly tap the space bar when you see it starting to load the kernel; with perseverance you will be able to abort the 1-second timeout to interact with the boot menu wherein single-user mode can be enabled.

Once you’re on the shell (either as root in single-user mode or as a secondary user with sudo access), the first task at-hand is to make the root filesystem read/write by executing:

mount -a         # Only required if running single-user
mount -uw /

Once this is completed, we can then perform the first of a two-step process to set a new password for the root account (restoring access to both SSH and the WebUI). Execute:

passwd root

This is only the first step on a FreeNAS system. The work done by the above command will be undone by the /etc/ix.rc.d/ix-passwd boot script every time the machine boots. We need to take the temporary work performed by the above command and make it permanent by copying the information into the FreeNAS SQLite accounting database.

This can be performed by executing:

sqlite3 /data/freenas-v1.db \
    "UPDATE account_bsdusers SET bsdusr_unixhash=\"$( \
        awk -F: '$1=="root"{print $2;exit}' /etc/master.passwd \
    )\" WHERE bsdusr_username=\"root\""

If you are running as a user with sudo privilege (instead of running as root in single-user mode), here is a sudo endowed version of the above command:

sudo sqlite3 /data/freenas-v1.db \
    "UPDATE account_bsdusers SET bsdusr_unixhash=\"$( \
        sudo awk -F: '$1=="root"{print $2;exit}' /etc/master.passwd \
    )\" WHERE bsdusr_username=\"root\""

To check your work, here’s a command to extract the hash from the SQLite database:

sqlite3 /data/freenas-v1.db "SELECT bsdusr_unixhash \
    FROM account_bsdusers WHERE bsdusr_username=\"root\""

At this point, if you are running in single-user mode, type…

exit

…to continue booting. If instead you used a sudo capable user to make the change, there is no need to reboot as the system accounting table in the FreeNAS SQLite database is in-sync with the FreeBSD system accounting files (/etc/master.passwd; /etc/spwd.db; etc.).

You should now be able to log into the FreeNAS WebUI with the updated password. Cheers!

I’ve been diving into FreeNAS development lately. In this blog post, I’ll talk a little bit about what I did to make development life on the FreeNAS appliance more livable from a FreeBSD developer’s perspective.

The first thing you notice after booting up FreeNAS is that it has a console menu. This was my first stop to getting the box online. A couple of keystrokes and we had an IP address. Load up the IP in your browser and you’re ready to start configuring FreeNAS. We set a root password (via the initial WebUI), and you’re off to the races.

Now comes time to customize FreeNAS for development purposes. Most people won’t want to (and shouldn’t) make these adjustments as they are strictly for the purpose of developing enhancements to FreeNAS (right on the box itself). These changes are not supported by the FreeNAS support team (I cannot emphasize this enough).

As a pre-requisite to getting any customizations in-place, we’ll need to be able to log into the UNIX command-line. In the WebUI, select “Services”, scroll down to “SSH” and hit the toggle to turn it “ON”. Then click the wrench logo just off to the right of the SSH toggle. To log in with the root password using SSH, we need to first enable the security feature which comes disabled by default. Check the configuration option labeled “Login as Root with password” (otherwise you can only login as root with a private key).

In my case, I want to develop enhancements before I configure my storage. So it is rather inconvenient that the FreeNAS WebUI only allows you to create users with home directories within configured storage. For your average consumer, this restriction is more than acceptable considering the boot media is often very small and has little free space. However, I need a place to store the patches that I will generate while working on the Volume Manager code and despite what it lacks in free space, the thumb drive or disk that FreeNAS boots from is ideal for this.

The trick to adding a user with non-standard home directory appears to be: first add your user from the WebUI, initially selecting “/nonexistent” as the home-directory (only valid choice when you have yet to configure a storage volume); and then afterward use the following command from the command-line of your FreeNAS appliance to change the home-directory:

sqlite3 /data/freenas-v1.db "UPDATE account_bsdusers \
    SET bsdusr_home=\"/data/home/USER\" WHERE bsdusr_username=\"USER\""

Note that every time you edit this user from the WebUI, you will need to change the home-directory to “/nonexistent” to force the WebUI to accept/save the changes. Afterward, repeat the above command to restore the desired home directory.

The next issue that I run into is perhaps less common. This issue is specific to my ISP. I require the ability to add the following line to the file /etc/dhclient.conf:

supersede domain-name "mydomain";

The reason for this is because I use my ISPs DHCP and the offer contains a domain of XYZ(dot)com, while I prefer ABC(dot)com. By adding the above line to /etc/dhclient.conf, the underlying FreeBSD system will “do the right thing” and supersede the domain provided by my ISP with my desired one. In the end, /etc/resolv.conf gets auto-populated with what I need while still using WAN-based DHCP (which itself is uncommon and seldom available).

The gotcha with simply modifying /etc/dhclient.conf is that you’ll lose your changes upon reboot. You really need to put the directive into:

/conf/base/etc/dhclient.conf

But before you can edit this file, you need to re-mount the root filesystem as read-write by executing:

mount -uw /

Then you can change the files in /conf and either reboot or execute the following to make the root filesystem read-only again:

mount -ur /

NOTE: This command may take a few moments as it needs to clear pending writes.

Next, I like to install a few packages like bash, vim-lite, pstree, iftop, and sudo (NB: root filesystem needs to be read-write before executing):

pkg_add -r bash
pkg_add -r vim-lite
pkg_add -r pstree
pkg_add -r iftop
pkg_add -r sudo

I’m sure that I install more than just the above, but since the FreeNAS appliance makes /var a memory filesystem, there’s no backup record (usually /var/db/pkg) of installed packages. Once you reboot, you’ll lose the record of what’s installed-versus-not.

ASIDE: It’s really convenient that FreeNAS allows you to specify that a group be allowed to execute sudo. Rather than checking the box for each individual user that you would like to allow sudo access, using a group means you can then store files as group-writable on the disk — simplifying team-based system administration.

After performing all these customizations, I think it’s time to make a backup. To make a backup of my FreeNAS thumb drive, I’ve had success in using version 0.9 of Win32DiskImager from Launchpad.net. Just stick the thumb drive into a Window system, launch Win32DiskImager, and use the “Read” option to make an image of your thumb drive. This image can then be stored for backup or imaged to another device.

I’m actually not a huge fan of running my OS from a thumb drive. So the first thing I do with my backup image is to write it to an internal Solid State Disk. I copied the backup image of my thumb drive to a FreeBSD server where the SSD was installed as an extra disk, and executed the following command to write the image to the drive:

dd if=backup.img of=/dev/ada3 bs=64k

NOTE: /dev/ada3 is the SSD installed as an extra disk.

After that has completed, I remove the SSD from my FreeBSD server and insert it into the 1U that used to boot FreeNAS from thumb drive.

All is dandy; the SSD boots up just like the thumb drive (except many times faster, shaving over 30 seconds from the boot process).

Next, I wish that the /data directory (mounted from /dev/ufs/FreeNASs4) was larger than the scant 20MB initially created by the installer. The SSD that I just cloned FreeNAS to is much larger than 2GB, leaving just over 80GB of unused free space. Being-one to never waste (and especially since I now have a user with a home directory of /data/home/user ready to utilize said space), I really want the /data partition to reach all the way to the end of the drive.

First, stick the drive into another FreeBSD or FreeNAS system (I used FreeBSD 9.2-STABLE @r260095M) where it is visible as an “extra” (unused) disk. Then execute the following steps to reclaim the unused free space (assigning it to the /data partition):

mount /dev/ufs/FreeNASs4 /mnt
mkdir /foo
rsync -avH /mnt/ /foo/
umount /mnt
gpart resize -i 4 ada3
newfs -b 32768 -f 4096 -L FreeNASs4 /dev/ada3s4
mount /dev/ufs/FreeNASs4 /mnt
rsync -avH --delete /foo/ /mnt/
umount /mnt
rm -Rf /foo

Now I remove the SSD, replace it into my FreeNAS chassis and boot it up. Voila, 82G of free space for /data.

FreeNAS is now ready for a collective team of multiple system administrators, able to treat the system less like an appliance and more like a sandbox for development. Cheers!