FreeNAS

Debugging FreeNAS 9.2.0 System

I recently found my FreeNAS box rebooting itself in the middle of the night (every night) and I wanted to know what was going on. I first tried enabling the debug kernel (see below picture) but that didn’t work nor did it get me any closer to understanding the issue at-hand.

"Enabling the FreeNAS Debug Kernel from the WebUI"
Enabling the FreeNAS Debug Kernel from the WebUI

The idea behind enabling the debug kernel was that it would drop me into the kernel debugger rather than rebooting 15 seconds after an unattended fatal error (leaving zero logs to analyze). But much to my surprise, the system rebooted anyway.

The next step was to set up remote syslog to allow FreeNAS to log to an external box. The first thing I did was connect my FreeNAS server directly to a FreeBSD system. The FreeNAS system will be configured to log everything to the FreeBSD system; so when FreeNAS reboots I will have a log of what lead up to that event.

I configured the secondary network interface on each server to talk on the same subnet. On the FreeNAS side, I went into the WebUI and configured the second interface of em1 to have IPv4 address 192.168.1.2 with a /24 subnet (255.255.255.0). When that was done, it looked like the below picture.

"Configuring a local second network interface in the FreeNAS WebUI"
Configuring a local second network interface in the FreeNAS WebUI

After that, I logged into my FreeBSD server and prepared it to receive remote syslog data.

ifconfig em1 inet 192.168.1.1/24
echo 192.168.1.2 freenas >> /etc/hosts
sysrc syslogd_enable=YES
sysrc syslogd_flags="-a freenas -v -v"
echo "+freenas" >> /etc/syslog.conf
echo "*.* /var/log/freenas.log" >> /etc/syslog.conf
touch /var/log/freenas.log
service syslogd restart

Now SSH into your FreeNAS box and execute the following:

mount -uw /
echo 192.168.1.1 freebsd >> /conf/base/etc/hosts
echo 192.168.1.1 freebsd >> /etc/hosts
mount -ur /

Then perform the last final bit, which is to go into the FreeNAS WebUI and enter a value of “freebsd” for the syslog server (see below picture).

"Where to configure a remote syslog server fro within the FreeNAS WebUI"
Where to configure a remote syslog server fro within the FreeNAS WebUI

At this point, you should be able to execute via SSH on the FreeNAS system:

logger foo

And get the following entry in /var/log/freenas.log on your FreeBSD system:

Jan 26 04:42:37 freenas root: foo

Cheers!

Of course, the jury is still out on exactly why my system is crashing, but I suspect a failing hard disk (the logs being streamed to the FreeBSD system seam to indicate via smartd messages a prefail condition for the boot disk).