Compiling bacula natively on the ReadyNAS

It works! Here's the Reader's Digest version:

apt-get install libc6-dev 
apt-get install libc6-dev 
apt-get install gcc 
apt-get install make autoconf automake libtool flex bison gdb
apt-get install libtag1-dev 
apt-get install uuid-dev
apt-get install g++

wget http://wwwmaster.postgresql.org/redir/198/h/source/v8.4.7/postgresql-8.4.7.tar.gz
tar zxf postgresql-8.4.7.tar.gz 
cd postgresql-8.4.7
./configure --without-readline --without-zlib
make install
/usr/local/pgsql/bin/psql --version
wget http://sourceforge.net/projects/bacula/files/bacula/5.0.1/bacula-5.0.1.tar.gz/download
tar zxf bacula-5.0.1.tar.gz 
cd bacula-5.0.1
./configure --prefix=/usr/local/bacula --with-postgresql=/usr/local/pgsql/ --build=sparc-linux && make install
/usr/local/bacula/sbin/bacula-sd --help  
tar zcf bacula-5.0.1-sparc.tgz /usr/local/bacula/
tar zcf postgresql-8.4.7-sparc.tgz /usr/local/pgsql/

Then I copied both of these tarballs of binaries over to a production ReadyNAS. It already had the bacula add-on installed, so I left that in place (so I could re-use the init script and config) and disabled it from running via Frontview. I changed to / and unpacked the tarballs, then tried manually running bacula-sd with the existing config. It worked!!!!

The big test was to then try a restore, since that was the bit I was trying to get working. I decided to try restoring a subset of a previous backup to a remote server. It worked. Flawlessly. I then proceeded outside and did a victory lap of my yard.

As a clean-up: I copied my original bacula-sd.conf to /etc/bacula, and modified the add-on-installed init script to run /usr/local/bacula/sbin/baculs-sd instead of /opt/rfw/sbin/bacula. I added exit 0 near the top of the installed bacula-dir and bacula-fd init scripts, just in case.

I also used update-rc.d to get the bacula-sd init script to start at boot, though... it didn't. Still puzzling over that one, but eh, I'll get there.

One minor nit: when you use the init script to stop bacula, it reports an error:

rm-nas-1:~# /etc/init.d/bacula-sd stop
Stopping Bacula Storage Daemon: start-stop-daemon: warning: failed to kill 7948: No such process
start-stop-daemon: warning: failed to kill 7946: No such process
bacula-sd

I think this is because killing one of the three running bacula-sd processes (old-school: uses multiple processes rather than threads, it would seem) causes the others to exit, and they're gone before the kill process can zap their process IDs. Not a problem, just caused me a few minutes' puzzlement.

Cross-compiling: I admit defeat

After days - and I mean days - of attempting to cross-compile sparc binaries for the ReadyNAS NV+ on an Ubuntu 10.04 build host, I've achieved little, other than remembering how much I loathe the old

./configure --with-billions-of-options && make

dance. For the record, here's what I tried.

Before I could compile bacula, I had to compile a database, as the configure script for bacula requires you to link against at least one of the available DBs (I suspect this is only needed when you are compiling the director, but there seems to be no make target to compile only the storage daemon). I initially tried to install libpq, but of course, that installed the intel version suited to my build host - doh! So I had to cross-compile postgresql, which is okay with me, as I have some familiarity with compiling it. I downloaded the source for 8.4.7 and made that:

./configure --with-postgresql=/usr/local/sparc-builds/postgresql --host=sparc-linux-gnu
make install
root@sparc-cc-10-04:~# file /usr/local/sparc-builds/postgresql/lib/libpq.so.5.2
/usr/local/sparc-builds/postgresql/lib/libpq.so.5.2: ELF 32-bit MSB shared object, SPARC32PLUS, V8+ Required, version 1 (SYSV), dynamically linked, not stripped

Hmmmm... file tell me this is a sparc binary, but it looks like it's built for a much beefier CPU than the ReadyNAS has. So I grabbed /bin/ls from a ReadyNAS and compared what kind of binary that is:

root@sparc-cc-10-04:/tmp# file ls
ls: ELF 32-bit MSB executable, SPARC, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.2.0, stripped

Hmmm... I ignored this for a bit, and decided to continue with bacula compilation: so long as I had a running bacula-sd, who cares what file says?

./configure --with-postgresql=/usr/local/sparc-builds/postgresql --host=sparc-linux-gnu --prefix=/usr/local/sparc-builds/bacula ; make install
file /usr/local/sparc-builds/bacula/sbin/bacula-sd 
/usr/local/sparc-builds/bacula/sbin/bacula-sd: ELF 32-bit MSB executable, SPARC32PLUS, V8+ Required, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.8, not stripped

Undeterred, I copied this to the ReadyNAS, just to try executing it.... no go:

rg-nas:~# /tmp/bacula-sd 
-bash: /tmp/bacula-sd: cannot execute binary file

After much fruitless thrashing about, including setting CFLAGS='mcpu=v7' and assorted incantations of -Av7, and a lot of manpage reading, I was still no closer to getting a binary that wasn't targeted at the SPARC32PLUS. I know the ReadyNAS unit is a cut-down sparc CPU - a neon, or a leon, or whatever. I figured V7 code would be a safe bet to run on it, and despite the man page for gcc stating "By default (unless configured otherwise), GCC generates code for the V7 variant of the SPARC architecture", and despite setting that CPU type explicitly, nothing would coax a V7 binary out of the process. I started to suspect an issue with libtool or maybe the make scripts... after a while, it dawned on me I should try something simpler:

root@sparc-cc-10-04:~/tmp# cat hello.c 
/* hello.c : prints "hello world" */
#include 
int main(void) {
    printf("hello world\n");
}

root@sparc-cc-10-04:~/tmp# sparc-linux-gnu-gcc -mcpu=v7 hello.c -o hello
root@sparc-cc-10-04:~/tmp# file hello
hello: ELF 32-bit MSB executable, SPARC32PLUS, V8+ Required, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.8, not stripped

And no, it didn't run when I copied it over to the ReadyNAS. Actually, nothing I tried seemed to have any influence on the compiler's binary output:

root@sparc-cc-10-04:~/tmp# file hello
hello: ELF 32-bit MSB executable, SPARC32PLUS, V8+ Required, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.8, not stripped
root@sparc-cc-10-04:~/tmp# export CFLAGS='-mcpu=v7'
root@sparc-cc-10-04:~/tmp# sparc-linux-gnu-gcc  hello.c -o hello
root@sparc-cc-10-04:~/tmp# file hello
hello: ELF 32-bit MSB executable, SPARC32PLUS, V8+ Required, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.8, not stripped

There's probably some trick I don't get to all this, and I'd love to know what it is. However, by the time I got to this point, too many days had wasted away, and I needed to get the backup project back on track. Time for option 2: use the sparc ReadyNAS as the build host to produce its own binaries. I haven't written that up yet, but: it worked :-)

Problems cross-compiling bacula-5.0.1 for ReadyNAS Sparc

It looks as though the ready-made sparc cross-compiler package will not compile Bacula 5.0.1 - stat.h seems to have changed enough that the build fails like this:

==>Entering directory /home/pyarra/bacula-5.0.1/src/findlib
make[1]: Entering directory `/home/pyarra/bacula-5.0.1/src/findlib'
Compiling attribs.c
attribs.c: In function `void encode_stat(char*, stat*, int, int)':
attribs.c:211: error: 'struct stat' has no member named 'st_flags'
attribs.c: In function `int decode_stat(char*, stat*, int32_t*)':
attribs.c:308: error: 'struct stat' has no member named 'st_flags'
attribs.c:310: error: 'struct stat' has no member named 'st_flags'
attribs.c: In function `bool set_attributes(JCR*, ATTR*, BFILE*)':
attribs.c:482: error: 'struct stat' has no member named 'st_flags'
attribs.c:482: error: `chflags' undeclared (first use this function)
attribs.c:482: error: (Each undeclared identifier is reported only once for 
   each function it appears in.)
make[1]: *** [attribs.lo] Error 1
make[1]: Leaving directory `/home/pyarra/bacula-5.0.1/src/findlib'


  ====== Error in /home/pyarra/bacula-5.0.1/src/findlib ====== 

Doh!

My latest attempt to cross-compile for sparc has meant setting up an Ubuntu 10,04 box, and I'm currently following this guide to get the cross-compile tools set up. It's not an area I know much about - although I "get" the theory behind cross-compiling, in the past, I've always just kept a build host for each architecture I'm building for (ultrasparc, alpha) and battled with drain-bamaged build environmnts as best I could. Hopefully, this initial pain of setting up a cross-compile environment will be worth it.

So far so good, then install libpq-dev (so I can --with-postgresql - you need to specify one DB to support when you compile, as there's no way to compile only the storage daemon).

Then ./configure --with-postgresql --host=sparc-linux-gnu and of course it bombs out here:

checking whether setpgrp takes no argument... configure: error: cannot check setpgrp when cross compiling

Agh, well how about not checking for it then?? I guess one day I'll have to learn how to use autoconf so I can fix this properly, but for now, edit configure to skip this check.

Bacula: mis-matched director and storage daemons, eeep!

The backup solution is coming along nicely, but... ehhh... it turns out there are really good reasons to not mix your Director and Storage Daemon versions:

11-Mar 14:45 rm-bac-1-dir JobId 5: Error: Bacula rm-bac-1-dir 5.0.1 (24Feb10): 11-Mar-2011 14:45:14
  Build OS:               i486-pc-linux-gnu ubuntu 10.04
  Job:                    RestoreFiles.2011-03-11_14.45.12_10
  Restore Client:         wd-server
  Termination:            *** Restore Error ***

11-Mar 14:48 rm-bac-1-dir JobId 6: Start Restore Job RestoreFiles.2011-03-11_14.48.53_13
11-Mar 14:48 rm-bac-1-dir JobId 6: Using Device "rm-nas-1-wd-server"
11-Mar 14:48 rm-nas-1 JobId 6: Fatal error: Bootstrap file error: Keyword Storage not found
            : Line 1, col 7 of file /opt/rfw/var/bacula/working/rm-nas-1.RestoreFiles.2011-03-11_14.48.53_13.3.bootstrap
Storage="rm-nas-1-wd-server"

11-Mar 14:48 rm-nas-1 JobId 6: Fatal error: Error parsing bootstrap file.
11-Mar 14:48 rm-bac-1-dir JobId 6: Fatal error: Bad response to Bootstrap command: wanted 3000 OK bootstrap
, got 3904 Error bootstrap

The ReadyNAS is running version 3.0.3 Storage Daemon. While it will quite happily accept the backup files sent to it, when you try to restore it bombs out - clearly the bootstrap file format has changed sufficiently that this will not work. Darn.

This leaves me with three options:

1. build my own Bacula packages for the NV+ units

2. build my own bacula storage daemon binary (and supporting libs), and
copy this over the older installed bacula on the NV+ units, or remove
packaged versions and make; make install

3. don't use independent bacula storage daemons on each ReadyNAS NV+ and
mount them via NFS

My preferred option is 1 and I am setting up a sparc cross-compile
environment and the ReadyNAS Add-on SDK - while this course has the
highest initial cost (time-wise) it should yield better results, and be
potentially useful later if we decide to package additional software for
the ReadyNAS units.

Option 3 would preclude us using local storage to keep
backup traffic local where that is desired, and would only be my
last-resort option.

You can download a pre-made cross-compile package for Linux here, which I did, and set up a VM to be the cross-compile host. Ubuntu 10.04 failed during configure:

pyarra@ubuntu-sparc-compile:~/bacula-5.0.1$ ./configure --prefix=/home/pyarra/ready-builds/ --host=sparc
-linux

[snip oodles of configure messages]
checking whether setpgrp takes no argument... configure: error: cannot check setpgrp when cross compiling

So I've decided to build an Ubuntu 8.04 host to try building with. More on that later!

Fixed: SNMP on Ubuntu listening only on localhost

By default, snmpd only listens on 127.0.0.1. Edit /etc/default/snmpd and change this line:

SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -g snmp -I -smux -p /var/run/snmpd.pid 127.0.0.1'

to this:

SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -g snmp -I -smux -p /var/run/snmpd.pid'

That is, remove the specification to run only on 127.0.0.1 - then restart snmpd.

I also removed any RW access that had been configured in /etc/snmp/snmpd.conf. I only want read access via SNMP.

Ubuntu Server on HP Proliant - managing RAID

As part of the disk-based backup project, my Bacula Director is going on an Ubunutu 10.04 LTS server on HP Proliant hardware. All well and good, but this is my first Ubuntu Server deployment on HP hardware. Under Dell, I'm used to using the MegaRAID CLI tools to monitor and manage the RAID controller, but was stuck trying to find the equivalent for Ubuntu.

The answer is to use HP's ProLiant Support Pack which offers a bunch of CLI tools. The one I was interested in here was hpacucli

To install (as root, obviously):

1. wget http://downloads.linux.hp.com/SDR/downloads/bootstrap.sh
2. ./bootstrap.sh ProLiantSupportPack (this adds the HP repo to apt's sources)
3. wget http://downloads.linux.hp.com/SDR/downloads/ProLiantSupportPack/GPG-KEY-ProLiantSupportPack
4. apt-key add GPG-KEY-ProLiantSupportPack
5. aptitude update
6. apt-get install hpacucli

Like any good CLI program, it is finicky to learn, and intuitive thereafter :-) Here's the thing I wanted to do:

root@rm-bac-1:~# hpacucli 
HP Array Configuration Utility CLI 8.50-6.0
Detecting Controllers...Done.
Type "help" for a list of supported commands.
Type "exit" to close the console.

==> ctrl slot=0 logicaldrive all show detail

Smart Array P410i in Slot 0 (Embedded)

   array A

      Logical Drive: 1
         Size: 136.7 GB
         Fault Tolerance: RAID 1
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 35132
         Stripe Size: 128 KB
         Status: OK
         Unique Identifier: 600508B1001031303434393834300300
         Disk Name: /dev/cciss/c0d0
         Mount Points: / 131.1 GB
         OS Status: LOCKED
         Logical Drive Label: A01197DF5001438010449840590B
         Mirror Group 0:
            physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 146 GB, OK)
         Mirror Group 1:
            physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 146 GB, OK)

Cool! There are other bits I'll want - I see one of the packages offers SNMP monitoring, which naturally, I'm going to use so Nagios can keep an eye on this server.

A list of the 7 packages in the Proliant Support Pack is in this file, but here's an abridged listing:

Package: hpacucli
Description: HP Command Line Array Configuration Utility
 The HP Command Line Array Configuration Utility is the disk
 array configuration program for Array Controllers.

Package: cpqacuxe
Description: HP Array Configuration Utility
 The HP Array Configuration Utility is the web-based disk array
 configuration program for Array Controllers.

Package: hpsmh
Description: HP System Management Homepage
 Provides HTTP infrastructure for HP Agent & Utility system packages.
Homepage: http://hp.com/go/proliantlinux

Package: hp-smh-templates
Description: HP System Management Homepage Templates
 This package contains the System Management Homepage Templates for all
 hp Proliant systems with ASM, ILO, & ILO2 embedded management asics.

Package: hp-snmp-agents
Description: Insight Management SNMP Agents for HP ProLiant Systems
 This package contains the SNMP server, storage, and nic agents for all
 hp Proliant systems with ASM, ILO, & ILO2 embedded management asics.

Package: hponcfg
Description: RILOE II/iLo online configuration utility
 Hponcfg is a command line utility that can be used to configure iLO/RILOE II
 from within the operating system without requiring a reboot of the server.
Homepage: http://hp.com/go/proliantlinux

Package: hp-health
Description: hp System Health Application and Command line Utility Package
 This package contains the System Health Monitor for all hp Proliant systems
 with ASM, ILO, & ILO2 embedded management asics.  Also contained are the
 command line utilities.

A disk-based backup solution: Bacula, ReadyNAS NV+

I decided to set up a new backup system. I don't like tapes, so I started looking at disk-based backups. I also settled on Bacula for handling the backups. I found I could buy a ReadyNAS NV+ for $380, then 2TB disks for $125... allowing for a cold spare, that means $625 for disks... total cost for one storage unit, about $1000 for 5.4TB of RAID-backed, hot-swappable storage. That's outstanding value!

Better yet, I discovered that some helpful people had compiled and packaged Bacula for the Sparc ReadyNAS units (of which the NV+ is one) so all I need do is install it. Then I can run the Director on my basic HP server, the Storage Daemon on the ReadyNAS, and the File Daemon on the clients to be backed up.

I also figured we'd get better redundancy by using a pair of ReadyNAS units, and writing alternate weeks' backups to each.. so weeks 1 3 and 5 go to nas-1, and weeks 2 and 4 go to nas-2.

After trying 3 different web UIs for controlling the backups, I settled on bweb, which seems to work pretty well (only a few broken bits).

I'm still part-way through the final implementation, but when I get the config all sorted out, I'll post a follow-up. So far, I have one of the ReadyNAS units running the Storage Daemon, and I've got my desktop computer backing up to it as I type. W00T!


At this stage, we're doing simple backups of user's network drives on a range of servers situated at 6 sites, to a central facility. If this scheme works out well, we may look at adding more storage, and expanding into doing bare-metal restore support for critical machines... could be exciting stuff ahead!