Author Topic: Advice needed for complete local disaster recovery.  (Read 4866 times)

Trym

  • Zen Warrior
  • ***
  • Posts: 117
  • Karma: +1/-0
    • View Profile
Advice needed for complete local disaster recovery.
« on: December 27, 2010, 04:49:59 am »
Dear forum-members, I've for the last week tried to re-create a fully functional Zentyal Server from a remote backup set (remote as in FTP, local as in the FTP-server being locally accessible)... and failed. I've come close, but I can't seem to get the last few bits and pieces together, no doubt partly caused by my limited knowledge of Linux in general, and lack of specific knowledge of the Zentyal components in particular. Luckily, these backups are only practice sets from virtual servers, the real physical servers I maintain are happily humming along. I am just waiting for the day when lightning strikes. (Literally, that's the kind of disaster I'm preparing for.)

I've read both "The Perfect Ebox Setup Backup and Restore", and "Zentyal 2.0 documentation – backup" , none of which work particularly well. Actually, the last guide, which really ought to be up to date, will in the best case leave you with only a partially restored system, at worst with a non-bootable or non-reachable server (see uuid and mac below).

The most likely disaster to hit is a simple one: Hard-drive failure. The second most likely in my part of the world is lightning-strike. Both of the guides mentioned above assume you are restoring to your original system drive. If you still have a working system drive, you haven't really had much of a disaster.

Zentyal 2.0 by default excludes /var/cache and /var/spool from backups.  This is probably a good thing if you're only interested in backing up your data, but then a lot of other system-stuff should be excluded as well. Both of the guides mentioned above instructs you to delete *everything* on the partition you are restoring to before restoring. When you try to restore a Zentyal server missing those two, you end up with a completely borked package-management-system and non-working services. (HTTP-Proxy/Squid for example.)

Secondly, Zentyal/Ubuntu identifies hard-drives (or partitions, rather) by UUID. After restoring to another drive, you have to jump through a few hoops to get it to boot. (Hold down shift at first boot, press 'e' on the rescue-line, replace UUID=xxxx with /dev/sda1, boot, re-install grub when menu appears, drop to shell, edit /etc/fstab, replace UUID=xxx with /dev/sda1 for root and /dev/sda5 for swap.) It's awkward, but managable.

Zentyal also identifies NIC's by the MAC-address. If you are restoring to a box with one or more different nics from the original you need to edit /etc/udev/rules.d/70-persistent-net.rules and delete the lines tying eth0 and eth1 to your former cards, and if your new cards have already been detected you need to rename eth2 and 3 to eth0 and eth1.

The closest I've come to success is:

Remove the exclusion of /var/cache in the backup config

Excluding only /var/spool/squid/* instead of all of /var/spool (squid is not able to recreate that directory, but is happy to recreate the sub-dirs, it seems.)

On a blank hard-drive, run through the initial part of a fresh install until grub has been installed, then ctrl-alt-deling out of the installation. (This is just to create an identical set of partitions and to get grub2 installed for the very first boot, where you will have to hold down shift and re-install grub2. If you know what you're doing this can be done a lot faster, of course.)

Follow the restore instructions from the second document mentioned above (disaster recovery part – and skip the apt-get install duplicity part, duplicity is pre-installed on grml. It is very strange you cannot run duplicity from the rescue mode on the zentyal CD, and instead have to resort to another CD.)

Rebooting and doing all of the holding-down-shift-and-replacing-uuids-and-mac-ids mentioned above.

This leads to a, at first, seemingly healthy system, but that's only on the surface. You can connect via SSH and HTTPS, which is the most important thing. You can not login at the actual server. («Cannot execute login script.») You can not login to Zarafa. And, importantly, you can only login to the web-interface if it was originally set at port 443. If you have changed ports, to use 443 for secure web-sites or https for Zarafa, the web-configurator is unreachable.

Oh, and all of this only works (or semi-works as the case may be) on native ext3 or ext4 partitions. I haven't even begun contemplating doing this on an LVM (the default Zentyal install mode.)

I've tried every combination of new installations and partial/full restorations under the sun, and I cannot get any closer.

So, finally, here is my question: Does anybody have any clue how to proceed from here? Is it at all doable, or do I have to ditch the built-in backup and rely on another backup-solution?

I will not be happy until I am able to completely restore a backup of a physical server onto a virtual machine and having it run every service succesfully, with every bit of data intact. If this is not doable with Zentyal I'd appreciate to hear it, as I then will have to hunt down other solutions and start the data-migration as soon as possible.

And before you answer: The online disaster recovery is not an option. Way too much data to back up remotely, and any lightning-strike will take out the both the local modem and the ISP's telephony boxes. I need to get the server back up for internal functions loooong before repairs are finished.

If you do have an authoritative answer, I don't care how many hoops I have to jump through to get it done, if it is doable, I will do it, even if it means typing in two hundred scripts by hand and/or building my own customized distribution.

But... is it doable?

::Trym
« Last Edit: December 28, 2010, 05:35:35 am by Trym »

kalganum

  • Zen Apprentice
  • *
  • Posts: 17
  • Karma: +0/-0
    • View Profile
Re: Advice needed for complete local disaster recovery.
« Reply #1 on: January 27, 2012, 12:47:03 pm »
This Topic is of particular concern for me as I have been asked to perform the same type of task.
Before to read this topic I have tried disaster procedure recommended but failed to restore an identical system on a new machine (real or virtual).
Furthermore duplicity did not restore user rights and permissions even after I have restored passwd and group files.

Did someone found the perfect procedure to restore a system backup with duplicity ?

Thanks

Trym

  • Zen Warrior
  • ***
  • Posts: 117
  • Karma: +1/-0
    • View Profile
Re: Advice needed for complete local disaster recovery.
« Reply #2 on: January 28, 2012, 02:32:48 pm »
As you can see the above post is very old.

I did manage to create a complete restore-solution which worked well for a while, but due to constant changes in Zentyal I abandoned it. You may still find some useful pointers in it, especially the part about backup and restore of Zarafa.

It is here: http://how-to.solheimsvollen.net/HOW-TO_Zentyal_2_Backup_and_Disaster_Recovery.html