Author Topic: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.  (Read 12168 times)

Trym

  • Zen Warrior
  • ***
  • Posts: 117
  • Karma: +1/-0
    • View Profile

The HOW-TO is no longer available, as it requires constant modifications to keep up with changes in Zentyal. At present it does not work. If you need to look at it anyway, you can find it here. Zentyals official backup and disaster recovery guide is here.


  • 0.97.2 - Added message that the HOW-TO is out of commision and will not be updated anymore.
  • 0.97.1 - Commented out restoration of firewall and services-settings and added the reason for it in "known issues."
  • 0.97.0 - Added a "Troubleshooting" section.
  • 0.96.5 - Added a warning about 64-bit installer bugs in the restoration procedure.
  • 0.96.4 - Fixed some typos. (Some $-signs had disappeared from the scripts during a conversion.)
  • 0.96.3 - Fine-tuned some explanations.
  • 0.96.2 - Added some illustrations.
  • 0.96.1 - Changed the restoration procedure to guide the user to only install needed modules. This is to avoid a bug in the restore-module function, which would crash if there were certain modules installed on the server being restored to which didn't exist in the configuration backup. Added a much more detailed procedure for installing a server for restoration-purposes. Added some more error-checking to the scripts.
  • 0.96 - Added webmail-settings and non-zarafa mail data-backup.

::Trym
« Last Edit: May 05, 2011, 02:48:38 pm by Trym »

jgggr

  • Zen Apprentice
  • *
  • Posts: 18
  • Karma: +0/-0
    • View Profile
Oh man! That's awsome!

I've just lost my ebox 1.4 over a power failure (lost some hardware) so I've opted to make a fresh install of Zentyal 2.0. The setup module of my ebox was not set. I did backed up all my files and wrote down all the configs though. But, still, did I got in trouble!

Just asking: using this setup you propose will I be safe with my PDC? I mean, I manually recreated all the users  from my old ebox, used the same netbios name, ip, etc and still I got errors when logging in the windows hosts... Had to recreate the profiles so that that every got "trusted" again.
(I'm glad we are a small engineering office and only have 9 users.)

I will now configure the backup module and I think I might follow your suggestion. I will wait if there are more comments to enrich this conversation.


Trym

  • Zen Warrior
  • ***
  • Posts: 117
  • Karma: +1/-0
    • View Profile
Quote
Just asking: using this setup you propose will I be safe with my PDC? I mean, I manually recreated all the users  from my old ebox, used the same netbios name, ip, etc and still I got errors when logging in the windows hosts..

Short answer: Yes.

Long answer: I've tested it several times (anytime I've changed the procedure I've run a full test before publishing it) by installing a fresh Zentyal (2.0.3 install CD) server as a Primary Domain Controller with roaming profiles enabled with Zarafa. I've created from 2 to 10 users.

I've installed 2-7 Windows client pc's, and joined them to the domain.

I've logged in to each user in turn and created a few files in each user's "My Documents" folder, logged in to zarafa and created a few calendar entries, and sent one mail to another user before logging off to save the roaming profile.

I've then followed the "Part 2" instructions for the server exactly as above.

I've made a manual, full backup using the script in Part 2.

I've shut down the server, and followed "Part 3" to the letter on different virtual hardware, sometimes on real hardware.

After that I've verified the server has been restored correctly by booting each of the clients in turn, logging on as each user in turn, checking they're able to log on to the domain and to Zarafa, to load and save the roaming profile, to access public and private shares, and that all documents and mail are present.

Yes, small-scale testing suggests that this is a working way to restore your users, PDC, profiles and shares.

::Trym
« Last Edit: January 14, 2011, 05:53:25 pm by Trym »

satyris

  • Zen Monk
  • **
  • Posts: 94
  • Karma: +0/-0
    • View Profile
very interesting !
Will try it later. Thanks so far.

jsalamero

  • Zentyal Staff
  • Zen Hero
  • *****
  • Posts: 1419
  • Karma: +45/-1
    • View Profile
Nice HOW-TO! Congrats! :)

Trym

  • Zen Warrior
  • ***
  • Posts: 117
  • Karma: +1/-0
    • View Profile
Thank you. I've worked pretty hard on this, glad somebody noticed ;-)

::Trym

Trym

  • Zen Warrior
  • ***
  • Posts: 117
  • Karma: +1/-0
    • View Profile
This HOW-TO has as of today undergone a major rewrite. The whole process is now a lot more streamlined. There's a little more configuration (a few more scripts), but the restoration process is now practically effortless.

If you need this kind of backup/disaster recovery, I feel confident enough now to encourage you to test it in a virtual machine. Normal caveats apply. (See top of HOW-TO and Part 2.5.)

::Trym

Tiss

  • Zen Apprentice
  • *
  • Posts: 20
  • Karma: +0/-0
    • View Profile
Excellent work, Trym!

Bookmarking this thread for future reference, but I hope I'll never need it.  ;D

c4rdinal

  • Zen Samurai
  • ****
  • Posts: 341
  • Karma: +4/-0
    • View Profile
All hands down for an excellent job :)

Thanks

Josep

  • Zen Samurai
  • ****
  • Posts: 255
  • Karma: +6/-0
    • View Profile
I have tried this Howto and run into problems.
For once, I believe that it should be clear (since it wasn't for me, yet) that for this to work you need to reinstall a server from scratch and reconfigure the packages as they are installed, namely, you will have to input the right distinguished name for your LDAP server. This means that the LDAP database is created anew.
It is also important to note that by not including the /var/lib/ebox/CA, all your existing certificates will be invalid, as you effectively creating a new root certificate.

Because of this (and probably a huge number of mistake from my part, as I wanted it to work effortlessly) I ended up with a borked server, completely useless.

But ... there is a bright side to this story: it was a virtual machine specifically setup to test the Restore process. :-)

I will try again, fine-tuning a number of steps if possible.
Trym, could you be a little bit more specific on what folders you backup and which ones you don't, as well as what packages do you install and which ones you have to  configure?

Has anyone tested this process entirely? What's your experience?

Trym

  • Zen Warrior
  • ***
  • Posts: 117
  • Karma: +1/-0
    • View Profile
Quote
I have tried this Howto and run into problems.
For once, I believe that it should be clear (since it wasn't for me, yet) that for this to work you need to reinstall a server from scratch and reconfigure the packages as they are installed, namely, you will have to input the right distinguished name for your LDAP server. This means that the LDAP database is created anew.
It is also important to note that by not including the /var/lib/ebox/CA, all your existing certificates will be invalid, as you effectively creating a new root certificate.

Taken directly from the HOW-TO:
Quote
The strategy is simple; instead of restoring a complete server, we will install a new one, and fetch the bits and pieces we need from a backup.

and

Quote
Now, simply install it (Zentyal). If you want, you can choose a different file system, or even switch "bitness" (32- or 64-bit.) Even though we will later restore users and passwords, that will not replace the Zentyal administrative user, so choose a username and password with care. If possible, choose the same server-name, adresses and subnets as before. Go through the entire install process, including installing and updating (if internet is still up) every piece of software you used before. After upgrading reboot the server to activate any new kernel. If you have your screenshot of installed components, great, install only those. If you don't, just install everything, you can uninstall stuff later. You do not have to configure any modules except for the network (to reach your backup-server) and the Webconfigurator, change its port from 443 to something else in System/General. Make sure that all installed modules are enabled.

So yes, I specifically state in several places this is indeed a new installation. That is in fact the point of it all. I also give a link to the post describing my futile attempts to restore a server using the Zentyal backup documentation.

Quote
Because of this (and probably a huge number of mistake from my part, as I wanted it to work effortlessly) I ended up with a borked server, completely useless.

I'm sad to hear that, but thank you for telling me. It is very, very strange. In theory that cannot happen. The only reason I can think of is that old files from previous backup sets from the default backup configuration have been restored, overwriting the new server's settings and system-files (like /etc, /var.). You should not be able to mess up the core of Zentyal by restoring stuff with the backup-settings from the HOW-TO, it only restores data-files. The worst that can happen, and it shouldn't, is that you lock yourself out of remote access with firewall-rules, but you should *always* be able to log in at the server itself.

The instructions have to be followed precisely. I've explained that this is indeed a new installation, and if you didn't read that, you may have overlooked some other things as well. I'd appreciate if you'd try again.

As for the certificates, you are indeed right. The base certificate will be recreated. I'm in the process of running a restore of a real physical server using 20+ VPN certificates onto a virtual server, I'll report back with my findings.

Quote
Trym, could you be a little bit more specific on what folders you backup and which ones you don't, as well as what packages do you install and which ones you have to  configure?

These are the backup-settings for my server at home:

Include Path    /etc/ppp/ip-up.d/trymddns    
Include Path    /etc/apache2/sites-available    
Include Path    /etc/ebox/hooks/firewall.postservice    
Include Path    /etc/ebox/hooks/ebackup.postservice    
Include Path    /scripts    
Include Path    /backup    
Include Path    /home    
Include Path    /srv    
Include Path    /var/vmail    
Include Path    /var/www    
Exclude path    /

It's running these modules: Network, firewall, antivirus, dhcp, DNS, backup, events, logs, e-mail filter, monitor, NTP, trafficshaping, users and groups, certificate authority, webserver, voip, jabber, e-mail, filesharing, http-proxy, usercorner, and groupware. (with approx 2GIG backup-file from Zarafa.)

(I wanted to switch to ext4 from LVM, so I did, using the procedure in the guide. New installation, restore everything, back in business. I'm shocked you didn't get it to work.)

Quote
Has anyone tested this process entirely? What's your experience?

Well, I have ;-), a ton of times, and I've never been able to completely destroy the restored server. The 'worst' I've been able to do is to make https-websites unaccessible, and that's only if the network adresses are different. (Certificate related, for sure.) That is only to be expected, which is why I recommend a simulated network environment in part 2.5, so you can use exactly the same network setup.

As for which modules have to be configured manually... as long as you use the same network adresses... using the modules above: Only the two already mentioned (network, webconfigurator https-port).

The modules which are restored are listed in the /scripts/restoreall script. (I've added more and more as I've tested each new addition. I must have made a mistake if indeed restoring certificates does not work. Note, we're not restoring the certificate-files, we're telling Zentyal to re-create them.)

You are raising some valid points, and as stated I will go check them out immedately, but probably won't finish until tomorrow.

I'd appreciate if you'd try again, it really shouldn't be possible to ruin your configuration to the point that it doesn't work at all. Just pretend you know nothing, and follow it step by step. If it indeed does not work, I'll remove it entirely until I can find out why.

Bye for now, more tomorrow.

::Trym
« Last Edit: January 20, 2011, 10:04:20 pm by Trym »

Trym

  • Zen Warrior
  • ***
  • Posts: 117
  • Karma: +1/-0
    • View Profile
I wasn't able to wait for a big restore process to complete, and this seemed serious to me, so I ran another 'short' test (if 2.5 hrs can be considered short).

3 Virtual Zentyl Servers, one a VPN-server, another as a gateway, a 3rd as an ftp backup-server, and a 4th virtual machine, windows 7 as a VPN-client. After configuring VPN, and successfully connecting the VPN client through the gateway (I of course made sure that it couldn't connect through the regular network, made doublesure by connecting to the 192.168.160.1 default vpn-ip-address), I followed part 2 exactly as above for the VPN server. Pure copy and paste, in addition to configuring the backup module.

I followed part 3 to the letter on a "new" virtual machine. (Virtual machines are by necessity very similar of course, but at least it has different MACs and UUIDs.)

To make a long story short, the certificate authority, the certificates and VPN-settings get properly restored, as when tested before.

The windows 7 VPN-client could connect to and use the VPN-server using exactly the same certificate as before (No changes were made to the VPN-client-machine whatsoever.)

What happens is this:

When you restore the Users module, which you have to do before the files, to make sure files get proper permissions, the LDAP-base also gets recreated to it's original state, recreating the identity of your original server, if you will. Just look at LDAP in the webconfig before and after restoring the Users to see what I mean. (But don't press the red "save changes" if you do, just continue from the command line.)

Later, when we restore the CA, all the certificates get recreated.

There is no need to backup anything else from the original server to get the original system state back, do not include things from the previous system like /var/lib/ebox/CA, that will probably break your system. Only include files you have put there yourself, like cronjobs and hook-scripts.


::Trym
« Last Edit: January 21, 2011, 01:24:42 am by Trym »

La Luz

  • Zen Apprentice
  • *
  • Posts: 13
  • Karma: +0/-0
  • Come on!
    • View Profile
VERY NICE HOW-TO!

Thank you very much for the effort.
I would be implementing that shortly.

See you

Josep

  • Zen Samurai
  • ****
  • Posts: 255
  • Karma: +6/-0
    • View Profile
I didn't mean to criticize you. Like I said previously it's a wonderful job and you shared your effort with the rest of us. All I'm saying is that I managed to bork my system. After all, this is part of practicing ;-). I assume that I learned my lesson and will try again on another new clean installation, and be more careful when following instructions.

A remaining concern is why we shouldn't back up /var/lib/ebox/CA, but I guess I can easily overwrite its contents from the original machine just to test things, in case a VPN connection fails.

Do you run the installation of components from the Web interface?
I do use the apt-get command directly as this offers me more control and provides with immediate feedback, but then I may be over-complicating it.

Trym

  • Zen Warrior
  • ***
  • Posts: 117
  • Karma: +1/-0
    • View Profile
I'm writing this from work, I'll have to be quick.

I didn't take it as criticism, I really want feedback, I rely on others testing the procedure on multiple configurations before I can remove the DRAFT-status.

As for /var/lib/ebox/CA, just follow my instructions, do not include it in the backup, and after you're done, check it. I think you'll find that they're all there. Then again, if I'm wrong, please tell me. The whole point of this method is to not restore any systemfiles, but to rebuild them using the configuration backup (the same method Zentyal uses when restoring from a cloud backup.)

I don't take other's data lightly, I wouldn't publish anything without thorougly testing it first. I will later change the HOW-TO to include som warnings about what not to do, but I'm running into a 20 000 character limit on a post. I've had to remove a lot of text, so I've removed most of tech. explanations and what not to do.

Quote
Do you run the installation of components from the Web interface?

Yes. Install everything you need of software, but don't bother to configure ind. modules. (New procedure in the how-to to only install the needed components.) Just answer the questions from the config-wizard. Use any e-mail domain and server names you like, they will be overwritten by the restore process anyway. The only important setting is the network address, and that all modules have been enabled at least once. If not, settings for that module will not be restored.

I really value feedback, but you have to do what the how-to actually says if the feedback is to be relevant.

I'm really busy today (watching over kids at a LAN-party after work), I'll get back to this tomorrow.

::Trym
« Last Edit: January 26, 2011, 10:44:00 pm by Trym »