Zentyal Forum, Linux Small Business Server

Zentyal Server => Installation and Upgrades => Topic started by: Trym on January 02, 2011, 06:42:46 pm

Title: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Trym on January 02, 2011, 06:42:46 pm

The HOW-TO is no longer available, as it requires constant modifications to keep up with changes in Zentyal. At present it does not work. If you need to look at it anyway, you can find it here. (http://how-to.solheimsvollen.net/HOW-TO_Zentyal_2_Backup_and_Disaster_Recovery.html) Zentyals official backup and disaster recovery guide is here. (http://doc.zentyal.org/en/backup.html#configuration-backup-in-a-zentyal-server)



::Trym
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: jgggr on January 04, 2011, 03:06:28 pm
Oh man! That's awsome!

I've just lost my ebox 1.4 over a power failure (lost some hardware) so I've opted to make a fresh install of Zentyal 2.0. The setup module of my ebox was not set. I did backed up all my files and wrote down all the configs though. But, still, did I got in trouble!

Just asking: using this setup you propose will I be safe with my PDC? I mean, I manually recreated all the users  from my old ebox, used the same netbios name, ip, etc and still I got errors when logging in the windows hosts... Had to recreate the profiles so that that every got "trusted" again.
(I'm glad we are a small engineering office and only have 9 users.)

I will now configure the backup module and I think I might follow your suggestion. I will wait if there are more comments to enrich this conversation.

Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Trym on January 04, 2011, 04:05:08 pm
Quote
Just asking: using this setup you propose will I be safe with my PDC? I mean, I manually recreated all the users  from my old ebox, used the same netbios name, ip, etc and still I got errors when logging in the windows hosts..

Short answer: Yes.

Long answer: I've tested it several times (anytime I've changed the procedure I've run a full test before publishing it) by installing a fresh Zentyal (2.0.3 install CD) server as a Primary Domain Controller with roaming profiles enabled with Zarafa. I've created from 2 to 10 users.

I've installed 2-7 Windows client pc's, and joined them to the domain.

I've logged in to each user in turn and created a few files in each user's "My Documents" folder, logged in to zarafa and created a few calendar entries, and sent one mail to another user before logging off to save the roaming profile.

I've then followed the "Part 2" instructions for the server exactly as above.

I've made a manual, full backup using the script in Part 2.

I've shut down the server, and followed "Part 3" to the letter on different virtual hardware, sometimes on real hardware.

After that I've verified the server has been restored correctly by booting each of the clients in turn, logging on as each user in turn, checking they're able to log on to the domain and to Zarafa, to load and save the roaming profile, to access public and private shares, and that all documents and mail are present.

Yes, small-scale testing suggests that this is a working way to restore your users, PDC, profiles and shares.

::Trym
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: satyris on January 05, 2011, 08:20:25 am
very interesting !
Will try it later. Thanks so far.
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: jsalamero on January 08, 2011, 09:48:49 am
Nice HOW-TO! Congrats! :)
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Trym on January 08, 2011, 07:08:54 pm
Thank you. I've worked pretty hard on this, glad somebody noticed ;-)

::Trym
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Trym on January 12, 2011, 10:05:47 pm
This HOW-TO has as of today undergone a major rewrite. The whole process is now a lot more streamlined. There's a little more configuration (a few more scripts), but the restoration process is now practically effortless.

If you need this kind of backup/disaster recovery, I feel confident enough now to encourage you to test it in a virtual machine. Normal caveats apply. (See top of HOW-TO and Part 2.5.)

::Trym
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Tiss on January 19, 2011, 09:53:31 am
Excellent work, Trym!

Bookmarking this thread for future reference, but I hope I'll never need it.  ;D
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: c4rdinal on January 19, 2011, 10:52:46 am
All hands down for an excellent job :)

Thanks
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Josep on January 20, 2011, 03:54:21 pm
I have tried this Howto and run into problems.
For once, I believe that it should be clear (since it wasn't for me, yet) that for this to work you need to reinstall a server from scratch and reconfigure the packages as they are installed, namely, you will have to input the right distinguished name for your LDAP server. This means that the LDAP database is created anew.
It is also important to note that by not including the /var/lib/ebox/CA, all your existing certificates will be invalid, as you effectively creating a new root certificate.

Because of this (and probably a huge number of mistake from my part, as I wanted it to work effortlessly) I ended up with a borked server, completely useless.

But ... there is a bright side to this story: it was a virtual machine specifically setup to test the Restore process. :-)

I will try again, fine-tuning a number of steps if possible.
Trym, could you be a little bit more specific on what folders you backup and which ones you don't, as well as what packages do you install and which ones you have to  configure?

Has anyone tested this process entirely? What's your experience?
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Trym on January 20, 2011, 09:16:13 pm
Quote
I have tried this Howto and run into problems.
For once, I believe that it should be clear (since it wasn't for me, yet) that for this to work you need to reinstall a server from scratch and reconfigure the packages as they are installed, namely, you will have to input the right distinguished name for your LDAP server. This means that the LDAP database is created anew.
It is also important to note that by not including the /var/lib/ebox/CA, all your existing certificates will be invalid, as you effectively creating a new root certificate.

Taken directly from the HOW-TO:
Quote
The strategy is simple; instead of restoring a complete server, we will install a new one, and fetch the bits and pieces we need from a backup.

and

Quote
Now, simply install it (Zentyal). If you want, you can choose a different file system, or even switch "bitness" (32- or 64-bit.) Even though we will later restore users and passwords, that will not replace the Zentyal administrative user, so choose a username and password with care. If possible, choose the same server-name, adresses and subnets as before. Go through the entire install process, including installing and updating (if internet is still up) every piece of software you used before. After upgrading reboot the server to activate any new kernel. If you have your screenshot of installed components, great, install only those. If you don't, just install everything, you can uninstall stuff later. You do not have to configure any modules except for the network (to reach your backup-server) and the Webconfigurator, change its port from 443 to something else in System/General. Make sure that all installed modules are enabled.

So yes, I specifically state in several places this is indeed a new installation. That is in fact the point of it all. I also give a link to the post describing my futile attempts to restore a server using the Zentyal backup documentation.

Quote
Because of this (and probably a huge number of mistake from my part, as I wanted it to work effortlessly) I ended up with a borked server, completely useless.

I'm sad to hear that, but thank you for telling me. It is very, very strange. In theory that cannot happen. The only reason I can think of is that old files from previous backup sets from the default backup configuration have been restored, overwriting the new server's settings and system-files (like /etc, /var.). You should not be able to mess up the core of Zentyal by restoring stuff with the backup-settings from the HOW-TO, it only restores data-files. The worst that can happen, and it shouldn't, is that you lock yourself out of remote access with firewall-rules, but you should *always* be able to log in at the server itself.

The instructions have to be followed precisely. I've explained that this is indeed a new installation, and if you didn't read that, you may have overlooked some other things as well. I'd appreciate if you'd try again.

As for the certificates, you are indeed right. The base certificate will be recreated. I'm in the process of running a restore of a real physical server using 20+ VPN certificates onto a virtual server, I'll report back with my findings.

Quote
Trym, could you be a little bit more specific on what folders you backup and which ones you don't, as well as what packages do you install and which ones you have to  configure?

These are the backup-settings for my server at home:

Include Path    /etc/ppp/ip-up.d/trymddns    
Include Path    /etc/apache2/sites-available    
Include Path    /etc/ebox/hooks/firewall.postservice    
Include Path    /etc/ebox/hooks/ebackup.postservice    
Include Path    /scripts    
Include Path    /backup    
Include Path    /home    
Include Path    /srv    
Include Path    /var/vmail    
Include Path    /var/www    
Exclude path    /

It's running these modules: Network, firewall, antivirus, dhcp, DNS, backup, events, logs, e-mail filter, monitor, NTP, trafficshaping, users and groups, certificate authority, webserver, voip, jabber, e-mail, filesharing, http-proxy, usercorner, and groupware. (with approx 2GIG backup-file from Zarafa.)

(I wanted to switch to ext4 from LVM, so I did, using the procedure in the guide. New installation, restore everything, back in business. I'm shocked you didn't get it to work.)

Quote
Has anyone tested this process entirely? What's your experience?

Well, I have ;-), a ton of times, and I've never been able to completely destroy the restored server. The 'worst' I've been able to do is to make https-websites unaccessible, and that's only if the network adresses are different. (Certificate related, for sure.) That is only to be expected, which is why I recommend a simulated network environment in part 2.5, so you can use exactly the same network setup.

As for which modules have to be configured manually... as long as you use the same network adresses... using the modules above: Only the two already mentioned (network, webconfigurator https-port).

The modules which are restored are listed in the /scripts/restoreall script. (I've added more and more as I've tested each new addition. I must have made a mistake if indeed restoring certificates does not work. Note, we're not restoring the certificate-files, we're telling Zentyal to re-create them.)

You are raising some valid points, and as stated I will go check them out immedately, but probably won't finish until tomorrow.

I'd appreciate if you'd try again, it really shouldn't be possible to ruin your configuration to the point that it doesn't work at all. Just pretend you know nothing, and follow it step by step. If it indeed does not work, I'll remove it entirely until I can find out why.

Bye for now, more tomorrow.

::Trym
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Trym on January 21, 2011, 12:46:27 am
I wasn't able to wait for a big restore process to complete, and this seemed serious to me, so I ran another 'short' test (if 2.5 hrs can be considered short).

3 Virtual Zentyl Servers, one a VPN-server, another as a gateway, a 3rd as an ftp backup-server, and a 4th virtual machine, windows 7 as a VPN-client. After configuring VPN, and successfully connecting the VPN client through the gateway (I of course made sure that it couldn't connect through the regular network, made doublesure by connecting to the 192.168.160.1 default vpn-ip-address), I followed part 2 exactly as above for the VPN server. Pure copy and paste, in addition to configuring the backup module.

I followed part 3 to the letter on a "new" virtual machine. (Virtual machines are by necessity very similar of course, but at least it has different MACs and UUIDs.)

To make a long story short, the certificate authority, the certificates and VPN-settings get properly restored, as when tested before.

The windows 7 VPN-client could connect to and use the VPN-server using exactly the same certificate as before (No changes were made to the VPN-client-machine whatsoever.)

What happens is this:

When you restore the Users module, which you have to do before the files, to make sure files get proper permissions, the LDAP-base also gets recreated to it's original state, recreating the identity of your original server, if you will. Just look at LDAP in the webconfig before and after restoring the Users to see what I mean. (But don't press the red "save changes" if you do, just continue from the command line.)

Later, when we restore the CA, all the certificates get recreated.

There is no need to backup anything else from the original server to get the original system state back, do not include things from the previous system like /var/lib/ebox/CA, that will probably break your system. Only include files you have put there yourself, like cronjobs and hook-scripts.


::Trym
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: La Luz on January 21, 2011, 01:43:11 am
VERY NICE HOW-TO!

Thank you very much for the effort.
I would be implementing that shortly.

See you
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Josep on January 21, 2011, 09:56:25 am
I didn't mean to criticize you. Like I said previously it's a wonderful job and you shared your effort with the rest of us. All I'm saying is that I managed to bork my system. After all, this is part of practicing ;-). I assume that I learned my lesson and will try again on another new clean installation, and be more careful when following instructions.

A remaining concern is why we shouldn't back up /var/lib/ebox/CA, but I guess I can easily overwrite its contents from the original machine just to test things, in case a VPN connection fails.

Do you run the installation of components from the Web interface?
I do use the apt-get command directly as this offers me more control and provides with immediate feedback, but then I may be over-complicating it.
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Trym on January 21, 2011, 10:41:29 am
I'm writing this from work, I'll have to be quick.

I didn't take it as criticism, I really want feedback, I rely on others testing the procedure on multiple configurations before I can remove the DRAFT-status.

As for /var/lib/ebox/CA, just follow my instructions, do not include it in the backup, and after you're done, check it. I think you'll find that they're all there. Then again, if I'm wrong, please tell me. The whole point of this method is to not restore any systemfiles, but to rebuild them using the configuration backup (the same method Zentyal uses when restoring from a cloud backup.)

I don't take other's data lightly, I wouldn't publish anything without thorougly testing it first. I will later change the HOW-TO to include som warnings about what not to do, but I'm running into a 20 000 character limit on a post. I've had to remove a lot of text, so I've removed most of tech. explanations and what not to do.

Quote
Do you run the installation of components from the Web interface?

Yes. Install everything you need of software, but don't bother to configure ind. modules. (New procedure in the how-to to only install the needed components.) Just answer the questions from the config-wizard. Use any e-mail domain and server names you like, they will be overwritten by the restore process anyway. The only important setting is the network address, and that all modules have been enabled at least once. If not, settings for that module will not be restored.

I really value feedback, but you have to do what the how-to actually says if the feedback is to be relevant.

I'm really busy today (watching over kids at a LAN-party after work), I'll get back to this tomorrow.

::Trym
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: sixstone on January 22, 2011, 05:24:35 pm
Hi there,

Regarding to your questions about /var/lib/conf/CA, the directory is backed up by the configuration backup as it is a configuration setting as LDAP directory content is so it is not necessary to include it in your data backup.

Best regards,
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Josep on January 22, 2011, 08:44:00 pm
Aha! :-D
There we have the CA question solved.
Like I said, I will try it on a clean installation, and repeat the same process at least twice before reporting a failure ... or success.
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Trym on January 23, 2011, 01:05:04 pm
First of all, thank you for the thank yous, messages like that encourage me to go on.

I've been mildly surprised that no-one has asked why I don't use the built-in configuration-restoration instead of a script to restore the modules. My original goal was to use the restore-process not only to restore a complete server, but for "splitting" a server as well, ie. restore only specific functions. (If you run into performance issues on a server for example, you could split the functions it performs between different servers using only the backup.) Part of that plan was to be able to do that live, without downtime and without unplugging any cables. To do that we'd have to be able to restore to a server with a different IP. If I'd used the built-in restoration-process, it would also restore the network-module, and hence, if static ips were configured, we'd get an ip-conflict. I was hoping that Zentyal, upon restoring certain module-settings, would automatically adjust IP/interface-dependent modules like firewall, DNS, DHCP etc. This is not the case. At this time it seems unavoidable to, at one point or another, set the restored server to the original ip-adress (eventually to change it to something else again, which Zentyal will happily do.) Therefore there's no longer any point in not using the built-in, complete restoration.

(A second goal was to make pretty much anyone able to follow the how-to, by generally being very specific about what to do, not how it works, and trying to avoid technical terms. I think most non-geeks will feel much safer using the webconfig instead of the shell where possible.)

So, this is just a heads-up that there will be some changes to the HOW-TO very soon, I need to run a battery of tests first. Good news is that you won't have to change anything if you've already implemented it, the only thing that will change is the text and the last part of the restore-procedure.

::Trym
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: sixstone on January 24, 2011, 09:02:20 am
Hello Trym,

If I'd used the built-in restoration-process, it would also restore the network-module, and hence, if static ips were configured, we'd get an ip-conflict. I was hoping that Zentyal, upon restoring certain module-settings, would automatically adjust IP/interface-dependent modules like firewall, DNS, DHCP etc.

If you use a static network configuration, it is currently impossible to restore the network module to adapt itself to choose another IP address. But it could be addressed in the future ;). However, you may set your server with a DHCP interface and use a static fixed address mapping in your DHCP server to serve the same IP address lease.

Thanks very much for your findings here, it is really appreciated ;)

Best regards,
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Trym on January 24, 2011, 05:38:25 pm
Quote
If you use a static network configuration, it is currently impossible to restore the network module to adapt itself to choose another IP address.

Thanks for confirming, and also for confirming the (absence of any) certificate issue. It's always nice to get clean, authoritative information.

Quote
However, you may set your server with a DHCP interface and use a static fixed address mapping in your DHCP server to serve the same IP address lease.

I certainly could, yes, but I'm not comfortable with having that as a requirement for the how-to. So for now I'll have to forget the idea of a live restore, and instead make a script which will examine the configuration backup and show the user the IP they have to set the server to before restoring settings. (...until I have time to study the Zentyal API and be able to do it from the script.)

Quote
Thanks very much for your findings here, it is really appreciated.

I appreciate the appreciation ;-)

Now, back to testing....

::Trym
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Trym on January 24, 2011, 10:00:37 pm
The guide has now been amended to restore roundcube (webmail) settings and mail. Note that there is an addition in the backup-settings ("var/mail" added) and that the "restoreall" script has been changed. (You cannot just insert a line with the webmail-module, the order has been changed as well.)

::Trym
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: richterd on February 17, 2011, 01:15:17 pm
Hi,

I read your Howto, becaue I want to resize the LVM-Volumes on my server and I dont want surprised of an disaster.
Btw. I am enthused of the howto. Its written simly and understandingly. Great Job!

Now I want test your howto with an virtual session I created for testing something on zentyal. (Its better test in virtual installation before I do it on a working Maschine)

I dont have an additional Backupserver. I want backup to an external USB-Drive.
How can I change your howto to backup and restore from a external USB-Drive?

Thanks and regards
richterd
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Trym on February 18, 2011, 03:10:18 pm
Well, you're asking outside the bounds of the HOW-TO, so I will reply out-of-bounds myself, meaning I have not tested this:

First, you need to learn how to mount a USB-drive. Look up the linux "mount" command on the internet, or look here for an example. (http://linuxservertutorials.blogspot.com/2008/11/ubuntu-server-mount-usb-drive.html)

In the backup-configuration, you need to choose "File System", and set your USB mountpoint as the backup-directory. (If you follow the link above that would be "/mnt/usb/".)

Next, replace all "ftp://user:password@backupserver" in the HOW-TO with "file://<usb_mountpoint>". (Again, "/mnt/usb/" if you followed the link. Note that the complete path to the root of the usb-drive will have three "/"'s in it, like "file:///mnt/usb/".)

When you are ready to restore the server, after installing it you need to mount the drive again before running any of the duplicity commands.

Good luck.

::Trym
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: satyris on April 08, 2011, 10:02:03 am
Hello Trym,
do you know that failure ?

Code: [Select]
duplicity restore ftp://ftpuser:password@backupserver/Verbatim-STORENGO-01/zenbackups / --no-encryption --force
NcFTP version is 3.2.2
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Apr  7 10:29:25 2011
Invalid data - SHA1 hash mismatch:
Calculated hash: 307e439c1f7af22d058ef268e897c4f093067f25
Manifest hash: 72aa9d920dcd89fcf451d355df5b3bc129e9a9d9
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Trym on April 08, 2011, 03:09:36 pm
Yes, I've seen it before. It appears when:

o The backup module has been upgraded

AND

o The upgraded backup module has appended backups to an already existing backup set.

The strange thing is that if you run the command again, it will not display the error. (If it does then your backups are probably corrupt.)

I never did find out if this is indeed an indication of a corrupt backup, or just versioning differences. Personally I archived old backups and started with a fresh backup set, haven't seen the error since then.

To be on the safe side, I advise to create a fresh backup set (if possible.)

I *think* you can restore the backup without incident, but it was pretty hard to recreate the circumstances leading up to the error (the ebackup-versions which cause this are no longer available (somewhere between versions 2.0.5 and 2.0.9), so I never did get a chance to test this thoroughly.)

::Trym
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: satyris on April 08, 2011, 04:06:06 pm
Different versions: I will check that tomorrow, but shouldnt be

Quote
to create a fresh backup set
Do you mean to make a fresh backup again ?


Reopend my ticket:
http://trac.zentyal.org/ticket/2512
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Trym on April 08, 2011, 04:27:17 pm
Yes, by "fresh backup" I mean first remove *all* of the existing files in the backup-destination, then create a new backup (manually or scheduled, doesn't matter.)

If you've been using the same ebackup-version all the time, then this error is serious, it means the checksum of the remote backup is different from what duplicity expects, in other words a corrupt backup.

From your ticket I see you can get it to work by using --file-to-restore, which is another indication the *some* of the files in the complete set are corrupt. It could also, of course, be an indication of a duplicity bug ;-)

Another quick thought: You say you're transferring the files to a USB-stick, then try to restore from that. Which file-system do you use on the stick? Tried changing it? Doubt it will help, but worth a shot.

Sorry to be of little help.

::Trym
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: satyris on April 09, 2011, 02:11:27 pm
Hello Trym,
on both systems are the same ebox-backup versions.

Code: [Select]
#apt-cache policy ebox-ebackup
ebox-ebackup:
  Installiert: 2.0.10
  Kandidat: 2.0.10

removed all the old backups from my backup-directory and started a new one whith:
/scripts/fullbackup

still waiting for finish  ;)

The usbstick has FAT32-Filesystem, my USBHD has ext3/4. Both tried and both failed.

But anyway, thanks for your help !

Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: satyris on April 09, 2011, 07:03:52 pm
Iam going crazy. After the fullbackup i have no "confbackup.tar" to restore ?

At the original server there is that file missing, too.
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: satyris on April 10, 2011, 09:23:24 am
Removed/installed duplicity and ebox-backup. Cleared all .cache-directories and made a new fullbackup.
Copied it on an usbHD (ext4) and from it to my virtualbox-zentyalserver.

This is the error-message:

Code: [Select]
Last full backup date: Sat Apr  9 20:36:22 2011
Error '[Errno 17] File exists' processing home/admint2s/Desktop/zentyal.desktop
Error '[Errno 13] Permission denied: '/home/aquota.group'' processing home/aquota.group
Error '[Errno 13] Permission denied: '/home/aquota.user'' processing home/aquota.user
Invalid data - SHA1 hash mismatch:
Calculated hash: f55eeac10d4c8203e9e90108eb3df79d4402cd9a
Manifest hash: 1bde9b8b7b53d27068300d502df65c92dd25b5e0

or with -v 9:

Code: [Select]
Deleting /tmp/duplicity-cAeg6A-tempdir/mktemp-gvJTJ1-2
Forgetting temporary file /tmp/duplicity-cAeg6A-tempdir/mktemp-gvJTJ1-2
Processed volume 1 of 214
Registering (mktemp) temporary file /tmp/duplicity-cAeg6A-tempdir/mktemp-X7JC6Y-3
Invalid data - SHA1 hash mismatch:
Calculated hash: f55eeac10d4c8203e9e90108eb3df79d4402cd9a
Manifest hash: 1bde9b8b7b53d27068300d502df65c92dd25b5e0

Removing still remembered temporary file /tmp/duplicity-cAeg6A-tempdir/mktemp-X7JC6Y-3
Removing still remembered temporary file /tmp/duplicity-cAeg6A-tempdir/mkstemp-8RJ077-1


The only difference in Version i found is that the originalserver is AMD64 and the virtual is i386

Dont know any further
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: droker on April 16, 2011, 03:55:28 pm
Hi Trym.

I have tried everything and i cant figure out what im doing wrong.

I followed your excellent instructions to the letter and everything gets restored except i cant login to Zarafa, either via outlook or the web interface.

It just says my username and login are incorrect.

I didnt get any error messages during the restore and the database all seems to have been imported correctly?

Any ideas?

Thanks

Dean
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Trym on April 16, 2011, 05:39:12 pm
I assume you have restarted the webserver already, and if that didn't work that you have rebooted.

You can safely uninstall the groupware-module and reinstall it, that doesn't delete the mail-database. That is the first thing I would try.

If you're running zarafa on a virtual domain, you could also try setting it to a different domain (no domain), save changes, and then setting it back and save.

::Trym
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: droker on April 16, 2011, 06:59:47 pm
Hi Trym,

I have tried everything, i have backed up and restored everthing about 4 times now all with the same outcome.

I have moved to no domain and then back to the actual domain. Nothing seems to work, but all the users appear in the LDAP database and have Zarafa accounts enabled.

in desperation I added var/lib/zarafa to the included files because i gather that is where the attachments are stored, but that made no difference.

I am at a loss.

Any other ideas? or things i can check, its almost as if it cant read the LDAP database. Where are the logs kept do you know? I did notice the password sourced in the scripts was different on the original server to the backup. Could that be it?

Thanks

Dean
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: droker on April 16, 2011, 07:30:54 pm
I just logged back into the original server, when i type zarafa-admin -l it shows all the users, when i do it on the backup it just shows the System user.

What the hell could i have done wrong?
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Trym on April 16, 2011, 08:05:03 pm
I really don't know, I've never seen this before.

The only thing I can think of is that there is a problem with the restoration of the zarafa-database.

Are you able to restore only the file /backup/zarafa.dump.gz?

If yes, does it appear to be the right size?

..and what happens when you run

/scripts/rebuildzarafa

(Restore the user-module again after you do the above.)

I'm sorry, this is uncharted territory for me as well.

Quote
I did notice the password sourced in the scripts was different on the original server to the backup. Could that be it?

Not sure which password you're talking about here. The scripts read the mysql zarafa-password from the current system (and they are supposed to be different from system to system), any other passwords shouldn't matter at all.

::Trym
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: droker on April 18, 2011, 01:29:14 pm
Hi Trym,

I have been scowering the Log files and during the restore the only thing i can see which might be causing a problem is


2011/04/18 12:23:44 DEBUG> MailUserLdap.pm:87 EBox::MailUserLdap::setUserAccount - mail account ssssss@rkrtech.co.uk already exists.
2011/04/18 12:23:44 INFO> MailUserLdap.pm:402 EBox::MailUserLdap::__ANON__ - Creation of email account for ssssss failed
2011/04/18 12:24:00 DEBUG> MailUserLdap.pm:87 EBox::MailUserLdap::setUserAccount - mail account aaaaaa@rkrtech.co.uk already exists.
2011/04/18 12:24:00 INFO> MailUserLdap.pm:402 EBox::MailUserLdap::__ANON__ - Creation of email account for aaaaaa failed
2011/04/18 12:24:15 DEBUG> MailUserLdap.pm:87 EBox::MailUserLdap::setUserAccount - mail account bbbbbb@rkrtech.co.uk already exists.
2011/04/18 12:24:15 INFO> MailUserLdap.pm:402 EBox::MailUserLdap::__ANON__ - Creation of email account for bbbbbb failed

Any Ideas whats would cause that? and do you think its the source of my problem?

Thanks

Dean
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Trym on April 18, 2011, 04:34:08 pm
Yes, that certainly seems like the problem. And no, I have no clue why that happens.

It seems I have to run some tests, Zentyal may have made some recent changes which introduce this error.

Right now I'm on holiday, so I can't promise I'll get it done anytime soon, but I will do it. Sit tight.

Thanks for reporting this.

::Trym
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: droker on April 18, 2011, 04:36:11 pm
Thanks Trym.

In the mean time i am going to try a fresh install and see if it was my inital install that was the problem.

Thanks

Dean
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: droker on April 18, 2011, 06:59:46 pm
Hi Trym,

I have just done a test from a fresh install and had exactly the same result. I am not sure whats going on. Enjoy your holiday and any help you can give when you get back would be really useful.

Thanks

Dean
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: droker on May 03, 2011, 05:24:36 pm
Hi Trym,

How was the holiday? I dont suppose you have had chance to have a gander at this have you? I have tried a couple more times, always with the same result. I cant work out why the user database does not sync to Zarafa.

Thanks

Dean
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Trym on May 05, 2011, 02:41:24 pm
Sorry for the delay.

So, I've done some testing.

First, I tried to restore a backup of my real home-server to a virtual machine. This procedure has worked countless times before. Now it doesn't. I didn't bother to even examine logs, cause the next logical step was to install a fresh virtual server and make a backup to restore.

Whaddayaknow, from a fresh 2.0.3 install to complete updated version, Zarafa now does not work at all.

Maybe I'm in a bad mood, but for me this is the final drop. I'm not even going to bother to try to make it work. I've had it with the endless string of bugs and modifications. I deeply regret (and have for a long time) ever deploying it. I can't count the man-hours it has cost constantly adjusting to a moving target.

(I wrote a page with detailed frustrations here, but decided to remove them, as they didn't add anything constructive.)

So basically what this means it that I will remove the how-to from the front-page, add a warning that it cannot longer be used, but still let people who want to read it (if they want to try additional adjustments themselves.)

This is not meant as a criticism of Zentyal per se, I'm sure it is useful for the majority of people/businesses out there, but for me personally I feel it does not suit my needs nor the needs of the people I do some work for.

I'm sorry for being unable to help, I'm sorry for withdrawing the how-to, but I cannot justify spending even more time trying to fix stuff.

Again, deeply sorry, and best of luck with Zentyal. Perhaps they have an updated backup and restore guide for you.

::Trym, over and out.
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: droker on May 05, 2011, 09:47:59 pm
Hi Trym,

Im really sorry to hear that you have given, up, I suspect the problem is actually quite a simple one to solve if we can find the cause. (anyone from Zentyal care to comment?)

But thank you very much for all your efforts. They have been really useful to me so far.

I have just tried some things tonight and seemed to have made a little headway.

if you disable and re-enable each of the zarafa user accounts they then appear in the user list when you run zarafa-admin -l but there is no mail when you log into the webmail (as it has created a new mailbox). if you run zarafa-admin --list-orphans it shows each of the orphaned mailboxes, you can then hook the exisiting mail accounts to the orphaned mailboxes and the emails appear back in the correct place again.

I will keep trying to see if i can work out why it doesnt work normally.

Do Zentyal support people look in here at all. Is there anything you can suggest?

Thanks

Dean
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: chunk.one on May 29, 2011, 09:22:53 am
Quote
if you disable and re-enable each of the zarafa user accounts they then appear in the user list when you run zarafa-admin -l

I have the same problem after export configuration and files from one server and import it to another server.

Is it possible to script Zentyal functions instead of clicking in the WebGUI??
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: chunk.one on June 02, 2011, 11:41:00 pm
No one know if it is possible to send a shell commands to Zentyal instead of clicking in the Zentyal GUI?
Title: Re: DRAFT: HOW-TO for Disaster Recovery from ftp, scp or rsync backup. Feedback wlc.
Post by: Josep on June 03, 2011, 11:22:19 am
You can do anything you want with Zentyal from the shell ... as long as you can code in Perl.