Let's Recover!

[SeanPF]

A while ago I posted looking for help on disaster recovery: http://forum.zentyal.org/index.php/topic,7613.msg30223.html#msg30223

The problem remains, it is easy to make backups, but it is not easy to use/restore them.

There haven't been any comprehensive or easy to understand solutions put forward yet. The official documentation is not complete and doesn't cover the most typical situations: http://doc.zentyal.org/en/backup.html#how-to-recover-from-a-disaster

For me, this is the biggest weakness in Zentyal. I love Zentyal, but I'm afraid to use it in a professional business place until I can figure out how to actually use the backups that I have in case a real disaster does happen.

It would be great if we can figure out a way to make disaster recovery feasible for those who are not Linux gurus (like me). Perhaps with this postenough ideas and suggestions can be put together to get a working solution.

Here is the typical situation that I need a solution to, and I think many others would like as well:

1) Imagine I have a working Zentyal server that does full backups every week, and incremental backups every day. It also does configuration backups every day. It runs the backups via ftp or scp to a different computer, or to an external hard drive using the built-in feature, duplicity.

2) Now imagine my server crashes. The motherboard is fried and the hard drives are shot. Or better yet, there is a fire and the machine is destroyed.

3) I need a replacement server, quickly, so that all the domain users can continue to work. Most likely the machine used to replace the destroyed server will have different hardware. This will cause conflicts when restoring the settings and configurations from the destroyed machine. Perhaps I had RAID set up. Perhaps I don't even remember how exactly the partitions were set up. Do they have to be exactly the same on the new machine? Etc. Etc. Etc. How do you restore the important configuration and user files onto a new machine with different hardware?

This seems a very typical situation many will find themselves in. Here is a brief summary of what I've tried to no avail (for more details look follow the link to my previous thread at the top of this post):

I tried taking a new machine with a blank hard drive, using the GRML as the official documentation suggested, and using duplicity to restore the backups that I had to the new machine. Of course it didn't work. It didn't even make the hard drive bootable. I thought the restoration of backups would also install the boot loader, but I guess that is another process (one I don't know how to do).

When that didn't work, I figured I needed to install a fresh copy of Zentyal on the machine before restoring the backups. I reformatted the drive and started over. This time I installed a fresh copy of Zentyal on the machine. Then I put in the GRML CD and again used duplicity to try to restore the backups. This time it gave me many, many errors saying that the files already existed. So obviously that was not the way to go about it.

Next, I reinstalled Zentyal, and this time, from the web interface of the new machine I tried to restore the configuration file of the destroyed server. This returned errors, like I expected, presumable because the hardware is different. Anyway, some modules were set up properly, and some weren't. The users and groups were successfully restored (at least they appeared in the web interface). Then I used duplicity from the CLI to try to restore only the /home folder, in order to get the user files onto the machine. I then tried to copy over the etc/passwd  but that messed everything up even more, and I wasn't even able to log in anymore with the root account.

As you can see I tried different approaches to this task that should be relatively simple, and will probably be quite a typical situation. What we (n00bs) need is some clear conceptual understanding of how this is supposed to work, and then some good documentation.

If you can contribute to solving this, I'd appreciate it. If you do reply, try to make it as easy to understand as possible. I've been working with computers for many, many years, but I still struggle to understand lots of what experienced Linux users say because I lack the context to understand (please avoid acronyms, etc.)

Thanks

[stuartiannaylor]

I understand where you are coming from. I have also had this happen on a microsoft platform several terms. One time I got this spooky scenario that all the files had some sort of duplicate ghost file. In that only one file existed but there seemed to be two sets of security descriptors.


Personally I would like to be able to dump all the user settings (ldap) to a backup dump. A system config isn't that important as I can set up zentyal in a blink of an eye nowadays 
Then all I want to do is backup up the samba files and databases.
If its different hardware or even a different setup I can still quickly get myself going and with the above inclusions I now I can get things back.


I am really not bothered about the system as I can always reinstall that.

[jsalamero]

All use cases you expose here are covered with Zentyal Disaster Recovery: http://www.zentyal.com/en/services/addons/disaster-recovery/.

Zentyal configuration backups include LDAP tree which will be restored as the users & groups module is restored.

[SeanPF]

Jsalamero, I don't see anything different with this expensive subscription than with what I am already doing. I am remotely backing up all my data securely. The problem is with restoring data on a new machine without getting everything messed up, especially when there is different hardware.

Restoring the users and groups module on a new machine produced unpredictable results, by the way. The documentation is just not there. In what order, for instance, are you supposed to do it; restore the user and group files first, and then restore the config file, or the other way around? What about file permissions? What about the passwd file? Should that be restored before or after restoring the config file, and the user files? What will happen when you copy the passwd file from another machine to the one you are working on? What if you have a different root username? I locked myself out of my own machine last time. How does restoring the config file work when it encounters different hardware? Etc. Etc. Etc.

[christian]

Sean,

If you REALLY need to restore "an image" rather then restoring data, then use of VM may help solving this potential issue.
With scenario you describe, I don't know any suitable solution and this is not due to Zentyal.
On other platforms you operate that are not running Zentyal, do you have any reliable DRP?

Then, trying to answer to your question from technical standpoint (because your point - that is to restore - is fully valid), I would approach it with different recovery plan depending on failure you have to face:
- in case you restore on same platform (from  hardware and OS standpoint), restoring the full image should work.
- in case you have changed significantly hardware and OS, you can still reinstall Zentyal and then restore config and files, can't you? Restoring users and groups should not provide unpredictable result at least from LDAP standpoint: generating an LDIF file permits to restore same content on "any" LDAP server. passwd file doesn't contain, as far as I understand, any input for end-users accounts (that are managed in LDAP).  it contains accounts for LDAP, Postgres etc...

[jsalamero]

Jsalamero, I don't see anything different with this expensive subscription than with what I am already doing. I am remotely backing up all my data securely. The problem is with restoring data on a new machine without getting everything messed up, especially when there is different hardware.

DR is not only about a remote backup but a well tested restore procedure which is integrated in the installer and allows to restore you data over a new clean install of Zentyal into the same or a similar server (same number of network interfaces, enough hard disk space...).

[SeanPF]

Christian, too many acronyms. I asked at the beginning of the post that I want a solution for non Linux gurus, in plain language with clear instruction. I really didn't understand most of what you said.

So what I gather so far is that the only solution to disaster recovery is to pay for the Zentyal Disaster Recovery subscription.

[stuartiannaylor]

I think the disaster recovery system is a little sketchy. Zentyal do offer an excellent subscribed recovery system.

The ability to move from one server scenario to another may have vastly different implementations. I would really like to see a user manager that can import and export Zentyal users. It covers a lot of bulk admin chores.

I do think the Zentyal core should provide a few tools for migration and recovery.

[christian]

Christian, too many acronyms.

Sorry 
DRP stands for Disaster Recovery Plan
OS stands for Operating System
LDAP stands for Lightweight Directory Access Protocol and LDIF is LDAP Data Interchange Format.

Does it clarify? 

[SeanPF]

Ok, let me pose some specific questions just to get things started.

In the official documentation on how to recover from a disaster, here: http://doc.zentyal.org/en/backup.html#how-to-recover-from-a-disaster , is it presumed that you are working with A) the same machine that experienced a disaster, or B) a new machine?

And secondly, is it presumed that you are applying these steps A) onto a blank hard drive before installing a fresh copy of Zentyal, or B) after installing a fresh copy of Zentyal?

If the presumption is that the steps outlined are to be done to a blank hard drive, the documentation completely ignores the step of how to make the drive bootable. On the other hand, if it is presumed that you already installed a fresh copy of Zentyal, then following those steps will actually break your new installation (as I've discovered after many attempts).

So, first of all, what does this documentation presume is your current situation?

[greavette]

Here's my two cents with how I support our small office and protect against disaster recovery.

I support our small office remotely (8 hours away from where I live) so early on when I was asked to help support them, I made the decision to use Virtual Machines (VMware) as much as possible.  We have 3 large Host Servers (all with either Raid 1 or 10 hardware controllers) running the following types of services for our business:
-  Lab Asset Management Database (SQL on Windows 2008)
-  Citadel Mail Server (installed on Ubuntu Server)
-  Quickbooks Accounting
-  Adito (OpenVPN-ALS) remote access portal (installed on Ubuntu Server)
-  Simple Groupware server (installed on Ubuntu Server)
-  eGroupware (Addressbook) server (installed on Ubuntu Server)
-  iFolder - for remote backups (installed on Ubuntu Server)

Where possible I have regular backups (sometimes many times a day..sometimes once a day) of data from each server sent offsite using our iFolder server.  We use Cobian Backup on one of our Host Servers to send files and folders to a location where our iFolder client will copy changed files and folders to our offsite iFolder PC. I also have an RD-1000 hard disk backup unit with 4 disks.  Each week I backup all Virtual Machines to the hard disk and the owners of this business take the disk home and replace it with another disk.  That way I have 3 weeks of Virtual Machines to restore from if needed.  The system is not fail-proof.  I'm still learning how to use crontab to setup the automatic backups from all of our Ubuntu servers.  But I can ensure the owners that if there was a fire or other disaster at their company, we can restore their most important data (like the LIMS database and Accounting files) and most of their other data fairly quickly on a new Host server once VMware has been installed.

In case any of the 3 host servers go offline, I do have a spare host server I can bring online to help with the load.

Looking at the links provided in this post, I will be investigating how to setup a cron job to run the ebox-make-backup utility so I can save our Zentyal configuration settings offsite.  But we've only just begun using Zentyal and it is only as a Domain Controller so saving our Virtual Machine each week would be enough for now to protect us.

I hope this helps you with your own disaster recovery.  I'll keep my eyes on this thread to get more ideas.

[christian]

I fully share that VM (virtual machine) is the easiest way to cover this kind of risk, assuming you understand there is a drawback with such design, either in term of cost, complexity, resource consumption...

Defining the ideal disaster recovery plan (DRP) really depends on risk you want to cover, RTO (return time objective) and RPO (return point objective). In some cases, it could be so complex because of your specific needs that buying service might be the best solution.
Unfortunately, there is no universal free solution that can be just explained with few pages of documentation 

[greavette]

I should mention as well that I discussed our backup and disaster recovery plan with the owners before I set this up for them and I gave them the option of buying offsite backup or some kind of service.  They opted to build an inhouse solution as I described above and to use an offsite server that they had control over (our ifolder server connects and sends data offsite to another office owned by them).

Depending upon your needs and what the owner/operator has set out as requirements will help you decide upon your own disaster recovery plan.  Christian is right...it may be easier and the best interest of the company to buy a solution rather than build your own.

[SeanPF]

Well thanks for the opinions, alternative suggestions and everything, guys, but I started this thread to focus on finding solutions to specific questions about Zentyal, which I still am determined to discover.

Again, let me pose this question:

Does anyone know if in the official documentation on how to recover from a disaster, found here: http://doc.zentyal.org/en/backup.html#how-to-recover-from-a-disaster , it is presumed that you are working with A) the same machine that experienced a disaster, or B) a new machine?

And secondly, is it presumed that you are applying these steps A) onto a blank hard drive before installing a fresh copy of Zentyal, or B) after installing a fresh copy of Zentyal?

[christian]

Sean,

To me it's pretty obvious that if you go for a complete restoration, your hardware has to be reasonably similar, especially in term of network and storage size.
You boot from CD and mount hard drive that will then hold your root partition (at least in example shown there).

Then I don't think your dichotomy is correct:
- to me (although I didn't try this howTo) you can restore to either same hardware or new hardware assuming the new one is able to handle what you will copy (in term of partition size) and similar (in term of network configuration)
- there is a lot of steps between "blank drive" and "after installing fresh copy of Zentyal"    e.g. you may have disk with partitions, made bootable (MBR, grub...) or with operating system installed.

Grml is there to provide tools allowing to deal with above steps depending on where you are and what you want to achieve.

If your idea is to finger-point that documentation is not perfect because not describing the very detail of every different case, you're right but I also don't think this is realistic.

I do agree however that disaster recovery process should try to reach level of easiness already reach by Zentyal product. I mean anyone with no sys admin skill can deploy Zentyal while recovering from disaster is not as easy. I think that this is the reason why Zentyal is offering such service