Zentyal Forum, Linux Small Business Server

Zentyal Server => Installation and Upgrades => Topic started by: drdebian on December 29, 2007, 04:04:11 am

Title: Add-On: BackupPC request.
Post by: drdebian on December 29, 2007, 04:04:11 am
OK, I'm not sure if this is the right place to post it, but here it goes anyway.

I think it would be cool if BackupPC (http://backuppc.sourceforge.net/) would be integrated into ebox, so it can really act as a central server for all needs, including backups.

Reason:
BackupPC is written in Perl, just like ebox.
In addition, it only uses tools that are already present on a stock ebox installation (excluding rsync perhaps).
Unlike other backup solutions, BackupPC requires no special client-side tools to be installed, which really helps ease of use. It can use anything from SSH, SMB or Rsync.
BackupPC is entirely disk based, so no tape handling is necessary.

Any thoughts?
Title: Re: Add-On: BackupPC request.
Post by: javi on January 02, 2008, 10:28:49 am
I've just taken a look at it and it looks very nice.

Let's see if I can have some time to play with it and I'll get back to you with some feedback about its use with eBox.

Thanks!
Title: Re: Add-On: BackupPC request.
Post by: drdebian on January 02, 2008, 02:48:44 pm
I've just taken a look at it and it looks very nice.

Let's see if I can have some time to play with it and I'll get back to you with some feedback about its use with eBox.

Thanks!

No problem.

I forgot to mention that the configuration files for BackupPC are really easy to generate and starting with version 3.x, it also comes with it's own powerful webfrontend, which I'm sure can be integrated into ebox with little effort.
Title: Re: Add-On: BackupPC request.
Post by: austin on January 03, 2008, 09:39:58 pm
+1   

 I also feel this would make a great a addition to ebox
Title: Re: Add-On: BackupPC request.
Post by: timeJunky on January 05, 2008, 10:41:08 am
Hello, I would also appreciate backuppc on ebox.
Title: Re: Add-On: BackupPC request.
Post by: patcunha on January 14, 2008, 08:19:05 pm
I also feel this is a very good addon to Ebox.
Title: Re: Add-On: BackupPC request.
Post by: chencho on January 28, 2008, 09:19:54 am
Hi all!

Is possible install backupPc plus ebox? (not as addon).

If we have it together will be perfect, but "as moment" will be nice to have it.

Previusly i used rdiff-backup, but it's not compatible with ebox
Title: Re: Add-On: BackupPC request.
Post by: chencho on February 27, 2008, 01:16:39 pm
rdiff-backup is compatible with ebox if we use Ubuntu Server instead Debian Sarge Dist. :)
Title: Re: Add-On: BackupPC request.
Post by: mlanner on May 25, 2008, 01:36:49 am
... or take a look at RESTORE at www.restore-backup.com.

It's super simple to use. I think even simpler than BackupPC. I've used them both.

RESTORE also runs on Ubuntu 8.04.

I'm considering trying to put eBox and RESTORE on the same box to see if it works.
Title: Re: Add-On: BackupPC request.
Post by: mlanner on May 26, 2008, 08:16:36 am
Just an update ...

I installed eBox from the ebox_installer iso today. After configuring the system I added RESTORE's Gutsy repository to the /etc/apt/sources.list file and installed RESTORE. So far the two systems seem to coexist just fine.
Title: Re: Add-On: BackupPC request.
Post by: patcunha on June 01, 2008, 03:03:12 pm
Can restore be used to make a image of a PC and cast this image to a lot of other PCs through the network?
Title: Re: Add-On: BackupPC request.
Post by: mlanner on June 06, 2008, 08:33:40 pm
patcunha,

As far as I know, no, you can't do a complete image of a system with RESTORE. RESTORE is not a system image tool, it's a backup tool. If you want to make system clones, use a Symantec Ghost or some open source cloner. I don't have a need for cloning systems very often, so sorry if my suggestions are of limited help.
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on July 22, 2009, 01:28:31 pm
Any word on if either BackupPC or RESTORE could be used? Both of them look like very capable solutions. I guess my asking for some kinda RSync or some way to do this was already looked at two years prior, haha. If it's not in eBox, I can at least install it manually no problem.
Title: Re: Add-On: BackupPC request.
Post by: SamK on July 22, 2009, 03:00:43 pm
Any word on if either BackupPC or RESTORE...

BackupPC - Most definitely, it has also been in the wish list for a while.

If you mean Restore-Backup
http://sourceforge.net/projects/restore/
it seems to no longer be in active development since January 2009.
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on July 23, 2009, 12:19:33 am
Yeah, it's gone :(. I wanted to try it out too. BackupPC is by far, the best solution out of all the backup ones I have messed around with. Now if it would only work. I have too new a version of Samba for it so I don't wanna mess around with installing it until there's a newer one which will work with Samba 4.7.
Title: Re: Add-On: BackupPC request.
Post by: SamK on July 23, 2009, 10:02:59 am
...I don't wanna mess around with installing it until there's a newer one which will work with Samba 4.7.

The project is constantly being developed. I've always found the main developer very approachable, the user group is also very supportive and responsive.  It might be worth you getting in touch to see what the plans are.
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on July 23, 2009, 10:05:09 am
Awesome. Then yeah. I have it installed actually now. I checked on another rig and it had Samba 3.2 or something. I dunno what was up with mine. I installed it and was so utterly confused I might frequent their IRC channel again sometime and hope someone's around. Since I'm moving to new hardware, that'll fix any issues I had before. I'm gonna just install everything fresh.
Title: Re: Add-On: BackupPC request.
Post by: SamK on July 23, 2009, 10:29:13 am
This post also requests BackupPc and lists some of the primary benefits it offers.
http://forum.ebox-platform.com/index.php?topic=1279.0

It will be good to hear from the eBox devs with a more detailed view of the future plans/ideas for backups of:

Also on the topic of disaster recovery of a single or multiple eBoxes

Javi how about an insight from a developer's perspective?
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on July 23, 2009, 10:40:16 am
Agreed.
What do you personally use BackupPC for? How many users do you run it on. Do you think my friend could have an RSync module installed on her Win7 laptop and have that setup an SSH tunnel to my machine using BackupPC/eBox and backup or sync certain folders from her computer in that way? If so, then it means eBox will finally have that functionality that everyone wants, the ability to backup to remote locations instead of just on the eBox itself. I could you could SSH mount a network location and have eBox backup to that directory too. Maybe. That's a bit of a hassle. Maybe having BackupPC running the backend of it might be easier since it would connect to eBox's LDAP users. That's probably the most important part I'd say, connection w/ eBox's LDAP users. If there was an API to connect to LDAP users, then there might be more add-on apps, but that's kind of out of the way I'd say, but it would definitely make eBox a platform people start developing for.
Title: Re: Add-On: BackupPC request.
Post by: SamK on July 23, 2009, 12:04:47 pm
I no longer manage the LAN on which I implemented BackupPC, but from memory...

The LAN hosted various server machines each having various data stores, user home, group shares and such.  The primary objectives were
There were of course others but it shows like a list of benefits of using BackupPC.

At the start I knew nothing of how to achieve these goals but with the help of the main developer Craig Barratt, the user group, and the project documentation, it was very successful.  It really is every bit as capable of securely backing up servers to a central resource as it is securely backing up PCs to the same resource.

With reference to backing up a Windows laptop via an SSH tunnel to a BackupPC server I have not used it this way but a quick search of the BackupPC wiki indicates it can be done. Have a look at
http://backuppc.wiki.sourceforge.net/CommonConfigurationIssues

I cannot overstate the helpfulness of the project and recommend you get in touch.
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on July 24, 2009, 07:53:07 am
Think you could PM me some contact info? I really wanna try. I think if I do I might be able to help give the eBox devs at least my own post of what I'd like to see in a solution like this if it was integrated.
Title: Re: Add-On: BackupPC request.
Post by: SamK on July 24, 2009, 11:23:14 am
Think you could PM me some contact info?
PM sent.
Title: Re: Add-On: BackupPC request.
Post by: stephdl on August 15, 2009, 10:42:00 pm
I use backuppc on sme-server (yeah it's the devil, it's RPM), but the sme use perl too.

http://wiki.contribs.org/BackupPC

i post this address if you want to ask somes questions to the developper.

for the moment on ebox, you can use backuppc, but you need to set manualy the user password.
 sudo htpasswd /etc/backuppc/htpasswd 'user'
Indeed it is interesting to create separate users to access for themselves to their backup, because the user base after the installation is the admin (backupc), so with all the rights.

also it would be more convenient if the management of users were included in ebox, certainly with LDAP authentication
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on August 16, 2009, 12:02:46 am
Yep. I'm in agreement with that.
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on January 12, 2010, 07:41:40 pm
One of the other threads on this is: http://forum.ebox-platform.com/index.php?topic=1279.0

Hey all, I'm back with some news about BackupPC itself. While drdebian's original posting has a lot of good points in it, asking the eBox devs to integrate BackupPC into eBox is quite a bit of work doing something which really isn't needed.

For one, Ubuntu Hardy has BackupPC 3.0.0 in the repositories. Yes, it's blue, not green/orange, but it's still able to do its job. I've been working with it on my eBox machine for about three weeks now and nothing's been breaking. eBox and BackupPC work in tandem. Make sure to turn off the Backup module in eBox as well because it can use up inodes and is doing the job BackupPC can do. Also, the eBox backup module is backing up slash (/) into your specified location. Great for off-site backups, terrible for on-site backups since your backing up onto the machine itself when BackupPC can do that and include pooling data from all of your other machines.

While they work great side-by-side, some sort of native, out-of-the-box, support would be nice, but it's not something that's on the map right now. Think about all the other features needing tending to. It's not that easy to integrate BackupPC. It seems easy, maybe have the "Backup" area of the sidebar actually point to a working BackupPC install, but other than that, there's not more the devs should do until eBox has a bit more development under its belt. Like, that might be an awesome eBox 2.2 feature for instance.

BackupPC 3.2.0 and the community edition are still in development thankfully so it's not like the eBox devs have to take charge of the project for us.

My only quarrel in all of this is the multitude of supported filesystems in Karmic which Hardy is clearly lacking. I can't wait for eBox 1.4's release because it will allow me to run Ext4 without any headache and have BackupPC, and in effect eBox, be more efficient because of it. I was considering integrating a VM using FreeNAS so I could backup to ZFS, but I gave up on that because it would be too hard on this particular machine solely because of the load put on the machine to do such an inane task. Best to have two machines unless your server is powerful enough to handle ZFS from a VM and has hardware virtualization.
Title: Re: Add-On: BackupPC request.
Post by: SamK on January 15, 2010, 10:56:05 am
I've been working with it on my eBox machine for about three weeks now and nothing's been breaking. eBox and BackupPC work in tandem.
This is really good news.  From previous conversations I know how long you have been looking to find a suitable backup system.

Would you be willing to provide a bit more information about your setup and the way in which you use BackupPC? (Perhaps even a quick 'n' dirty diagram of your rigs.)

Unfortunately, during the last few months, I have not been able to test any of the recent eBox developments. Have you compared the forthcoming (v 1.4) inbuilt eBox backup system with BackupPC?  If yes it would be great if you could be the first to provide some feedback on the feature sets. If no what are the particular BackupPC features you favour? 

Make sure to turn off the Backup module in eBox as well because it can use up inodes and is doing the job BackupPC can do.
I may have missed the point here, are you meaning that BackupPc can be used as a complete replacement for the eBox backup systems? (data, configuration files, and system files)

Did your tests cover restores?  This is an area which has given rise to requests in the forum for assistance when the eBox restore does not work as expected. I am vary of setting up eBox version 'X' and upgrading to version 'Y' by restoring backups made on 'X' to a newly created machine based on version 'Y'.

While they work great side-by-side, some sort of native, out-of-the-box, support would be nice... ... maybe have the "Backup" area of the sidebar actually point to a working BackupPC install,
This seems to be an excellent idea.

I can't wait for eBox 1.4's release because it will allow me to run Ext4 without any headache and have BackupPC, and in effect eBox, be more efficient because of it.
Again my ignorance is on display, how does ext4 improve the efficiency of eBox?
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on January 15, 2010, 12:36:15 pm
Nice to see you back. I was hoping to hear back from you SamK :P.

While they work great side-by-side, some sort of native, out-of-the-box, support would be nice... ... maybe have the "Backup" area of the sidebar actually point to a working BackupPC install,
This seems to be an excellent idea.
If any devs see this, we're waiting for you :D! I think Javi saw one of my LDAP+BackupPC posts a long time ago and added a ticket request for it. I can't think of anyone else asking for it but me: http://trac.ebox-platform.com/ticket/1463. If you go through the "nice to have" tickets in the roadmap, you start to feel really loved as an eBox user. I see a lot of things that you and I had suggested in the past already implemented and used as well as things other people have said. The devs really do listen to what the users want probably more than what they want.

There's a lot of people wanting BackupPC integration, but the best way to integrate it is to just connect to the BackupPC CGI interface through the eBox BackupPC module (which doesn't exist), and then configure LDAP user association so you don't have to setup your users in two places. That would allow you to use both BackupPC and eBox together without having to go crazy and add in 100 users (larger network) with passwords and then have to manage changing passwords in two places instead of one.

A better way to integrate the two, like a feature you'd see later on, would be if the objects you setup in eBox such as "computer-name"--referencing DHCP entries such as 10.10.10.10 and 10.10.10.100 for WLAN and LAN adapters on the same machine--could be used in BackupPC. When you say host computer-name, eBox would handle the setting up of BackupPC hosts that way by telling it which IP(s) computer-name is.

I can't wait for eBox 1.4's release because it will allow me to run Ext4 without any headache and have BackupPC, and in effect eBox, be more efficient because of it.
Again my ignorance is on display, how does ext4 improve the efficiency of eBox?
ext4 is massively faster than ext3 in both initialization, filetransfer, and so many other things. It's also better suited for file and directory fragmentation can occur over time. BackupPC uses a ton of hardlinks to work and having ext3 is a dated approach. I've been looking at benchmarks all around seeing how filesystem performance affects how BackupPC works, and saw that jfs was the most reliably fast and xfs was getting the highest numbers; albeit, not all the time.

ext3, while reliable is slow slow slow especially wen my partions are starting to go over terrabytes of space. ext4 on the other hand is close to the speed of xfs and jfs but can recover from a filesystem fault. Think of what would be combining ext3 and jfs, you get ext4 which is both speed and reliability.

From my own testing, this is the current best way of running a Linux OS. 1.4 is supposed to support Jaunty and Karmic so that's why I bring up ext4 because I will definitely be upgrading the filesystem.

Make sure to turn off the Backup module in eBox as well because it can use up inodes and is doing the job BackupPC can do.
I may have missed the point here, are you meaning that BackupPc can be used as a complete replacement for the eBox backup systems? (data, configuration files, and system files)

Did your tests cover restores?  This is an area which has given rise to requests in the forum for assistance when the eBox restore does not work as expected. I am vary of setting up eBox version 'X' and upgrading to version 'Y' by restoring backups made on 'X' to a newly created machine based on version 'Y'.

My tests did cover restores on both Windows and Linux machines, but only file and folder restores, not the OS. I don't ever trust eBox's configuration and full backup module. Sure I've done some backups, but now I don't even bother. I used to try to backup at one point when testing 1.1, and it was broken in that version. Restores were successful when I did them in 1.0 and 1.1 because I had often broken something which caused DHCP to stop working.

I believe I posted those bug reports in the forums and in the IRC channel, but there was no documentation of why certain things like a space in a hostname would cause the DHCP module to not start after saving the settings. It was just little things, but restoring would fix them.

I don't know exactly what the configuration and full backup module does when you backup and restore so I really can't say if restoring from older versions works or not. When I setup my second eBox from a fresh 1.2 ISO instead of 1.0 -> 1.1 -> 1.2 like I had done before, I actually did all the configuration by hand and when the time comes, will just copy/paste my dhcpd.conf, /etc/apache2, and /etc/bind files and folders and be done. I just want to move those when it's time so I don't have to make changes in two places.

I've been working with it on my eBox machine for about three weeks now and nothing's been breaking. eBox and BackupPC work in tandem.
This is really good news.  From previous conversations I know how long you have been looking to find a suitable backup system.

Would you be willing to provide a bit more information about your setup and the way in which you use BackupPC? (Perhaps even a quick 'n' dirty diagram of your rigs.)

Unfortunately, during the last few months, I have not been able to test any of the recent eBox developments. Have you compared the forthcoming (v 1.4) inbuilt eBox backup system with BackupPC?  If yes it would be great if you could be the first to provide some feedback on the feature sets. If no what are the particular BackupPC features you favour? 

I always wanted to use BackupPC, but it was far too complicated to figure out. Since eBox was easy to configure, I always though some sort of integration of the two would lighten up the learning curve needed to sustain two separate managment systems. After around 6 months, my general knowledge of how to use Linux software has increased to the point that this was finally possible.

I highly doubt some people that get eBox are very well-versed in Linux though. The BackupPC documentation, while awesome, is extremely confusing as to how you should go about setting up hosts and what some examples might be. Once I got at least one computer setup, I finally figured out what I needed to do. I had it setup before, but I did it in the most confusing way that I decided to not longer want to use it.

Here's what you need to know about BackupPC and how it's helpful. When you first run it, try to limit it to backing up one or two machines at a time as it will be building the pool and possibly compressing data at the same time. Make sure to turn compression on level 3 (a good overall value) before you start backing up, or you'll have to go through a very complicated episode of trying to run the eBox compression module manually and then waiting some 15-200 hours for your pool to compress. Trust me, you will want the compression, and it won't kill your CPU provided you aren't starting out backing up more than 2 hosts at a time when you start out.

My next post will include a full-description of using BackupPC.

I haven't done anything with 1.3 or 1.4. I actually have the line commented out in sources.list, just ready and waiting for me to enable it. I have a new eBox which is my BackupPC server right now. I'll eventually make that my main eBox machine and move my current eBox as the test machine which will test 1.3 and 1.4 and then 2.0 beta or whatever. I want that one to be the guinnea pig as it's been all this time.

The Backup module in eBox, the one that does automatic backups, is really terrible to use if you want to backup to the machine you're on. I figure it's probably a bug, but don't know. I thought backing up eBox would backup just eBox folders. This thing backs up all of /. If you backup / to itself, that means you'll be using twice as much space as you normally would be. This happened on my 1.2TB drive and oh man, that wasn't fun. Took me about 3 days to figure out what was happening. Lots and lots of "rm -r" going on to correct the issue. BackupPC works with the pooling and compressing of data. I don't think eBox 1.2's simplistic rdiff backup all of / does that.
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on January 15, 2010, 05:49:54 pm
GENERAL INFO

I have never had to edit the configuration files manually, but I have checked them for some reference points just to get a better understanding of what's going on. If you're an experienced user, it's worth your time to take a look at some just to know what's all in those files. I've even setup a new host one time using a copy/paste from the config file of another host. All config files have a .pl (perl) after them.

BackupPC has two methods of configuration. One is global, the other is host-specific. What you do is setup your global configuration by going to "Edit Config" in the "Server" area of the sidebar. You should edit it so the settings you will most-commonly have set are configured here. For instance, all my Windows machines have a C drive share, not all of them have a D drive share. The C drive share name is also different on all of those machines, but Windows luckily has a default C$ share name configured for me with the same exact permissions as the user-configured C drive share.

You can even configure settings for all dropdown options. I have both 'smb' and 'rsyncd' configured in the BackupPC CGI, but only 'smb' is enabled. When I want to setup a host with 'rsyncd', I can go to the host configuration file and change 'smb' to 'rsyncd' and the global settings for 'rsyncd' are also represented such as the RsyncShareName "Full-Drive".

WINDOWS MACHINES

Windows machines are setup with password-protected shares. I've tired with guest shares and without as well as with non-password-protected shares. Everything worked just fine. I backup Windows machines using the 'smb' option in BackupPC for ease-of-use. I tried setting up Cygwin in my sister's laptop and netbook so I could possibly backup her computer using the Internet, but I couldn't get it to work and didn't have enough time to figure out how before she left again.

As of right now, I'm still looking for a suitable way to backup machines over the Internet even if behind a NAT router. What I'd like to do is have something running on the machine itself figuring out how fast the upload speed is and then determining which files/folders to back up based on a priority list I've setup. The user should be able to choose which file/folders go where in the priority list via a Windows executable or from the BackupPC CGI, but I currently do not see this being a viable option right now as it does not exist, haha.

The best thing to do is setup some kind of SSH tunnel or setup a DDNS service and domain name so I can just backup public.mydomain.com via SSH in Cygwin or something. It's better than setting up a VPN because that would restrict Internet access of those machines to the network they are VPN'd to like my own at home which would definitely slow it down.

If any of your Windows machines have symlinks setup and they reference back to a parent directory, do not use Samba to back those up, use 'rsync' or 'rsyncd' or mount the share and run 'tar' on the local machine under directory /mnt/Windows-Machine-Name (I think that works).

LINUX MACHINES

Good tip, don't backup Linux machines with Samba because it follows symlinks and that leads to infinite loops because Linux is installed and used in the funkiest way. If you're primarily a Windows user, you'll know what I mean if someone explains to you how /dev, /proc, and /sys work. Another reason is that UNIX file attributes are also not backed up over 'smb'.

Linux machines can be backed up using 'rsync', but I set mine up with 'rsyncd'. I believe the only way to get the standard 'rsync' to work is by using SSH keys. I have one setup with one machine, but 'rsyncd' is supposed to be faster and better to use. Since one of my linux machines is connected internally using a wireless adapter, I went ahead and just setup both with the 'rsyncd' option.

My BackupPC version is 3.0.0. In this version, there seems to be a missing RsyncdUserName field when you are configuring individual hosts so I globally set that value. It can be whatever you want, doesn't even have to be a valid username. If you want to be even more secure, mix case because I believe it's case-sensitive. Leave the password field here blank.

What you'll need to do is create an rsyncd.conf and rsyncd.secrets file in /etc. This is the easiest setup because you can just copy/paste mine.

Let's start with the rsyncd.secrets file. For passwords, I recommend going to https://www.grc.com/passwords.htm as it's the best way to make a secure password. I highly recommend you do not make one up yourself. All you need in the file is this:
Code: [Select]
username:passwordWhere the username is the username you chose and the password you got from grc.com or a modified version of it such as:
Code: [Select]
bAcUppC:17EED4ABBD104F185A8E6ABD57AF4C0CD2776F486DB8289CDC9A732ABE913A65I don't know if there's a limit to password size. I shortened mine a bit (randomly) so no one knows how long the password is either. I was also afraid the password might be too long, but haven't seen the limit of the length yet. After you've saved the file, run these commands:
Code: [Select]
chmod 600 /etc/rsync.secrets
chgrp root /etc/rsync.secrets
chown root /etc/rsync.secrets
For good measure, I did that to the rsyncd.conf file as well which I recommend you do.

Here's an example of a BackupPC configuration file with a modified version for this example:
Code: [Select]
uid = 0
gid = 0
max connections = 2
read only = yes
list = yes
secrets file = /etc/rsyncd.secrets
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = daemon

[full-drive]
path = /
comment = Full Drive
auth users = bAcUppC
hosts allow = 1.1.1.1
hosts deny = *
read only = no
list = no

Code: [Select]
uid = 0
gid = 0
This stands for using root as the user and group giving this user full access rights. As another security precaution, you might also want to change the group and user id to:
Code: [Select]
uid = nobody
gid = nogroup
I have these both setup as 0 because I want to backup everything, but maybe you just wanna backup a few files which don't require root access. You can also, in effect, change those to another user's uid and gid on the machine you want to backup, but I haven't tried it that way and do not intend to.

The read only and list options are global settings in the config file which you can manually change by specifying shares as you can see in my example. I do not want to list the full-drive rsyncd share because that's a security risk. It's best if it's unidentifiable by any means.

The only thing here you really need to change is the hosts allow area. Set that to the IP(s) of your BackupPC server with a space between all the ones you want to allow. You can also put in fully domain names but that's requiring you to have a static IP from your ISP and a custom reverse DNS lookup for your domain. I don't know what else you can do with this configuration setting.

For security reasons, I would normally have:
Code: [Select]
read only = yesI changed that in my configuration file because I wanted to be able to restore files from BackupPC. Thing is, while Linux is case-sensitive, Windows is not so if I wanted to say, download a ZIP or Tar of a directory of files, if any of those files share the same name but have different case, I will be unable to transfer those mixed case files together over Samba to a Linux share, it will require me to pick one or the other. Since this caused me problems, leaving the full drive read/write is fine so long as my password and username aren't easily-identifiable.

When you've created both of those files and are fine with your settings, go ahead and run:
Code: [Select]
rsync --daemonAnd then open /etc/inetd.conf and add this anywhere in there (I put it at the very bottom):
Code: [Select]
rsync stream tcp nowait root /usr/bin/rsync rsyncd --daemonFeel free to comment the line above it so you know what it's for. You can verify if it's working by typing in
Code: [Select]
rsync hostname::It shouldn't display anything because it doesn't have anything to list (in my configuration). There's also no error though either; keep that in mind.

FILE EXCLUDES
If you have any computers being backed up from 'smb', in your global configuration, do not use both BackupFilesOnly and BackupFilesExclude. I use BackupFilesExclude and have no need for the BackupFilesOnly at the moment, but it should be either/or with 'smb'. If you do use both with a machine configured with 'smb', BackupFilesExclude is ignored. Here's the list of files I globally excluded:
Code: [Select]
pagefile.sys
hiberfil.sys
/proc
/mnt
*.tmp
/tmp
/media
There may be more I'll do later on, but that's all I need for now. If you want to exclude folders for machines using 'smb', make sure to use /* at the end such as /Windows/*.
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on January 15, 2010, 05:50:50 pm
STATUS SCREEN
http://badmarkup.com/ebox/backuppc-config-01a.png

What's going on here is a full backup of 1.1.1.1. For my Linux boxes, since the only things that really change are log files and since the 'rsyncd' method is so effective, I have them scheduled to do a full backup once a week. It takes a long time to do one so, to minimize network traffic, hard drive usage, and CPU cycles, I try to do this as rarely as it is needed. The incremental backup method actually does an amazing job and is extremely useful. In my opinion, I don't see why I'd need to do full backups very often at all.

One of my linux boxes is setup over wireless, and because of that, I'd normally want to minimize full backs and try to do more incrementals of incrementals especially if not much changes on the machine, but since all of my Linux rigs that aren't fileserves use very little space, doing a full backup weekly won't be very difficult to do. Backing up 8.5GB over LAN and 3.1GB over Wi-Fi really isn't going to hurt anyone especially if the interfaces aren't even being used for anything but backups. I might also add my Linux boxes are servers. I have no Linux desktops except for testing purposes.

This screen also shows if the trash collector or nightly jobs are running. Here's a breakdown of what's shown in:

You'll also notice that there are failures here needing attention. BackupPC was backing up my mom's laptop and she turned it off in the process. Since it was doing a scheduled full backup, that's gonna take quite some time to finish over wi-fi. When this happens, I have it configured to wait a maximum of 3 days before only saving this as a partial backup. If mom's laptop is turned on during a time when I have BackupPC wake up, then it'll begin to complete the full backup by doing an incremental backup of what's been partially backed up so far. I don't quite know how this works though.

When you see "inet connect: Connection refused", this comes from a failed 'rsyncd' backup because the machine didn't have the daemon running or was not accepting connections on that port. I got this error a few days ago when I tried to setup my sister's Windows laptop with Cygwin to be able to 'rsyncd' it from remote locations, BackupPC woke up and tried to back it up. It failed, and I didn't have enough time to figure out what was going on before she had to leave.

There are a few ways to get "tree connect failed: NT_STATUS_BAD_NETWORK_NAME", but they all stem from Samba issues (notice the NT in the error). The machine should be backed up using 'smb', but the last time it was turned on, I had Linux Mint 8 in there and no shares were available so it's complaining that it couldn't get in. I'm still working on finding a new board for this machine and this machine is the entire reason I finally reinstalled BackupPC. Funny how I don't have backups of it though.

NOTE OF WARNING eBox + BackupPC

If your drives are fast enough, you can canibalize your Internet connection by using BackupPC because it's going to suck in all the data it can while its doing backups which will cause network utilization on the Ethernet card to be higher than it normally would. I currently have mine on a RAID3 of three 640GB drives hooked into a hardware RAID card. Thankfully, the PCI bus limits the speed to around 100MB/s because of overhead which gives me enough leeway to have an Internet connection sustained in full. I know no way to manage this bandwidth.

SETTINGS & CONFIGURATIONS

I have globally set for machines to do full backups every couple months and incremental backups either daily or a few times a day. I then modified the full backup number for machines I want to backup more often like those on LAN connections. There are currently only 6 machines connected via LAN whereas there are 9 connected via WLAN, maybe more since I bring things on and off all the time. LAN machines I want to fully backup every 14 or so days and for WLAN, the global 2 months sounds good unless it's someone that changes a ton of files all the time.

When doing incremental backups, note that the 'smb' option is kind of a dumb setting in that it just looks for files that were modified. If you copy/pasted a file in and then renamed it to be the same name as another file but the modification date was older, it wouldn't be backed up; also, if a file was deleted, it won't notice. It's actually more beneficial to do full backups for 'smb' hosts than for 'rsyncd' hosts, but if you have people wirelessly connected, trust me, do incremental backups of incremental backups.

A neat setting I've configured is doing this round of incremental backs where a full backup is level 0 and an incremental is level 1 and an incremental of that incremental is level 2, I setup a system so it cycles backups like this after each full: 1, 2, 3, 1, 2, 3, 4, 5, 6. That ensures I have good variation and just about everything is backed up; it also speeds up the backup process since a level 6 backup will probably backup only a few files.

I limited the number of incremental backups, but I think I might change that eventually. I started saving only 6, then 9, and now 24. I can also set a minimum to keep. If you have incrementals of incrementals, you have to get rid of the highest levels first before you can get rid of the lower ones because there are dependencies needing to be met. My full backups stay, currently, for 365 days, but I might change that if I feel I need to. I'd rather like to be able to restore my MSOCache (Microsoft Office cache) folder when I need it and delete it afterwards rather than having 450MB of space on each machine taken up by what I'd consider junk files.

Technically, I never want to delete my backups, that's the whole point of this solution, but I'm not hard set on keeping all of them right now either because I want to play around with it a bit more and see if it slows down or how takes up a lot more space if I do keep all the old stuff.

Another thing about having Wi-Fi hosts is you'll need BackupPC to be a bit more persistent. I setup half-hour wakeups for BackupPC during times I know wireless hosts would be on, but not for long. I can set it to wake up in 15 minute intervals as well, but I'm not sure when things are gonna show up during those times either.

I have yet to test out backing up my networked printers so I really can't give any input on those as of yet nor do I know if it's even possible. The newest BackupPC, 3.2.0beta0 has ftp support which is helpful since some printers (like mine) also support FTP. I wonder if SFTP is included when it says FTP as an added feature.

HOST SUMMARY SCREEN
http://badmarkup.com/ebox/backuppc-config-02.png
http://badmarkup.com/ebox/backuppc-config-02a.png

What I'm showing here is a play-by-play. You can see BackupPC's Host Summary screen over the course of an hour and a half. Notice how not that much changes since I took the data during the early morning when only I'm awake.

1.1.4.1 is my laptop, and it was backed up with an incremental backup just recently. BackupPC shows that light green color when a machine was recently backed up, but I don't know if recent means "since the last wakeup" or "in the last hour".

At the top of the screen you'll see a very interesting readout of how much data those 10 hosts would be taking up had pooling and compression not been enabled. Compare 538.33 with the 127.83GB used by the compressed pool, and you start to realize how important BackupPC is to your network. I feel like I have a lot of power with BackupPC. I truely feel like I can backup just anything now and barely feel it on the machine which is mostly true. The compression rate seems to average around 65% meaning files are 65% lower in size and pooling usually eats up a lot of files as well.

Notice how much space just 12 full backups take up comparing to 80 incremental backups of varying levels. There's a reason there aren't many fully backups being done. You have to realize, each time it does a full backup, it's going to be adding quite a few gigs down the network pipe every time it runs. For 1.1.3.1 and 1.1.3.2, they'll both be due for a full backup here in a day or so which will be another 77GB to add to that already-large 368GB. Thankfully, with pooling and compression, that will probably not even add a gigabyte to the full size of the pool. I really wish it would just take a look at the SHA1 hashes of all the files in the pool against the SHA1 hash of the files in a Samba share and then choose to hardlink that file or download it if it's different, but that's not what's happening sadly.

You'll notice all the transfer speeds are really really slow. Besides 1.1.1.1 which has a ton of small files, the only other two LAN-connected devices are 1.1.3.1 and 1.1.3.2. For all other machines, the speed of Wi-Fi is going to max out around 3MB/s with all the right conditions so anything close to 2MB/s, especially with all the small files being transferred, is really darn good. The BackupPC is setup using gigabit speeds, but it's currently limited to ~10MB/s total read/write speeds, and I haven't figured out why yet so that's why those LAN speeds are as low as they are. Apparently 100mbit/s speed over gigabit using BackupPC is normal though as there's a huge overhead in doing all the file lookups and copies.

Color coding is pretty obvious:
If there are any other colors, I have not yet seen them or completely forgot to make a note of them.

There you go SamK :P.
Title: Re: Add-On: BackupPC request.
Post by: SamK on January 17, 2010, 01:20:41 pm
Nice to see you back. I was hoping to hear back from you SamK :P.
Kind of you to say so, although I have never really been away - just not posting.

You have written a couple of extensive posts here - did you get anything else done on that day? (Work for instance)

I haven't used BackupPC in the last couple of years so my comments may not reflect the current version.  For that matter they may not be entirely accurate as they depend on my ability to remember accurately.  At that time I was only interested in backing up non-eBox, Ubuntu servers to a central BackupPC machine. The LAN workstations were predominately Ubuntu (a few MS-Windows workstations).  User home dirs were located on a PDC (no storage of local data at all).  The BackupPC pooling feature was of secondary interest; the ability for users to conduct restores of single/multiple files via the web interface was a high priority.  Also of interest was the inbuilt ability of BackupPC to automatically create copies of the backups.  These were committed to encrypted, external disks for off-site storage.


...I think Javi saw one of my LDAP+BackupPC posts a long time ago and added a ticket request for it.
...the best way to integrate it is to just connect to the BackupPC CGI interface through the eBox BackupPC module (which doesn't exist), and then configure LDAP user association so you don't have to setup your users in two places. That would allow you to use both BackupPC and eBox together without having to go crazy and add in 100 users (larger network) with passwords and then have to manage changing passwords in two places instead of one.
In the above scenario, each user needed an account to be created in BackupPC as a requirement of them being able to restore their own data.  Authentication would usually be conducted by BackupPC.  If I recall correctly, applying the appropriate PAM module allowed authentication to be conducted by the PDC which used synchronised Unix and Samba accounts.  It is likely to be that a PAM moudule exists that works with LDAP.


ext4 is massively faster than ext3 in both initialization, filetransfer, and so many other things.
It sounds worth investigating - thanks.  How well is it supported by filesystem maintenance/recovery tools.  (From memory, the Ubuntu installer CD includes ext3 support tools for rescue situations.)


My tests did cover restores on both Windows and Linux machines, but only file and folder restores, not the OS.
If you do decide to conduct tests on restoring the OS it will be interesting to here the results.  Does the BackupPc documentation give guidance on this?

Your posts contain a lot of information that might be useful to the eBox community.  Extract it to a HowTo perhaps?
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on January 17, 2010, 02:03:32 pm
I only spent 6 hours writing those posts, no biggy. I really enjoyed doing it so it wasn't a hassle to me. That was my work. I've been wanting to tell somebody about how I setup BackupPC so this was great. I still to talk more about the host interfaces and some more specific topics of interest but I purposefully left them for another time.

The LDAP one is the one I'm hoping for the most since I haven't, and probably won't, create BackupPC users until 2 things are fulfilled, LDAP users sync'd with eBox users and HTTPS connections. Right now, it's only HTTP so I don't want access to it from the outside at all.

I've been shrinking my 1.2TB ext3 to a 160GB on a hardware RAID3 using Gparted on the Linux Mint 8 Live. It's been taking quite a bit of time; resize2fs has been running for 29 hours now. I feel like I should've made the partition larger to fit the 465GiB partition I'm moving it to, but I didn't do that because I wanted the dd to move faster. Well... Bad idea I guess.

I've already setup a RAID1 and RAID5 using mdadm and the SATA ports on the motherboard to prepare for the move. I'm moving to 3x250GB drive. I setup a 131MiB ext2 partition for /boot in RAID1 mode with all 3 drives and the rest is for / in RAID5. Had to do that or I couldn't boot to the RAID5. Just waiting for this resize2fs operation in Gparted to finish so I can dd the data. I thought I'd easily need some 300-400GB for BackupPC, but how I have it configured, I don't need that much at all. Now say I backup a lot of data I have in my RAID setups, sure, I could fill that sucker up but I'm not doing that for the reason that I don't have the space requirements or the finances to have the space requirements to do so since it'd easily require at least 2TB of storage capacity, maybe more. Don't forget the amount of bandwidth and computational resources and heat/power from backing up 1TB+ of data in a full backup. It still would have to look for modifications to files in those files as well adding a ton of disk reads adding network traffic and more heat/power.

BackupPC is a major player in the backup arena, but I don't think there's any solution for this massive of a data-set unless I have two pools or large enough drives. I really don't /need/ to backup terrabytes of data, but it would be nice yes. That's actually not a bad idea if I could get a hold of some more 640GB drives or some TB+ ones in the future. Keep that in mind, no backup solution is going to be really good at backing up many terrabytes of data unless you have a SAN and some fiber links.

I don't have time (motivation) to write a HowTo. If someone else wants to, I can give them all the info necessary that they'd probably ever need. Best to ask me sooner than later because I tend to forget stuff pretty quick that I don't use daily and then I'll remember everything as it comes back which is terribly out of order.
Title: Re: Add-On: BackupPC request.
Post by: SamK on January 17, 2010, 03:04:07 pm
The LDAP one is the one I'm hoping for the most since I haven't, and probably won't, create BackupPC users until 2 things are fulfilled, LDAP users sync'd with eBox users and HTTPS connections.
It looks like bad news; the Devs have decided not to integrate LDAP user authentication with BackupPC.
See http://trac.ebox-platform.com/ticket/1463
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on January 17, 2010, 03:24:50 pm
At least it says "we currently have a backup module based on duplicity, we could add this integration as a tip in the wiki if someone contributes it."

I have no clue what duplicity is but I know they're using that and rsync in 1.4.
Title: Re: Add-On: BackupPC request.
Post by: SamK on January 17, 2010, 04:54:51 pm
I have no clue what duplicity is but I know they're using that and rsync in 1.4.
Have a look here http://duplicity.nongnu.org

On the irc #ebox channel, foolano recently indicated that version 1.4 will allow backups to be stored on a remote LAN server or a web based one.  User conducted restores via a web interface will not be included but will be available in a later eBox release.  Disaster recovery will be catered for by a bootable CD that includes networking and duplicity.
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on January 17, 2010, 05:31:50 pm
Well that's cool they're doing that. Too bad I've grown very attached to BackupPC, the CGI, and the documentation. I wish I could actually help improve the documentation on the project because, in reading it over and over and over again, I've started to see what things mean and how to better write those sections.

I was pretty into the original eBox documentation, but it just wasn't informative enough for what I wanted. If they made the other one they were talking about I'd gladly help out if I needed to use it, haha. When I use something for a while I start noticing the flaws pretty quickly because my productivity becomes halted.

Best thing I can say is, eBox 1.4 isn't out and nor is the ability to allow users to control backups on their own so I can keep using BackupPC. Plus, I'll need to use BackupPC now because that's where all my backups are stored. I already have about 500GB of stuff I never want to delete so I think I'm good. It's nice there will be something because I sure as heck don't want to setup BackupPC again if I don't have to. It's not hard, it's just not for every setup and is a huge CPU hog. I think duplicity looks a lot lighter-weight and better for the sort of things eBox is supposed to be installed in.

I might try it out at least to see how it works and do a comparison.

What was that thing in the ticket about a tip in the wiki on how to do it if someone wrote it? I would love to make an eBox module anyone can download that adds that functionality.
Title: Re: Add-On: BackupPC request.
Post by: SamK on January 17, 2010, 06:02:16 pm
BackupPC offers the pooling feature to reduce the size of the backups, the eBox backup system will not do this.  If this important to you it is another good reason to continue its use.
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on January 17, 2010, 06:59:55 pm
That's an incredibly important reason as well as compression. If I installed eBox into a ZFS system, then the pooling isn't so necessary, but compression is extremely welcome as I'm getting anywhere from 32-65% compression.
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on January 17, 2010, 11:15:07 pm
http://badmarkup.com/gparted/ Have fun looking at that data. 39 hours and 8 minutes.

I've begun the dd operation. Seems to be doing well. Reading at ~31MB/s and writing at 15-17MB/s
Title: Re: Add-On: BackupPC request.
Post by: sixstone on January 19, 2010, 01:06:58 pm
Hi there,

As we indicated in the ticket, Saturn2888 you may write a howto integrating eBox LDAP and BackupPC in our wiki if that is enough for you o:).

Cheers,
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on January 19, 2010, 01:35:56 pm
If I knew how to use LDAP I certainly would :P. Anything I need to know before I start some research?
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on January 20, 2010, 09:23:13 am
Wow, if only I remembered to go back to this thread. I have an update. A dude, Tolaris, setup a repo with a backport of BackupPC 3.1.0 so I'm up-to-date. The upgrade was minor, but I like the additions like graphics on the Server Status page and pooled/compressed info on the Host Summary screen since it used to only say how much would be taken up if compression and pooling were off.

Under Edit Config > CGI, there are a number of options I apparently looked over, one neat one controls all the colors on the Host Summary screen as well as giving you a description of them all.

$CgiStatusHilightColor
Code: [Select]
Disabled_AllBackupsDisabled
Disabled_OnlyManualBackups
Reason_backup_canceled_by_user
Reason_backup_done
Reason_backup_failed
Reason_no_ping
Status_backup_in_progress

I only wish it allowed me to change the names of what those refer to since the only few that make sense to most are "backup in progress" and "idle".
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on February 03, 2010, 12:29:10 pm
I've been working on this and am yet unable to get this to work. Apparently BackupPC gains LDAP user access so long as Apache has access to LDAP users. Configuring Apache to use LDAP users isn't easy at all considering some features of LDAP in Apache are apparently missing in eBox. Maybe it'd be best if eBox configured this internally, which, I already thought it was doing.

Link to the new thread:
http://forum.ebox-platform.com/index.php?topic=2813
Title: Re: Add-On: BackupPC request.
Post by: ichat on March 08, 2010, 10:04:57 am
there is a better option that backupPC  - 

bacula,  Yes it requires a client installtion  (witch in my opinion is more a good than a bad thing,
this will for example handle a lot of processing of the backup client-side, rather than server-side,

this is good because now a cheaper,  server can service more clients,

remember that most  OS'sen  even windows can run  on just about any   Intel Pentium  4(r)   2+ ghz.  - where most people have    dualcore  (e4400 / athlon 4000+ x2's  )  so plenty to offload the file server. 

bacula can run in both mysql  or (in our case)   postgresql,   and seems to work well with  samba and ldap,

ther is however 1 downside,  -      it has a php /  rather than a perl    web-frontend,    this means that, someone has to port the php script to perl,

however,  php is not a real hard language to read,   so a skill perl programmer could probbably why up a module  for bacula quite fast..

ALL configuration for bacula is done in its data-base so its not raly hard to manage from ebox's point of view. 

bacula COULD and SHOULD be tightly intergrated as a default  module. 

the biggest problem of backula is however this:

how should one generate bacula config scripts for the clients.   

Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on March 08, 2010, 10:19:26 am
I've heard of people moving from BackupPC to Bacula, not the other way around. The Deduplication of data is the big hit for me because I'm storing 1.2TB of data into less than 220GB of space. BackupPC's biggest advantage is the ability to do server-side backups rather than client-side. The thing that slows it down the most is actually writing to disk and Ethernet packet issues, not the processor. Cheap fileservers don't ever max out the processor, they max out disk writing. BackupPC can also do backups close to client-side ones if you use Rsyncd which connects to an Rsyncd server on the cilent.

eBox's backup solution is separate though and not designed to replace anything else like Bacula or BackupPC. Storing configuration in a database is for of a preference and ldap support, which is extremely difficult to get working in Apache2 with eBox (for BackupPC, would be nice if it was needed for an Enterprise or medium-sized business.
Title: Re: Add-On: BackupPC request.
Post by: Saturn2888 on March 09, 2010, 07:35:00 am
I found what I was looking for, this guy gives a good reason to pull backups (BackupPC) rather than push them (Bacula) for security reasons: http://www.sanitarium.net/golug/rsync_backups_2010.html

Quote
Security: One of the reasons we have backups is because of the possibility of malicious activity (hackers, worms, trojans, etc). If your backup device is plugged into the computer being backed up then any malicious users or software that can destroy your data can also destroy your backups. Keeping your backups on a separate isolated server protects them from this possibility. Note that this is also why I prefer to pull backups from a script running on the backup server rather than pushing backups from a script running on the backup clients.
Title: Re: Add-On: BackupPC request.
Post by: ichat on August 16, 2010, 08:25:01 am
im sorry to say but i find this 'security' argument really 'thin' ...
its so thin that it will break before  you even truely concider it.

it sounds to me like:   using NAT (network addres translation)  for a firewall.
-----

i may have said it allready but here to say it again, backup is for desaster recovery, and nothing should be able to interfeer with it.  If that is even possible you did a verry poor job indead. 


now lets clearefy what i ment earlier,  - backup pc pulls data server side.
that has 2 downsides:

1.  to takes a whole lot of computting power, on the server.   
2.  it takes al lot of bandwidt... 



1:  even though it may seem rather unimportand,  but that is really a mistake.
backing up is more than.

serv> hé im the server, do you want a backup?
pc>    euuhhh
serv> well i think im just going to get all your data>
pc>    euhhhhh

the problem with this, is that the server's cpu will now have to do everything.
i/o  listing (of the filesystem),   getting the data (with no form of compression what so ever).
so as soon as you want to start pulling data,  no advanced technolegy features can/ will  be used.

but even when you finaly browsed the 'smb' sharess,  and you coppied over the data, you still want one more thing to  be done.  - (at least i would)..  that is, you want your data to be compressed a bit to save pressious dataspace on your server.   mind you!   compression takes a lot of computting power.

wouldn't it be better to do all of that clients side:

<server>  hee im the server, and its time to do a backup,
<client>  oh okey mr server, hang on im going to do a bit of work now,
<client>  here is the file mr. server i compressed it allready, so mr. router wont be mad this time its now only 1.3gb (instead of 10gb) 
----