Author Topic: Add-On: BackupPC request.  (Read 19176 times)

SamK

  • Zen Samurai
  • ****
  • Posts: 283
  • Karma: +3/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #15 on: July 23, 2009, 10:02:59 am »
...I don't wanna mess around with installing it until there's a newer one which will work with Samba 4.7.

The project is constantly being developed. I've always found the main developer very approachable, the user group is also very supportive and responsive.  It might be worth you getting in touch to see what the plans are.

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #16 on: July 23, 2009, 10:05:09 am »
Awesome. Then yeah. I have it installed actually now. I checked on another rig and it had Samba 3.2 or something. I dunno what was up with mine. I installed it and was so utterly confused I might frequent their IRC channel again sometime and hope someone's around. Since I'm moving to new hardware, that'll fix any issues I had before. I'm gonna just install everything fresh.

SamK

  • Zen Samurai
  • ****
  • Posts: 283
  • Karma: +3/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #17 on: July 23, 2009, 10:29:13 am »
This post also requests BackupPc and lists some of the primary benefits it offers.
http://forum.ebox-platform.com/index.php?topic=1279.0

It will be good to hear from the eBox devs with a more detailed view of the future plans/ideas for backups of:
  • Multiple eBoxes on a LAN
  • Single eBox
  • eBox configuration data

Also on the topic of disaster recovery of a single or multiple eBoxes

Javi how about an insight from a developer's perspective?
« Last Edit: July 23, 2009, 10:56:02 am by SamK »

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #18 on: July 23, 2009, 10:40:16 am »
Agreed.
What do you personally use BackupPC for? How many users do you run it on. Do you think my friend could have an RSync module installed on her Win7 laptop and have that setup an SSH tunnel to my machine using BackupPC/eBox and backup or sync certain folders from her computer in that way? If so, then it means eBox will finally have that functionality that everyone wants, the ability to backup to remote locations instead of just on the eBox itself. I could you could SSH mount a network location and have eBox backup to that directory too. Maybe. That's a bit of a hassle. Maybe having BackupPC running the backend of it might be easier since it would connect to eBox's LDAP users. That's probably the most important part I'd say, connection w/ eBox's LDAP users. If there was an API to connect to LDAP users, then there might be more add-on apps, but that's kind of out of the way I'd say, but it would definitely make eBox a platform people start developing for.

SamK

  • Zen Samurai
  • ****
  • Posts: 283
  • Karma: +3/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #19 on: July 23, 2009, 12:04:47 pm »
I no longer manage the LAN on which I implemented BackupPC, but from memory...

The LAN hosted various server machines each having various data stores, user home, group shares and such.  The primary objectives were
  • Backup these to a central resource
  • Enable each user to restore their own files
  • Enable group members to restore their files
  • Use a Web interface
  • Use PDC authentication of individual accounts
  • Be transparent to both Ubuntu and Windows users
There were of course others but it shows like a list of benefits of using BackupPC.

At the start I knew nothing of how to achieve these goals but with the help of the main developer Craig Barratt, the user group, and the project documentation, it was very successful.  It really is every bit as capable of securely backing up servers to a central resource as it is securely backing up PCs to the same resource.

With reference to backing up a Windows laptop via an SSH tunnel to a BackupPC server I have not used it this way but a quick search of the BackupPC wiki indicates it can be done. Have a look at
http://backuppc.wiki.sourceforge.net/CommonConfigurationIssues

I cannot overstate the helpfulness of the project and recommend you get in touch.
« Last Edit: July 23, 2009, 12:14:04 pm by SamK »

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #20 on: July 24, 2009, 07:53:07 am »
Think you could PM me some contact info? I really wanna try. I think if I do I might be able to help give the eBox devs at least my own post of what I'd like to see in a solution like this if it was integrated.

SamK

  • Zen Samurai
  • ****
  • Posts: 283
  • Karma: +3/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #21 on: July 24, 2009, 11:23:14 am »
Think you could PM me some contact info?
PM sent.

stephdl

  • Zen Apprentice
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
    • association rouegate des utilisateurs de logiciels libres
Re: Add-On: BackupPC request.
« Reply #22 on: August 15, 2009, 10:42:00 pm »
I use backuppc on sme-server (yeah it's the devil, it's RPM), but the sme use perl too.

http://wiki.contribs.org/BackupPC

i post this address if you want to ask somes questions to the developper.

for the moment on ebox, you can use backuppc, but you need to set manualy the user password.
 sudo htpasswd /etc/backuppc/htpasswd 'user'
Indeed it is interesting to create separate users to access for themselves to their backup, because the user base after the installation is the admin (backupc), so with all the rights.

also it would be more convenient if the management of users were included in ebox, certainly with LDAP authentication
« Last Edit: August 15, 2009, 11:04:31 pm by stephdl »

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #23 on: August 16, 2009, 12:02:46 am »
Yep. I'm in agreement with that.

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #24 on: January 12, 2010, 07:41:40 pm »
One of the other threads on this is: http://forum.ebox-platform.com/index.php?topic=1279.0

Hey all, I'm back with some news about BackupPC itself. While drdebian's original posting has a lot of good points in it, asking the eBox devs to integrate BackupPC into eBox is quite a bit of work doing something which really isn't needed.

For one, Ubuntu Hardy has BackupPC 3.0.0 in the repositories. Yes, it's blue, not green/orange, but it's still able to do its job. I've been working with it on my eBox machine for about three weeks now and nothing's been breaking. eBox and BackupPC work in tandem. Make sure to turn off the Backup module in eBox as well because it can use up inodes and is doing the job BackupPC can do. Also, the eBox backup module is backing up slash (/) into your specified location. Great for off-site backups, terrible for on-site backups since your backing up onto the machine itself when BackupPC can do that and include pooling data from all of your other machines.

While they work great side-by-side, some sort of native, out-of-the-box, support would be nice, but it's not something that's on the map right now. Think about all the other features needing tending to. It's not that easy to integrate BackupPC. It seems easy, maybe have the "Backup" area of the sidebar actually point to a working BackupPC install, but other than that, there's not more the devs should do until eBox has a bit more development under its belt. Like, that might be an awesome eBox 2.2 feature for instance.

BackupPC 3.2.0 and the community edition are still in development thankfully so it's not like the eBox devs have to take charge of the project for us.

My only quarrel in all of this is the multitude of supported filesystems in Karmic which Hardy is clearly lacking. I can't wait for eBox 1.4's release because it will allow me to run Ext4 without any headache and have BackupPC, and in effect eBox, be more efficient because of it. I was considering integrating a VM using FreeNAS so I could backup to ZFS, but I gave up on that because it would be too hard on this particular machine solely because of the load put on the machine to do such an inane task. Best to have two machines unless your server is powerful enough to handle ZFS from a VM and has hardware virtualization.
« Last Edit: January 12, 2010, 07:45:48 pm by Saturn2888 »

SamK

  • Zen Samurai
  • ****
  • Posts: 283
  • Karma: +3/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #25 on: January 15, 2010, 10:56:05 am »
I've been working with it on my eBox machine for about three weeks now and nothing's been breaking. eBox and BackupPC work in tandem.
This is really good news.  From previous conversations I know how long you have been looking to find a suitable backup system.

Would you be willing to provide a bit more information about your setup and the way in which you use BackupPC? (Perhaps even a quick 'n' dirty diagram of your rigs.)

Unfortunately, during the last few months, I have not been able to test any of the recent eBox developments. Have you compared the forthcoming (v 1.4) inbuilt eBox backup system with BackupPC?  If yes it would be great if you could be the first to provide some feedback on the feature sets. If no what are the particular BackupPC features you favour? 

Make sure to turn off the Backup module in eBox as well because it can use up inodes and is doing the job BackupPC can do.
I may have missed the point here, are you meaning that BackupPc can be used as a complete replacement for the eBox backup systems? (data, configuration files, and system files)

Did your tests cover restores?  This is an area which has given rise to requests in the forum for assistance when the eBox restore does not work as expected. I am vary of setting up eBox version 'X' and upgrading to version 'Y' by restoring backups made on 'X' to a newly created machine based on version 'Y'.

While they work great side-by-side, some sort of native, out-of-the-box, support would be nice... ... maybe have the "Backup" area of the sidebar actually point to a working BackupPC install,
This seems to be an excellent idea.

I can't wait for eBox 1.4's release because it will allow me to run Ext4 without any headache and have BackupPC, and in effect eBox, be more efficient because of it.
Again my ignorance is on display, how does ext4 improve the efficiency of eBox?

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #26 on: January 15, 2010, 12:36:15 pm »
Nice to see you back. I was hoping to hear back from you SamK :P.

While they work great side-by-side, some sort of native, out-of-the-box, support would be nice... ... maybe have the "Backup" area of the sidebar actually point to a working BackupPC install,
This seems to be an excellent idea.
If any devs see this, we're waiting for you :D! I think Javi saw one of my LDAP+BackupPC posts a long time ago and added a ticket request for it. I can't think of anyone else asking for it but me: http://trac.ebox-platform.com/ticket/1463. If you go through the "nice to have" tickets in the roadmap, you start to feel really loved as an eBox user. I see a lot of things that you and I had suggested in the past already implemented and used as well as things other people have said. The devs really do listen to what the users want probably more than what they want.

There's a lot of people wanting BackupPC integration, but the best way to integrate it is to just connect to the BackupPC CGI interface through the eBox BackupPC module (which doesn't exist), and then configure LDAP user association so you don't have to setup your users in two places. That would allow you to use both BackupPC and eBox together without having to go crazy and add in 100 users (larger network) with passwords and then have to manage changing passwords in two places instead of one.

A better way to integrate the two, like a feature you'd see later on, would be if the objects you setup in eBox such as "computer-name"--referencing DHCP entries such as 10.10.10.10 and 10.10.10.100 for WLAN and LAN adapters on the same machine--could be used in BackupPC. When you say host computer-name, eBox would handle the setting up of BackupPC hosts that way by telling it which IP(s) computer-name is.

I can't wait for eBox 1.4's release because it will allow me to run Ext4 without any headache and have BackupPC, and in effect eBox, be more efficient because of it.
Again my ignorance is on display, how does ext4 improve the efficiency of eBox?
ext4 is massively faster than ext3 in both initialization, filetransfer, and so many other things. It's also better suited for file and directory fragmentation can occur over time. BackupPC uses a ton of hardlinks to work and having ext3 is a dated approach. I've been looking at benchmarks all around seeing how filesystem performance affects how BackupPC works, and saw that jfs was the most reliably fast and xfs was getting the highest numbers; albeit, not all the time.

ext3, while reliable is slow slow slow especially wen my partions are starting to go over terrabytes of space. ext4 on the other hand is close to the speed of xfs and jfs but can recover from a filesystem fault. Think of what would be combining ext3 and jfs, you get ext4 which is both speed and reliability.

From my own testing, this is the current best way of running a Linux OS. 1.4 is supposed to support Jaunty and Karmic so that's why I bring up ext4 because I will definitely be upgrading the filesystem.

Make sure to turn off the Backup module in eBox as well because it can use up inodes and is doing the job BackupPC can do.
I may have missed the point here, are you meaning that BackupPc can be used as a complete replacement for the eBox backup systems? (data, configuration files, and system files)

Did your tests cover restores?  This is an area which has given rise to requests in the forum for assistance when the eBox restore does not work as expected. I am vary of setting up eBox version 'X' and upgrading to version 'Y' by restoring backups made on 'X' to a newly created machine based on version 'Y'.

My tests did cover restores on both Windows and Linux machines, but only file and folder restores, not the OS. I don't ever trust eBox's configuration and full backup module. Sure I've done some backups, but now I don't even bother. I used to try to backup at one point when testing 1.1, and it was broken in that version. Restores were successful when I did them in 1.0 and 1.1 because I had often broken something which caused DHCP to stop working.

I believe I posted those bug reports in the forums and in the IRC channel, but there was no documentation of why certain things like a space in a hostname would cause the DHCP module to not start after saving the settings. It was just little things, but restoring would fix them.

I don't know exactly what the configuration and full backup module does when you backup and restore so I really can't say if restoring from older versions works or not. When I setup my second eBox from a fresh 1.2 ISO instead of 1.0 -> 1.1 -> 1.2 like I had done before, I actually did all the configuration by hand and when the time comes, will just copy/paste my dhcpd.conf, /etc/apache2, and /etc/bind files and folders and be done. I just want to move those when it's time so I don't have to make changes in two places.

I've been working with it on my eBox machine for about three weeks now and nothing's been breaking. eBox and BackupPC work in tandem.
This is really good news.  From previous conversations I know how long you have been looking to find a suitable backup system.

Would you be willing to provide a bit more information about your setup and the way in which you use BackupPC? (Perhaps even a quick 'n' dirty diagram of your rigs.)

Unfortunately, during the last few months, I have not been able to test any of the recent eBox developments. Have you compared the forthcoming (v 1.4) inbuilt eBox backup system with BackupPC?  If yes it would be great if you could be the first to provide some feedback on the feature sets. If no what are the particular BackupPC features you favour? 

I always wanted to use BackupPC, but it was far too complicated to figure out. Since eBox was easy to configure, I always though some sort of integration of the two would lighten up the learning curve needed to sustain two separate managment systems. After around 6 months, my general knowledge of how to use Linux software has increased to the point that this was finally possible.

I highly doubt some people that get eBox are very well-versed in Linux though. The BackupPC documentation, while awesome, is extremely confusing as to how you should go about setting up hosts and what some examples might be. Once I got at least one computer setup, I finally figured out what I needed to do. I had it setup before, but I did it in the most confusing way that I decided to not longer want to use it.

Here's what you need to know about BackupPC and how it's helpful. When you first run it, try to limit it to backing up one or two machines at a time as it will be building the pool and possibly compressing data at the same time. Make sure to turn compression on level 3 (a good overall value) before you start backing up, or you'll have to go through a very complicated episode of trying to run the eBox compression module manually and then waiting some 15-200 hours for your pool to compress. Trust me, you will want the compression, and it won't kill your CPU provided you aren't starting out backing up more than 2 hosts at a time when you start out.

My next post will include a full-description of using BackupPC.

I haven't done anything with 1.3 or 1.4. I actually have the line commented out in sources.list, just ready and waiting for me to enable it. I have a new eBox which is my BackupPC server right now. I'll eventually make that my main eBox machine and move my current eBox as the test machine which will test 1.3 and 1.4 and then 2.0 beta or whatever. I want that one to be the guinnea pig as it's been all this time.

The Backup module in eBox, the one that does automatic backups, is really terrible to use if you want to backup to the machine you're on. I figure it's probably a bug, but don't know. I thought backing up eBox would backup just eBox folders. This thing backs up all of /. If you backup / to itself, that means you'll be using twice as much space as you normally would be. This happened on my 1.2TB drive and oh man, that wasn't fun. Took me about 3 days to figure out what was happening. Lots and lots of "rm -r" going on to correct the issue. BackupPC works with the pooling and compressing of data. I don't think eBox 1.2's simplistic rdiff backup all of / does that.
« Last Edit: January 15, 2010, 12:38:15 pm by Saturn2888 »

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #27 on: January 15, 2010, 05:49:54 pm »
GENERAL INFO

I have never had to edit the configuration files manually, but I have checked them for some reference points just to get a better understanding of what's going on. If you're an experienced user, it's worth your time to take a look at some just to know what's all in those files. I've even setup a new host one time using a copy/paste from the config file of another host. All config files have a .pl (perl) after them.

BackupPC has two methods of configuration. One is global, the other is host-specific. What you do is setup your global configuration by going to "Edit Config" in the "Server" area of the sidebar. You should edit it so the settings you will most-commonly have set are configured here. For instance, all my Windows machines have a C drive share, not all of them have a D drive share. The C drive share name is also different on all of those machines, but Windows luckily has a default C$ share name configured for me with the same exact permissions as the user-configured C drive share.

You can even configure settings for all dropdown options. I have both 'smb' and 'rsyncd' configured in the BackupPC CGI, but only 'smb' is enabled. When I want to setup a host with 'rsyncd', I can go to the host configuration file and change 'smb' to 'rsyncd' and the global settings for 'rsyncd' are also represented such as the RsyncShareName "Full-Drive".

WINDOWS MACHINES

Windows machines are setup with password-protected shares. I've tired with guest shares and without as well as with non-password-protected shares. Everything worked just fine. I backup Windows machines using the 'smb' option in BackupPC for ease-of-use. I tried setting up Cygwin in my sister's laptop and netbook so I could possibly backup her computer using the Internet, but I couldn't get it to work and didn't have enough time to figure out how before she left again.

As of right now, I'm still looking for a suitable way to backup machines over the Internet even if behind a NAT router. What I'd like to do is have something running on the machine itself figuring out how fast the upload speed is and then determining which files/folders to back up based on a priority list I've setup. The user should be able to choose which file/folders go where in the priority list via a Windows executable or from the BackupPC CGI, but I currently do not see this being a viable option right now as it does not exist, haha.

The best thing to do is setup some kind of SSH tunnel or setup a DDNS service and domain name so I can just backup public.mydomain.com via SSH in Cygwin or something. It's better than setting up a VPN because that would restrict Internet access of those machines to the network they are VPN'd to like my own at home which would definitely slow it down.

If any of your Windows machines have symlinks setup and they reference back to a parent directory, do not use Samba to back those up, use 'rsync' or 'rsyncd' or mount the share and run 'tar' on the local machine under directory /mnt/Windows-Machine-Name (I think that works).

LINUX MACHINES

Good tip, don't backup Linux machines with Samba because it follows symlinks and that leads to infinite loops because Linux is installed and used in the funkiest way. If you're primarily a Windows user, you'll know what I mean if someone explains to you how /dev, /proc, and /sys work. Another reason is that UNIX file attributes are also not backed up over 'smb'.

Linux machines can be backed up using 'rsync', but I set mine up with 'rsyncd'. I believe the only way to get the standard 'rsync' to work is by using SSH keys. I have one setup with one machine, but 'rsyncd' is supposed to be faster and better to use. Since one of my linux machines is connected internally using a wireless adapter, I went ahead and just setup both with the 'rsyncd' option.

My BackupPC version is 3.0.0. In this version, there seems to be a missing RsyncdUserName field when you are configuring individual hosts so I globally set that value. It can be whatever you want, doesn't even have to be a valid username. If you want to be even more secure, mix case because I believe it's case-sensitive. Leave the password field here blank.

What you'll need to do is create an rsyncd.conf and rsyncd.secrets file in /etc. This is the easiest setup because you can just copy/paste mine.

Let's start with the rsyncd.secrets file. For passwords, I recommend going to https://www.grc.com/passwords.htm as it's the best way to make a secure password. I highly recommend you do not make one up yourself. All you need in the file is this:
Code: [Select]
username:passwordWhere the username is the username you chose and the password you got from grc.com or a modified version of it such as:
Code: [Select]
bAcUppC:17EED4ABBD104F185A8E6ABD57AF4C0CD2776F486DB8289CDC9A732ABE913A65I don't know if there's a limit to password size. I shortened mine a bit (randomly) so no one knows how long the password is either. I was also afraid the password might be too long, but haven't seen the limit of the length yet. After you've saved the file, run these commands:
Code: [Select]
chmod 600 /etc/rsync.secrets
chgrp root /etc/rsync.secrets
chown root /etc/rsync.secrets
For good measure, I did that to the rsyncd.conf file as well which I recommend you do.

Here's an example of a BackupPC configuration file with a modified version for this example:
Code: [Select]
uid = 0
gid = 0
max connections = 2
read only = yes
list = yes
secrets file = /etc/rsyncd.secrets
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
syslog facility = daemon

[full-drive]
path = /
comment = Full Drive
auth users = bAcUppC
hosts allow = 1.1.1.1
hosts deny = *
read only = no
list = no

Code: [Select]
uid = 0
gid = 0
This stands for using root as the user and group giving this user full access rights. As another security precaution, you might also want to change the group and user id to:
Code: [Select]
uid = nobody
gid = nogroup
I have these both setup as 0 because I want to backup everything, but maybe you just wanna backup a few files which don't require root access. You can also, in effect, change those to another user's uid and gid on the machine you want to backup, but I haven't tried it that way and do not intend to.

The read only and list options are global settings in the config file which you can manually change by specifying shares as you can see in my example. I do not want to list the full-drive rsyncd share because that's a security risk. It's best if it's unidentifiable by any means.

The only thing here you really need to change is the hosts allow area. Set that to the IP(s) of your BackupPC server with a space between all the ones you want to allow. You can also put in fully domain names but that's requiring you to have a static IP from your ISP and a custom reverse DNS lookup for your domain. I don't know what else you can do with this configuration setting.

For security reasons, I would normally have:
Code: [Select]
read only = yesI changed that in my configuration file because I wanted to be able to restore files from BackupPC. Thing is, while Linux is case-sensitive, Windows is not so if I wanted to say, download a ZIP or Tar of a directory of files, if any of those files share the same name but have different case, I will be unable to transfer those mixed case files together over Samba to a Linux share, it will require me to pick one or the other. Since this caused me problems, leaving the full drive read/write is fine so long as my password and username aren't easily-identifiable.

When you've created both of those files and are fine with your settings, go ahead and run:
Code: [Select]
rsync --daemonAnd then open /etc/inetd.conf and add this anywhere in there (I put it at the very bottom):
Code: [Select]
rsync stream tcp nowait root /usr/bin/rsync rsyncd --daemonFeel free to comment the line above it so you know what it's for. You can verify if it's working by typing in
Code: [Select]
rsync hostname::It shouldn't display anything because it doesn't have anything to list (in my configuration). There's also no error though either; keep that in mind.

FILE EXCLUDES
If you have any computers being backed up from 'smb', in your global configuration, do not use both BackupFilesOnly and BackupFilesExclude. I use BackupFilesExclude and have no need for the BackupFilesOnly at the moment, but it should be either/or with 'smb'. If you do use both with a machine configured with 'smb', BackupFilesExclude is ignored. Here's the list of files I globally excluded:
Code: [Select]
pagefile.sys
hiberfil.sys
/proc
/mnt
*.tmp
/tmp
/media
There may be more I'll do later on, but that's all I need for now. If you want to exclude folders for machines using 'smb', make sure to use /* at the end such as /Windows/*.
« Last Edit: January 21, 2010, 06:12:24 pm by Saturn2888 »

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #28 on: January 15, 2010, 05:50:50 pm »
STATUS SCREEN
http://badmarkup.com/ebox/backuppc-config-01a.png

What's going on here is a full backup of 1.1.1.1. For my Linux boxes, since the only things that really change are log files and since the 'rsyncd' method is so effective, I have them scheduled to do a full backup once a week. It takes a long time to do one so, to minimize network traffic, hard drive usage, and CPU cycles, I try to do this as rarely as it is needed. The incremental backup method actually does an amazing job and is extremely useful. In my opinion, I don't see why I'd need to do full backups very often at all.

One of my linux boxes is setup over wireless, and because of that, I'd normally want to minimize full backs and try to do more incrementals of incrementals especially if not much changes on the machine, but since all of my Linux rigs that aren't fileserves use very little space, doing a full backup weekly won't be very difficult to do. Backing up 8.5GB over LAN and 3.1GB over Wi-Fi really isn't going to hurt anyone especially if the interfaces aren't even being used for anything but backups. I might also add my Linux boxes are servers. I have no Linux desktops except for testing purposes.

This screen also shows if the trash collector or nightly jobs are running. Here's a breakdown of what's shown in:
  • Process ID in case you need to kill it.
  • How many backups did not happen last time BackupPC woke up because it couldn't back those up as per the config file's restrictions. I have never had this number change from 0 because I purposefully spread out my backups.
  • Users, as opposed to admins, can initiate backup requests, and you can even specify the maximum able to be done in this way. Any user backups past your restriction would be pending user backup requests.
  • I don't know what pending command requests are yet.
  • The entire size of the pool and the amount of space it's taking up on the drive. I believe this number is actually smaller than the size of the BackupPC folder itself (by a few gigs), but I do not know why.
  • Since the whole point of the pool is to pool files together as one, pool hashing is going to show you how many of your files there are hardlinks to (virtual copies) and what file has the most hardlinks chaining from it.
  • Nightly cleanup happens every night. There are a number of ways to configure it and it such that you can align it to your BackupPC server's system specs. When nightly cleanup runs, nothing else does so if you have a few terrabytes of a pool, you might want to run the nightly backup over a few nights so that sections 0-127 and 128-255 (of 256) happen every other night. You can even break it up into smaller bits like 1/4th and 1/16th. Because my system hasn't has very many issues running it, I run 4 concurrent nightly cleanup processes all at once which finish in about 20 minutes give or take. For some reason, there were no files cleaned up, but it's been around 250-350MB for the last few weeks.
  • The most important part to look at is the filesystem's freespace. What this section shows you is the amount of total space used up on the partition that the pool is also on whether this be / or /mnt/BackupPC. It shows you three data points: right now, before nightly cleanup (mine runs at 1am), and the highest amount of used space from the day before. I utilize this part the most to see what's going on on the system. I was also using this system as a fileshare but will be changing it to eBox + BackupPC server and that's it. That will give me the ability to backup more often as that'll be one of the system's only responsibilities.

You'll also notice that there are failures here needing attention. BackupPC was backing up my mom's laptop and she turned it off in the process. Since it was doing a scheduled full backup, that's gonna take quite some time to finish over wi-fi. When this happens, I have it configured to wait a maximum of 3 days before only saving this as a partial backup. If mom's laptop is turned on during a time when I have BackupPC wake up, then it'll begin to complete the full backup by doing an incremental backup of what's been partially backed up so far. I don't quite know how this works though.

When you see "inet connect: Connection refused", this comes from a failed 'rsyncd' backup because the machine didn't have the daemon running or was not accepting connections on that port. I got this error a few days ago when I tried to setup my sister's Windows laptop with Cygwin to be able to 'rsyncd' it from remote locations, BackupPC woke up and tried to back it up. It failed, and I didn't have enough time to figure out what was going on before she had to leave.

There are a few ways to get "tree connect failed: NT_STATUS_BAD_NETWORK_NAME", but they all stem from Samba issues (notice the NT in the error). The machine should be backed up using 'smb', but the last time it was turned on, I had Linux Mint 8 in there and no shares were available so it's complaining that it couldn't get in. I'm still working on finding a new board for this machine and this machine is the entire reason I finally reinstalled BackupPC. Funny how I don't have backups of it though.

NOTE OF WARNING eBox + BackupPC

If your drives are fast enough, you can canibalize your Internet connection by using BackupPC because it's going to suck in all the data it can while its doing backups which will cause network utilization on the Ethernet card to be higher than it normally would. I currently have mine on a RAID3 of three 640GB drives hooked into a hardware RAID card. Thankfully, the PCI bus limits the speed to around 100MB/s because of overhead which gives me enough leeway to have an Internet connection sustained in full. I know no way to manage this bandwidth.

SETTINGS & CONFIGURATIONS

I have globally set for machines to do full backups every couple months and incremental backups either daily or a few times a day. I then modified the full backup number for machines I want to backup more often like those on LAN connections. There are currently only 6 machines connected via LAN whereas there are 9 connected via WLAN, maybe more since I bring things on and off all the time. LAN machines I want to fully backup every 14 or so days and for WLAN, the global 2 months sounds good unless it's someone that changes a ton of files all the time.

When doing incremental backups, note that the 'smb' option is kind of a dumb setting in that it just looks for files that were modified. If you copy/pasted a file in and then renamed it to be the same name as another file but the modification date was older, it wouldn't be backed up; also, if a file was deleted, it won't notice. It's actually more beneficial to do full backups for 'smb' hosts than for 'rsyncd' hosts, but if you have people wirelessly connected, trust me, do incremental backups of incremental backups.

A neat setting I've configured is doing this round of incremental backs where a full backup is level 0 and an incremental is level 1 and an incremental of that incremental is level 2, I setup a system so it cycles backups like this after each full: 1, 2, 3, 1, 2, 3, 4, 5, 6. That ensures I have good variation and just about everything is backed up; it also speeds up the backup process since a level 6 backup will probably backup only a few files.

I limited the number of incremental backups, but I think I might change that eventually. I started saving only 6, then 9, and now 24. I can also set a minimum to keep. If you have incrementals of incrementals, you have to get rid of the highest levels first before you can get rid of the lower ones because there are dependencies needing to be met. My full backups stay, currently, for 365 days, but I might change that if I feel I need to. I'd rather like to be able to restore my MSOCache (Microsoft Office cache) folder when I need it and delete it afterwards rather than having 450MB of space on each machine taken up by what I'd consider junk files.

Technically, I never want to delete my backups, that's the whole point of this solution, but I'm not hard set on keeping all of them right now either because I want to play around with it a bit more and see if it slows down or how takes up a lot more space if I do keep all the old stuff.

Another thing about having Wi-Fi hosts is you'll need BackupPC to be a bit more persistent. I setup half-hour wakeups for BackupPC during times I know wireless hosts would be on, but not for long. I can set it to wake up in 15 minute intervals as well, but I'm not sure when things are gonna show up during those times either.

I have yet to test out backing up my networked printers so I really can't give any input on those as of yet nor do I know if it's even possible. The newest BackupPC, 3.2.0beta0 has ftp support which is helpful since some printers (like mine) also support FTP. I wonder if SFTP is included when it says FTP as an added feature.

HOST SUMMARY SCREEN
http://badmarkup.com/ebox/backuppc-config-02.png
http://badmarkup.com/ebox/backuppc-config-02a.png

What I'm showing here is a play-by-play. You can see BackupPC's Host Summary screen over the course of an hour and a half. Notice how not that much changes since I took the data during the early morning when only I'm awake.

1.1.4.1 is my laptop, and it was backed up with an incremental backup just recently. BackupPC shows that light green color when a machine was recently backed up, but I don't know if recent means "since the last wakeup" or "in the last hour".

At the top of the screen you'll see a very interesting readout of how much data those 10 hosts would be taking up had pooling and compression not been enabled. Compare 538.33 with the 127.83GB used by the compressed pool, and you start to realize how important BackupPC is to your network. I feel like I have a lot of power with BackupPC. I truely feel like I can backup just anything now and barely feel it on the machine which is mostly true. The compression rate seems to average around 65% meaning files are 65% lower in size and pooling usually eats up a lot of files as well.

Notice how much space just 12 full backups take up comparing to 80 incremental backups of varying levels. There's a reason there aren't many fully backups being done. You have to realize, each time it does a full backup, it's going to be adding quite a few gigs down the network pipe every time it runs. For 1.1.3.1 and 1.1.3.2, they'll both be due for a full backup here in a day or so which will be another 77GB to add to that already-large 368GB. Thankfully, with pooling and compression, that will probably not even add a gigabyte to the full size of the pool. I really wish it would just take a look at the SHA1 hashes of all the files in the pool against the SHA1 hash of the files in a Samba share and then choose to hardlink that file or download it if it's different, but that's not what's happening sadly.

You'll notice all the transfer speeds are really really slow. Besides 1.1.1.1 which has a ton of small files, the only other two LAN-connected devices are 1.1.3.1 and 1.1.3.2. For all other machines, the speed of Wi-Fi is going to max out around 3MB/s with all the right conditions so anything close to 2MB/s, especially with all the small files being transferred, is really darn good. The BackupPC is setup using gigabit speeds, but it's currently limited to ~10MB/s total read/write speeds, and I haven't figured out why yet so that's why those LAN speeds are as low as they are. Apparently 100mbit/s speed over gigabit using BackupPC is normal though as there's a huge overhead in doing all the file lookups and copies.

Color coding is pretty obvious:
  • Dark Green = backup in progress
  • Light Green = recently backed up
  • Orange = Backup stopped by user or admin
  • Yellow = Machine wasn't online when BackupPC wanted to back it up and hasn't been online since
  • Red = Backup failed (usually because of a disconnect)
  • White = idle
If there are any other colors, I have not yet seen them or completely forgot to make a note of them.

There you go SamK :P.
« Last Edit: January 15, 2010, 06:04:24 pm by Saturn2888 »

SamK

  • Zen Samurai
  • ****
  • Posts: 283
  • Karma: +3/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #29 on: January 17, 2010, 01:20:41 pm »
Nice to see you back. I was hoping to hear back from you SamK :P.
Kind of you to say so, although I have never really been away - just not posting.

You have written a couple of extensive posts here - did you get anything else done on that day? (Work for instance)

I haven't used BackupPC in the last couple of years so my comments may not reflect the current version.  For that matter they may not be entirely accurate as they depend on my ability to remember accurately.  At that time I was only interested in backing up non-eBox, Ubuntu servers to a central BackupPC machine. The LAN workstations were predominately Ubuntu (a few MS-Windows workstations).  User home dirs were located on a PDC (no storage of local data at all).  The BackupPC pooling feature was of secondary interest; the ability for users to conduct restores of single/multiple files via the web interface was a high priority.  Also of interest was the inbuilt ability of BackupPC to automatically create copies of the backups.  These were committed to encrypted, external disks for off-site storage.


...I think Javi saw one of my LDAP+BackupPC posts a long time ago and added a ticket request for it.
...the best way to integrate it is to just connect to the BackupPC CGI interface through the eBox BackupPC module (which doesn't exist), and then configure LDAP user association so you don't have to setup your users in two places. That would allow you to use both BackupPC and eBox together without having to go crazy and add in 100 users (larger network) with passwords and then have to manage changing passwords in two places instead of one.
In the above scenario, each user needed an account to be created in BackupPC as a requirement of them being able to restore their own data.  Authentication would usually be conducted by BackupPC.  If I recall correctly, applying the appropriate PAM module allowed authentication to be conducted by the PDC which used synchronised Unix and Samba accounts.  It is likely to be that a PAM moudule exists that works with LDAP.


ext4 is massively faster than ext3 in both initialization, filetransfer, and so many other things.
It sounds worth investigating - thanks.  How well is it supported by filesystem maintenance/recovery tools.  (From memory, the Ubuntu installer CD includes ext3 support tools for rescue situations.)


My tests did cover restores on both Windows and Linux machines, but only file and folder restores, not the OS.
If you do decide to conduct tests on restoring the OS it will be interesting to here the results.  Does the BackupPc documentation give guidance on this?

Your posts contain a lot of information that might be useful to the eBox community.  Extract it to a HowTo perhaps?