Author Topic: Add-On: BackupPC request.  (Read 19186 times)

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #30 on: January 17, 2010, 02:03:32 pm »
I only spent 6 hours writing those posts, no biggy. I really enjoyed doing it so it wasn't a hassle to me. That was my work. I've been wanting to tell somebody about how I setup BackupPC so this was great. I still to talk more about the host interfaces and some more specific topics of interest but I purposefully left them for another time.

The LDAP one is the one I'm hoping for the most since I haven't, and probably won't, create BackupPC users until 2 things are fulfilled, LDAP users sync'd with eBox users and HTTPS connections. Right now, it's only HTTP so I don't want access to it from the outside at all.

I've been shrinking my 1.2TB ext3 to a 160GB on a hardware RAID3 using Gparted on the Linux Mint 8 Live. It's been taking quite a bit of time; resize2fs has been running for 29 hours now. I feel like I should've made the partition larger to fit the 465GiB partition I'm moving it to, but I didn't do that because I wanted the dd to move faster. Well... Bad idea I guess.

I've already setup a RAID1 and RAID5 using mdadm and the SATA ports on the motherboard to prepare for the move. I'm moving to 3x250GB drive. I setup a 131MiB ext2 partition for /boot in RAID1 mode with all 3 drives and the rest is for / in RAID5. Had to do that or I couldn't boot to the RAID5. Just waiting for this resize2fs operation in Gparted to finish so I can dd the data. I thought I'd easily need some 300-400GB for BackupPC, but how I have it configured, I don't need that much at all. Now say I backup a lot of data I have in my RAID setups, sure, I could fill that sucker up but I'm not doing that for the reason that I don't have the space requirements or the finances to have the space requirements to do so since it'd easily require at least 2TB of storage capacity, maybe more. Don't forget the amount of bandwidth and computational resources and heat/power from backing up 1TB+ of data in a full backup. It still would have to look for modifications to files in those files as well adding a ton of disk reads adding network traffic and more heat/power.

BackupPC is a major player in the backup arena, but I don't think there's any solution for this massive of a data-set unless I have two pools or large enough drives. I really don't /need/ to backup terrabytes of data, but it would be nice yes. That's actually not a bad idea if I could get a hold of some more 640GB drives or some TB+ ones in the future. Keep that in mind, no backup solution is going to be really good at backing up many terrabytes of data unless you have a SAN and some fiber links.

I don't have time (motivation) to write a HowTo. If someone else wants to, I can give them all the info necessary that they'd probably ever need. Best to ask me sooner than later because I tend to forget stuff pretty quick that I don't use daily and then I'll remember everything as it comes back which is terribly out of order.
« Last Edit: January 17, 2010, 02:27:17 pm by Saturn2888 »

SamK

  • Zen Samurai
  • ****
  • Posts: 283
  • Karma: +3/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #31 on: January 17, 2010, 03:04:07 pm »
The LDAP one is the one I'm hoping for the most since I haven't, and probably won't, create BackupPC users until 2 things are fulfilled, LDAP users sync'd with eBox users and HTTPS connections.
It looks like bad news; the Devs have decided not to integrate LDAP user authentication with BackupPC.
See http://trac.ebox-platform.com/ticket/1463

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #32 on: January 17, 2010, 03:24:50 pm »
At least it says "we currently have a backup module based on duplicity, we could add this integration as a tip in the wiki if someone contributes it."

I have no clue what duplicity is but I know they're using that and rsync in 1.4.

SamK

  • Zen Samurai
  • ****
  • Posts: 283
  • Karma: +3/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #33 on: January 17, 2010, 04:54:51 pm »
I have no clue what duplicity is but I know they're using that and rsync in 1.4.
Have a look here http://duplicity.nongnu.org

On the irc #ebox channel, foolano recently indicated that version 1.4 will allow backups to be stored on a remote LAN server or a web based one.  User conducted restores via a web interface will not be included but will be available in a later eBox release.  Disaster recovery will be catered for by a bootable CD that includes networking and duplicity.

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #34 on: January 17, 2010, 05:31:50 pm »
Well that's cool they're doing that. Too bad I've grown very attached to BackupPC, the CGI, and the documentation. I wish I could actually help improve the documentation on the project because, in reading it over and over and over again, I've started to see what things mean and how to better write those sections.

I was pretty into the original eBox documentation, but it just wasn't informative enough for what I wanted. If they made the other one they were talking about I'd gladly help out if I needed to use it, haha. When I use something for a while I start noticing the flaws pretty quickly because my productivity becomes halted.

Best thing I can say is, eBox 1.4 isn't out and nor is the ability to allow users to control backups on their own so I can keep using BackupPC. Plus, I'll need to use BackupPC now because that's where all my backups are stored. I already have about 500GB of stuff I never want to delete so I think I'm good. It's nice there will be something because I sure as heck don't want to setup BackupPC again if I don't have to. It's not hard, it's just not for every setup and is a huge CPU hog. I think duplicity looks a lot lighter-weight and better for the sort of things eBox is supposed to be installed in.

I might try it out at least to see how it works and do a comparison.

What was that thing in the ticket about a tip in the wiki on how to do it if someone wrote it? I would love to make an eBox module anyone can download that adds that functionality.

SamK

  • Zen Samurai
  • ****
  • Posts: 283
  • Karma: +3/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #35 on: January 17, 2010, 06:02:16 pm »
BackupPC offers the pooling feature to reduce the size of the backups, the eBox backup system will not do this.  If this important to you it is another good reason to continue its use.

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #36 on: January 17, 2010, 06:59:55 pm »
That's an incredibly important reason as well as compression. If I installed eBox into a ZFS system, then the pooling isn't so necessary, but compression is extremely welcome as I'm getting anywhere from 32-65% compression.

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #37 on: January 17, 2010, 11:15:07 pm »
http://badmarkup.com/gparted/ Have fun looking at that data. 39 hours and 8 minutes.

I've begun the dd operation. Seems to be doing well. Reading at ~31MB/s and writing at 15-17MB/s

sixstone

  • Zentyal Staff
  • Zen Hero
  • *****
  • Posts: 1417
  • Karma: +26/-0
    • View Profile
    • Sixstone's blog
Re: Add-On: BackupPC request.
« Reply #38 on: January 19, 2010, 01:06:58 pm »
Hi there,

As we indicated in the ticket, Saturn2888 you may write a howto integrating eBox LDAP and BackupPC in our wiki if that is enough for you o:).

Cheers,
My secret is my silence...

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #39 on: January 19, 2010, 01:35:56 pm »
If I knew how to use LDAP I certainly would :P. Anything I need to know before I start some research?

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #40 on: January 20, 2010, 09:23:13 am »
Wow, if only I remembered to go back to this thread. I have an update. A dude, Tolaris, setup a repo with a backport of BackupPC 3.1.0 so I'm up-to-date. The upgrade was minor, but I like the additions like graphics on the Server Status page and pooled/compressed info on the Host Summary screen since it used to only say how much would be taken up if compression and pooling were off.

Under Edit Config > CGI, there are a number of options I apparently looked over, one neat one controls all the colors on the Host Summary screen as well as giving you a description of them all.

$CgiStatusHilightColor
Code: [Select]
Disabled_AllBackupsDisabled
Disabled_OnlyManualBackups
Reason_backup_canceled_by_user
Reason_backup_done
Reason_backup_failed
Reason_no_ping
Status_backup_in_progress

I only wish it allowed me to change the names of what those refer to since the only few that make sense to most are "backup in progress" and "idle".

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #41 on: February 03, 2010, 12:29:10 pm »
I've been working on this and am yet unable to get this to work. Apparently BackupPC gains LDAP user access so long as Apache has access to LDAP users. Configuring Apache to use LDAP users isn't easy at all considering some features of LDAP in Apache are apparently missing in eBox. Maybe it'd be best if eBox configured this internally, which, I already thought it was doing.

Link to the new thread:
http://forum.ebox-platform.com/index.php?topic=2813

ichat

  • Zen Hero
  • *****
  • Posts: 795
  • Karma: +28/-16
  • RTFM!
    • View Profile
Re: Add-On: BackupPC request.
« Reply #42 on: March 08, 2010, 10:04:57 am »
there is a better option that backupPC  - 

bacula,  Yes it requires a client installtion  (witch in my opinion is more a good than a bad thing,
this will for example handle a lot of processing of the backup client-side, rather than server-side,

this is good because now a cheaper,  server can service more clients,

remember that most  OS'sen  even windows can run  on just about any   Intel Pentium  4(r)   2+ ghz.  - where most people have    dualcore  (e4400 / athlon 4000+ x2's  )  so plenty to offload the file server. 

bacula can run in both mysql  or (in our case)   postgresql,   and seems to work well with  samba and ldap,

ther is however 1 downside,  -      it has a php /  rather than a perl    web-frontend,    this means that, someone has to port the php script to perl,

however,  php is not a real hard language to read,   so a skill perl programmer could probbably why up a module  for bacula quite fast..

ALL configuration for bacula is done in its data-base so its not raly hard to manage from ebox's point of view. 

bacula COULD and SHOULD be tightly intergrated as a default  module. 

the biggest problem of backula is however this:

how should one generate bacula config scripts for the clients.   

All tips hints and advices are based on my personal experience.
As I try my best to be as accurate as possible, following my advice is always at your own risk,
I claim absolutely NO responsibility in any way!

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #43 on: March 08, 2010, 10:19:26 am »
I've heard of people moving from BackupPC to Bacula, not the other way around. The Deduplication of data is the big hit for me because I'm storing 1.2TB of data into less than 220GB of space. BackupPC's biggest advantage is the ability to do server-side backups rather than client-side. The thing that slows it down the most is actually writing to disk and Ethernet packet issues, not the processor. Cheap fileservers don't ever max out the processor, they max out disk writing. BackupPC can also do backups close to client-side ones if you use Rsyncd which connects to an Rsyncd server on the cilent.

eBox's backup solution is separate though and not designed to replace anything else like Bacula or BackupPC. Storing configuration in a database is for of a preference and ldap support, which is extremely difficult to get working in Apache2 with eBox (for BackupPC, would be nice if it was needed for an Enterprise or medium-sized business.

Saturn2888

  • Zen Hero
  • *****
  • Posts: 707
  • Karma: +1/-0
    • View Profile
Re: Add-On: BackupPC request.
« Reply #44 on: March 09, 2010, 07:35:00 am »
I found what I was looking for, this guy gives a good reason to pull backups (BackupPC) rather than push them (Bacula) for security reasons: http://www.sanitarium.net/golug/rsync_backups_2010.html

Quote
Security: One of the reasons we have backups is because of the possibility of malicious activity (hackers, worms, trojans, etc). If your backup device is plugged into the computer being backed up then any malicious users or software that can destroy your data can also destroy your backups. Keeping your backups on a separate isolated server protects them from this possibility. Note that this is also why I prefer to pull backups from a script running on the backup server rather than pushing backups from a script running on the backup clients.