Zentyal Forum, Linux Small Business Server
Zentyal Server => Installation and Upgrades => Topic started by: half_life on December 03, 2012, 05:32:00 am
-
I think it is a good time to discuss the details of how to really go about clustering a zentyal server so as to achieve software high availability in addition to hardware high availability. Anyone else interested in working on a specification?
-
Good idea!
I'm very prone to take part to this debate, although I may not share title as far as high availability is our target.
What I mean to say is that cluster is one (and only one) of the various solution to provide higher availability.
-
Personally with 3.0 I think there is much that needs to be done with what we have.
Many of the modules need refining as there are still many raw edges.
I like the idea of clustering but my area is the SMB market and also Zentyal is aimed as a small business server.
There is a simple method where rsync and mysql replication with configuration shipping could provide a simple redundant backup server.
I believe it would be a relatively simple chore to allow modules to enabled and a secondary mirrored server to go live on the failure of a heartbeat.
This would provide infrastructure costs more inline with my client needs as the hardware needs for enterprise clustering usually is a minimum of three servers with mirrored san solutions.
So I would like to propose that a simple mirror server solution would be a good place to start in the SMB arena and then maybe we could move to the options of enterprise clustering.
Maybe a poll of what zentyal users infrastructure are in terms of server quantity and site layout.
If the majority of us are M$ SBS equivalents then enterprise clustering might be above the remit of the majority.
-
Stuart,
I can understand your feelings and I concur that Zentyal is targeting the SBS market. Where I don't agree is in thinking that "big iron" is necessary for clustering. 3 Nodes make sense for quorum purposes but I believe that it is possible with only two. My ultimate goal here is to be able to add Zentyal machines at will to improve availability. To my eyes this necessitates clustering technology. Christian, I am teachable if you have some other paradyne in mind. It would proabably make sense to actually put all of the services down on paper with the required backend software then go from there identifying how to replicate each service.
-
If you can do it with two servers then I am all for it.
From a start point its always good to see what the competition is doing.
I am a big fan of proxmox and they provide clustering with two servers in a way but need a small partition on a third to co-ordinate the cluster.
There method is rock solid but there is still the third partition needed.
-
As you say a quorum disk is required and shared storage. You can use DRBD NFS or SAN. But to be clear Proxmox provides hardware reliability, not really software reliability. I run Proxmox here and I can tell you that a mysql crash = both copies crashed. Proxmox or similar will figure heavily into my final solution but not necessarily what I emphasize here. BTW it is MUCH better to use a third node in Proxmox even if it is underpowered hardware. The third node can be fore voting purposes only.
-
I am interested in what you have in mind half life.
I like to quote something that I have used and I have been really happy with proxmox and the cluster was so easy to administer and install.
Thing I didn't like was the third partition. I ended up utilising and old pc that was also a cctv recorder. A third partition requires another computer and we are stuck with three again.
I am lost for ideas's where two servers alone can be termed as an effective enterprise cluster with hitting the word "Cost".
That why I am really interested in another term and looking provding two simple raid servers that are identical.
I am having some problems with ldap syncronisation and samba additional controllers at the moment.
I have a site that is being handed to zentyal for the new year.
Its a simple setup and mysql replication and rysnc can mirror the server if I could get user syncronisation. I believe this is fit for purpose and of the lowest cost.
This would be enough for many smb solutions.
As I say I am interested in your proposal.
-
Just a quick suggestion as a possible solution for quorum disk take a look at http://www.stephenwagner.com/?p=607 (http://www.stephenwagner.com/?p=607) $35 should be in budget.
-
Helf_life,
Your are indeed finger-pointing that "cluster" is not the holy-grail.
End-user need is never cluster but availability.
Cluster is one (part) of the answer but it doesn't prevent to perform back-up neither to provide redundant services like DNS, LDAP, proxy, DB or whatever depending on what matters for your own specific situation.
That's why I don't fully share the "Zentyal clustering" title.
From technical standpoint, cluster with only 2 servers doesn't work. Third server is mandatory to minimize risk of split-brain. It doesn't mean it has to be powerful but it must be different server.
If you don't have too aggressive RPO/RTO, cluster might be the right answer but it doesn't cover data corruption (especially at DB level) neither covers problem like "oh &$#ยง@% !!!! I just deleted thousand of records from my DB using wrong SQL syntax"
or "my network switch is broken therefore my cluster is down too".
-
Because Proxmox is a virtual platform in similar modes to zen, vmware the cluster is superb as the virtual server just swaps from one metal platform to another.
Christian is correct as there is much to clustering and service clustering and data integritity are very important.
Saying that, Christian I have asked you on several occasions to brief details on how you see a future with zentyal clustering.
This is just opinion but I feel you are dodging the request because the solution does involve cost that is way above the majority usage of zentyal.
This is why I feel a simple methodology to configuration ship to a backup server that is user synced, file synced and db synced.
Would provide the simplest developement and be very much what we all know as zentyal.
Currently through the webadmin we can all go to the module section and disable / enable modules at will.
Programatically this would also be very easy to accomplish.
The only thing extra which is currently not supported is to be able to enable network settings so that a second server can go live with the exact settings as the failed downed server.
What I do say is that I don't believe saying no your wrong and not offering an alternative isn't in any way providing any value.
half_life I like your idea I am not sure of the data throughput for the quorum disk and I am sure the Pi would cope but I am unsure if cheap flash would. Then again I have to be honest that I am blind on how much data the quorum partition recieves.
Christian I would like your idea aswell as you are very knowledgeable but for some reason you seem reticent to share?
I do think it might be good to start talking about some new file systems such as btrfs cow and new directions to stop bit rot and general storage methods.
-
A quick look see at the quorum disk docs over at the proxmox wiki tend to make me want to say that an sd card or usb harddrive/thumb drive would be more than enough. Each node writes a short file containing its health status and the other nodes read them. Shouldn't be too much IO in my opinion. In a two node setup you would give the quorum disk one vote meaning that a single server could continue as long as the quorum disk is on its side of the breakdown.
-
Its a really good idea, I have been meaning to have a play with a Pi. I have been interested to see how bodhilinux runs on one and just a whim of an idea the cheapest thin client possible.
As a quorum node its a great idea.
Still with proxmox all the servers are virtualised and the image swapping and services wouldn't really lend itself to Zentyal in its current guise.
If I was going to have a punt at what I would like in the future it would be a hybrid of zentyal and proxmox. With zentyal distributing services into isolated containers that have the inherent clustering ability that virtualisation offers. Separating services over distributed isolated processes, would make a much stronger product. I have a feeling IT wise things are going to be more cloudy even if its a local private cloud or hosted.
The SBS model of everything in one box reminds me of the wordperfect days where (was it ver 7) it did all things and was usurped by an upstart that distributed functionality into separate programs.
Zentyal is beginning to remind of that wordperfect model maybe M$ will come to the rescue again :)
-
Zentyal is actually very happy in the Proxmox world. With two heavy duty servers + light duty machine here for quorum. I never really went with the quorum disk idea. DRBD between the two servers for shared storage allows HA features to protect from hardware failure. What is stopping us from adding more zentyal machines for redundancy?
Samba4 from what I can see takes care of replicating itself. Mysql ( I am thinking of Xtradb as a replacement here for a packaged cluster setup) is not hard to make redundant. We have two types of mail to deal with (Zarafa is handled in mysql) +postfix ( I think it would be hard to guarantee delivery during a failover though) and several other modules to address ( I think most would be handled by a rsync approach) . Tie it all together with a loadbalancer such as haproxy (would need to be redundant also) and use heartbeat to control failovers.
I know this is all rather glib and detail sparse. Consider it a back of the napkin idea.
Thoughts?
My grand master plan would be to get the pieces snapped together and then engage the developers to help get it integrated into the zentyal gui.
-
Its a really good idea, I have been meaning to have a play with a Pi. I have been interested to see how bodhilinux runs on one and just a whim of an idea the cheapest thin client possible.
<snip>
I have run bodhi on a PI. It runs very nice indeed. You might also take a look at https://www.miniand.com/products/Hackberry%20A10%20Developer%20Board (https://www.miniand.com/products/Hackberry%20A10%20Developer%20Board) I have one of these also.
-
We have, unfortunately, for the time being, very few design choices with Zentyal:
- because Zentyal doesn't provide any feature that would offer high(er) availability (1), the remaining choice is active/passive cluster, using either highly available SAN or DRBD based on dedicated network. As said above, "quorum" node is required too.
- if Zentyal evolves toward something more flexible (2), we can think about better and lighter but more complex redundant design where only services that need "data" replication will be part of this active/passive cluster, others being redundant, therefore with higher availability.
- thinking about Zentyal landscape alone would result in truncated view. E.g, look at network (3) :-\
- again and again, backup strategy must also be part of your HA design: what if you wrongly delete accounts or corrput your DB? This operation will be quickly duplicated and cluster will not help. Is current Zentyal backup fitting your needs or would you dream about snapshot with quicker rollback?
Because of this and depending of your RPO/RTO requirements, it may happen that "state-of-the-art" highly available Zentyal platform is far too much for SMB market and more efficient approach is active/passive cluster. This is, for one part, due, IMHO, to the "all-in- one" initial design. But one can't have both "easy to manage simple box" and (complex) highly available environment. There is no such thing as a free lunch isn't it? :-[
(1)
Zentyal is not designed to handle "multiple Zentyal box" management. in case of multiple Zentyal servers, there is no "service redundancy" but different services, per server.
- no DNS replication, zone transfer but local DNS
- one single mail service per mail domain
- LDAP replication still not flexible enough (+ internal LDAP for Samba)
(2)
If Zentyal was able to split "infrastructure" services then handle high availability at this level and permit to configure "front-end services" (i.e. end-users oriented services") so that it can benefit from the highly available Zentyal infrastructure, then we could imagine to have:
- redundant DNS, LDAP, NTP, Kerberos, CA
- multiple MTA, HTTP proxy, web servers (?),
- failover (meaning active/passive mode) for file sharing, DHCP
- we should then look at all Zentyal services and decide whenever we can or want to have it redundant or in failover mode.
(3)
Highly available Zentyal doesn't make sense without highly available network, at least for the core part. This means that one should think about redundancy in term of service and server but also in term of network, to be split into core network and end-user network, including dedicated network for DRBD and heartbeat.
-
Christian thanks, your bang on the button in my mind.
The only area I differ is that I think actually zentyal will fit a multiple server ditsributed mode.
Zentyal is a strange product in that its a product that controls the settings of other products.
For me its a common admin interface that is a familar friend for the sysadmin. Because of this it makes configuration tasks very easy.
I have set up ubuntu services the hard way via manuals and /etc config files. What has taken days and much head scratching can be accomplished with zentyal in minutes.
A complete server with most of the zentyal module offerings can be implemented from an iso to blank drives to a working server in 30 minutes. That is a huge ammount of value.
I feel btrfs will provide the snapshots, dynamic inode allocation, efficient incremental backups, background scrub processes, object level striping and mirroring.
I feel LXC can provide the isolation and distribution required and whilst linking at the network layer and providing control through this great web admin console we are all familiar with.
I know btrfs and lxc are still under heavy development and I am not saying they should be employed now. I feel that development should have brtfs and lxc in mind as without doubt they will become common technologies in majority usuage.
The monolithic all in one approach is similar to XP EOL yes many will argue that it is a valid and usefull platform and on the flip side with people for an eye to the future will believe its already dead in the water.
As a product and light weight admin console that works at a network level, Zentyal is cutting edge and its future is only dependent on direction.
-
(1)
Zentyal is not designed to handle "multiple Zentyal box" management. in case of multiple Zentyal servers, there is no "service redundancy" but different services, per server.
- no DNS replication, zone transfer but local DNS
- one single mail service per mail domain
- LDAP replication still not flexible enough (+ internal LDAP for Samba)
Christian I don't think zentyal should be designed to handle "multiple Zentyal box" management. I think it is what it is and its a per server admin console that administers the server settings locally.
I do think that via distributed admin consoles of common familarity could become part and parcel of a multiple zentyal box scenario.
-
Its a really good idea, I have been meaning to have a play with a Pi. I have been interested to see how bodhilinux runs on one and just a whim of an idea the cheapest thin client possible.
<snip>
I have run bodhi on a PI. It runs very nice indeed. You might also take a look at https://www.miniand.com/products/Hackberry%20A10%20Developer%20Board (https://www.miniand.com/products/Hackberry%20A10%20Developer%20Board) I have one of these also.
I have a current love affair with bodhilinux to me its the perfect thin-client. I have it running on some seriously old and slow desktops and its amazing how such a beautiful desktop can be so slim and functional.
My users love it and find it simple to use and its ubuntu base has that huge scope that canonical provide.
On this forum we have a complete section called Zentyal desktop which is totally fubar. It is completely reinventing the wheel and again comes from this monolithic thought path of zentyal providing everything.
It has as much worth as the activity in development and in the forum. Totally misguided iniative that limits choice to a proscribed solution.
Zentyal is a server product that should have modules that allow multiple common distro's be it ubuntu, bodhilinux, fedora or suse to connect and utilise services.
The idea of a dedicated server and desktop platform is so M$ and narrow minded I could spit.
-
It has been a very long day. I have not forgotten this but I will be taking the evening off.
-
Hey everybody I am so apologetic as I know, I have brought to much angst to the forum.
I am passionate and I believe in zentyal not being just a product, collaborative meritocracy, pushing for a fairer more efficient solution.
Getting it right.
Two guys on here that I hold in the utmost respect are Sam Graff and Christian.
There knowledge and out look is truly outstanding.
I am apologetic because I have totally enjoyed finding some real voices with knowledge and I know I have been pushing to hard.
For me open source and collaboration isn't just a technical geek thing its a modern morality.
Lol I know, how it sounds but hey, I spent a year in South Africa and to be honest it has changed my life and perspective.
I have been really critical about the project because I believe in it.
I am court tomorrow and I am going to tell them to F*** off. Can't think of a better way to do it.
Hit a wave of depression as the last two years haven't gone my way.
I have been certain that I believe in open source meritocracy and very much a open source left winger.
I am back purely to get the community centres I have been involved with using samba4 and not M$.
I had this grand plan of open source and providing a solution that doesn't entrap users into a product racket.
Its been a total failure and in my sphere of educating grannies, displaced kids and generally the under privileged I have been trapped.
I have provided several centres with total open source solutions and can't get past the barriers of council education.
The tutors have M$ qualifications and teach only M$ products, show people Itunes and state that they are expensive but don't show alternatives.
Anyway I have had my own problems and an avid open source fanatic for want of better words. So much so I have refused to use M$ as a server product. With XP all was well, Vista was a thankfully a disaster but win7 has been a killer.
I have a two contracts left and been pretty much breadline for two years now.
I have been preaching Zentyal 3 and I know it will get there. Thing is if it is going to get any professional kudos then you can't release a beta. A Beta it is, and has been for a while.
Its a superb product and I believe it will be, but you can't release and advertise a product as it has been in its current state.
It just loses faith and it can have huge impacts on those who expect a normal cycle of a release to work.
Its simple its either a working bug checked release or you state that the bleeding edge is so and use as so.
The amount of users that have run into problems with 3 and looked at zentyal as a flaky product and gone elsewhere is truly depressing. All was needed was more conversation to the community.
This is open-source and be open, say it is how it is. That is why it is more efficient, that is why your users benefit via contribution.
Organic meritocracies are not gay, religious tree hugging freaks. If you have enough experience in IT then nothing is black and white. Denial and not being open just wastes time and doesn't indicate to knowledge users where there skills might be beneficial.
The trade mark of zentyal alone if allowed to move in similar directions to products such as MySql could make the original crowd very wealthy. There are huge financial incentives to be reaped.
This will never happen by blind allegiance and drunken friendships. It isn't fair to the community for a few to party and dictate on there own ego's.
http://trac.zentyal.org/wiki/Structure/Governance
(http://trac.zentyal.org/wiki/Structure/Governance)
It is so obvious its not a meritocracy, firstly the community council is top heavy and consists mainly of commercial zentyal staff.
That is not a community council.
I did for a brief moment become a community council member and was just stunned by the ego of the council as for some reason they believe they know better than the users of the product.
Going back to Sam and Christian the community is in good hands.
Sorry Rob and Ishat, but wow guys look at everything you have forwarded, been involved with... Its so far from a meritocracy it is just deeply sad.
That is just my opinion, but if you are going to have a meritocracy as stated then do so. If this is going to be the control of a SADFL then please state so. Just don't pretend in the long term that it is something that it plainly isn't.
Why can't the community vote and provide indication to members of value. This doesn't mean it that its set in stone and always up to the SADFL to adjudicate.
So anyway I have gone bank, my probs. Court thing is a bit of a worry as I haven't a clue where my mind was at. Generally I apologise and to a few I do actually have a huge amount of respect for and best wishes.
Wish the project the best but by providing more discussion to the community and being transparent increases trust and participation. Stop the commercial denial as its losing trust.
Stuart.
Best wishes seriously.