Invitation to discuss methods of clustering Zentyal

[half_life]

I think it is a good time to discuss the details of how to really go about clustering a zentyal server so as to achieve software high availability in addition to hardware high availability.  Anyone else interested in working on a specification?

[christian]

Good idea!

I'm very prone to take part to this debate, although I may not share title as far as high availability is our target.
What I mean to say is that cluster is one (and only one) of the various solution to provide higher availability.

[stuartiannaylor]

Personally with 3.0 I think there is much that needs to be done with what we have.

Many of the modules need refining as there are still many raw edges.

I like the idea of clustering but my area is the SMB market and also Zentyal is aimed as a small business server.

There is a simple method where rsync and mysql replication with configuration shipping could provide a simple redundant backup server.

I believe it would be a relatively simple chore to allow modules to enabled and a secondary mirrored server to go live on the failure of a heartbeat.

This would provide infrastructure costs more inline with my client needs  as the hardware needs for enterprise clustering usually is a minimum of three servers with mirrored san solutions.

So I would like to propose that a simple mirror server solution would be a good place to start in the SMB arena and then maybe we could move to the options of enterprise clustering.

Maybe a poll of what zentyal users infrastructure are in terms of server quantity and site layout.

If the majority of us are M$ SBS equivalents then enterprise clustering might be above the remit of the majority.

 

[half_life]

Stuart,
I can understand your feelings and I concur that Zentyal is targeting the SBS market.  Where I don't agree is in thinking that "big iron" is necessary for clustering.  3 Nodes make sense for quorum purposes but I believe that it is possible with only two.  My ultimate goal here is to be able to add Zentyal machines at will to improve availability.  To my eyes this necessitates clustering technology.  Christian, I am teachable if you have some other paradyne  in mind.  It would proabably make sense to actually put all of the services down on paper with the required backend software then go from there identifying how to replicate each service.

[stuartiannaylor]

If you can do it with two servers then I am all for it.

From a start point its always good to see what the competition is doing.

I am a big fan of proxmox and they provide clustering with two servers in a way but need a small partition on a third to co-ordinate the cluster.

There method is rock solid but there is still the third partition needed.

[half_life]

As you say a quorum disk is required and shared storage.  You can use DRBD NFS or SAN.  But to be clear Proxmox provides hardware reliability, not really software reliability.  I run Proxmox here and I can tell you that a mysql crash = both copies crashed.  Proxmox or similar will figure heavily into my final solution  but not necessarily what I emphasize here.  BTW it is MUCH better to use a third node in Proxmox even if it is underpowered hardware. The third node can be fore voting purposes only. 

[stuartiannaylor]

I am interested in what you have in mind half life.

I like to quote something that I have used and I have been really happy with proxmox and the cluster was so easy to administer and install.

Thing I didn't like was the third partition. I ended up utilising and old pc that was also a cctv recorder. A third partition requires another computer and we are stuck with three again.

I am lost for ideas's where two servers alone can be termed as an effective enterprise cluster with hitting the word "Cost".

That why I am really interested in another term and looking provding two simple raid servers that are identical.

I am having some problems with ldap syncronisation and samba additional controllers at the moment.
I have a site that is being handed to zentyal for the new year.

Its a simple setup and mysql replication and rysnc can mirror the server if I could get user syncronisation. I believe this is fit for purpose and of the lowest cost.
This would be enough for many smb solutions.

As I say I am interested in your proposal.

[half_life]

Just a quick suggestion as a possible solution for quorum disk  take a look at http://www.stephenwagner.com/?p=607  $35 should be in budget.

[christian]

Helf_life,

Your are indeed finger-pointing that "cluster" is not the holy-grail.
End-user need is never cluster but availability.

Cluster is one (part) of the answer but it doesn't prevent to perform back-up neither to provide redundant services like DNS, LDAP, proxy, DB or whatever depending on what matters for your own specific situation.
That's why I don't fully share the "Zentyal clustering" title.

From technical standpoint, cluster with only 2 servers doesn't work. Third server is mandatory to minimize risk of split-brain. It doesn't mean it has to be powerful but it must be different server.
If you don't have too aggressive RPO/RTO, cluster might be the right answer but it doesn't cover data corruption (especially at DB level) neither covers problem like "oh &$#§@% !!!! I just deleted thousand of records from my DB using wrong SQL syntax"
or "my network switch is broken therefore my cluster is down too".

[stuartiannaylor]

Because Proxmox is a virtual platform in similar modes to zen, vmware the cluster is superb as the virtual server just swaps from one metal platform to another.

Christian is correct as there is much to clustering and service clustering and data integritity are very important.

Saying that, Christian I have asked you on several occasions to brief details on how you see a future with zentyal clustering.
This is just opinion but I feel you are dodging the request because the solution does involve cost that is way above the majority usage of zentyal.

This is why I feel a simple methodology to configuration ship to a backup server that is user synced, file synced and db synced.
Would provide the simplest developement and be very much what we all know as zentyal.

Currently through the webadmin we can all go to the module section and disable / enable modules at will.
Programatically this would also be very easy to accomplish.
The only thing extra which is currently not supported is to be able to enable network settings so that a second server can go live with the exact settings as the failed downed server.

What I do say is that I don't believe saying no your wrong and not offering an alternative isn't in any way providing any value.

half_life I like your idea I am not sure of the data throughput for the quorum disk and I am sure the Pi would cope but I am unsure if cheap flash would. Then again I have to be honest that I am blind on how much data the quorum partition recieves.

Christian I would like your idea aswell as you are very knowledgeable but for some reason you seem reticent to share?

I do think it might be good to start talking about some new file systems such as btrfs cow and new directions to stop bit rot and general storage methods.

[half_life]

A quick look see at the quorum disk docs over at the proxmox wiki tend to make me want to say that an sd card or usb harddrive/thumb drive would be more than enough.  Each node writes a short file containing its health status and the other nodes read them.  Shouldn't be too much IO in my opinion.  In a two node setup you would give the quorum disk one vote meaning that a single server could continue as long as  the quorum disk is on its side of the breakdown.

[stuartiannaylor]

Its a really good idea, I have been meaning to have a play with a Pi. I have been interested to see how bodhilinux runs on one and just a whim of an idea the cheapest thin client possible.

As a quorum node its a great idea.

Still with proxmox all the servers are virtualised and the image swapping and services wouldn't really lend itself to Zentyal in its current guise.

If I was going to have a punt at what I would like in the future it would be a hybrid of zentyal and proxmox. With zentyal distributing services into isolated containers that have the inherent clustering ability that virtualisation offers. Separating services over distributed isolated processes, would make a much stronger product. I have a feeling IT wise things are going to be more cloudy even if its a local private cloud or hosted.
The SBS model of everything in one box reminds me of the wordperfect days where (was it ver 7) it did all things and was usurped by an upstart that distributed functionality into separate programs.
Zentyal is beginning to remind of that wordperfect model maybe M$ will come to the rescue again

[half_life]

Zentyal is actually very happy in the Proxmox world.  With two heavy duty servers + light duty machine here for quorum.  I never really went with the quorum disk idea.  DRBD between the two servers for shared storage allows HA features to protect from hardware failure.  What is stopping us from adding more zentyal machines for redundancy?

  Samba4 from what I can see takes care of replicating itself.  Mysql ( I am thinking of Xtradb as a replacement here for a packaged cluster setup) is not hard to make redundant.  We have two types of mail to deal with (Zarafa is handled in mysql) +postfix ( I think it would be hard to guarantee delivery during a failover though)  and several other modules to address ( I think most would be handled by a rsync approach) .  Tie it all together with a loadbalancer such as haproxy (would need to be redundant also) and use heartbeat to control failovers. 

I know this is all rather glib and detail sparse.  Consider it a back of the napkin idea.

Thoughts?

 My grand master plan would be to get the pieces snapped together and then engage the developers to help get it integrated into the zentyal gui.

[half_life]

Its a really good idea, I have been meaning to have a play with a Pi. I have been interested to see how bodhilinux runs on one and just a whim of an idea the cheapest thin client possible.
<snip>

I have run bodhi on a PI.  It runs very nice indeed.  You might also take a look at https://www.miniand.com/products/Hackberry%20A10%20Developer%20Board I have one of these also.

[christian]

We have, unfortunately, for the time being, very few design choices with Zentyal:
- because Zentyal doesn't provide any feature that would offer high(er) availability ⁽¹⁾, the remaining choice is active/passive cluster, using either highly available SAN or DRBD based on dedicated network. As said above, "quorum" node is required too.
- if Zentyal evolves toward something more flexible ⁽²⁾, we can think about better and lighter but more complex redundant design where only services that need "data" replication will be part of this active/passive cluster, others being redundant, therefore with higher availability.
- thinking about Zentyal landscape alone would result in truncated view. E.g, look at network ⁽³⁾ 
- again and again, backup strategy must also be part of your HA design: what if you wrongly delete accounts or corrput your DB? This operation will be quickly duplicated and cluster will not help. Is current Zentyal backup fitting your needs or would you dream about snapshot with quicker rollback?

Because of this and depending of your RPO/RTO requirements, it may happen that "state-of-the-art" highly available Zentyal platform is far too much for SMB market and more efficient approach is active/passive cluster. This is, for one part, due, IMHO, to the "all-in- one" initial design. But one can't have both "easy to manage simple box" and (complex) highly available environment. There is no such thing as a free lunch isn't it? 

⁽¹⁾
Zentyal is not designed to handle "multiple Zentyal box" management. in case of multiple Zentyal servers, there is no "service redundancy" but different services, per server.
- no DNS replication, zone transfer but local DNS
- one single mail service per mail domain
- LDAP replication still not flexible enough (+ internal LDAP for Samba)

⁽²⁾
If Zentyal was able to split "infrastructure" services then handle high availability at this level and permit to configure "front-end services" (i.e. end-users oriented services") so that it can benefit from the highly available Zentyal infrastructure, then we could imagine to have:
- redundant DNS, LDAP, NTP, Kerberos, CA
- multiple MTA, HTTP proxy, web servers (?),
- failover (meaning active/passive mode) for file sharing, DHCP
- we should then look at all Zentyal services and decide whenever we can or want to have it redundant or in failover mode.
 
⁽³⁾
Highly available Zentyal doesn't make sense without highly available network, at least for the core part. This means that one should think about redundancy in term of service and server but also in term of network, to be split into core network and end-user network, including dedicated network for DRBD and heartbeat.