Zentyal 3.0 manage-logs process using 50-100% cpu and using 80% RAM

[djedge]

Hi,

My zentyal 3.0 (Clean install) is having a problem. I've been monitoring performance since it was installed on 9/14/2012. The process manage-logs is using at random a huge amt of cpu (50-100% of one core of the quad core cpu) and also using 80-90% of ram on the server at times. This behavior is affecting performance in a fairly negative way i believe.

I have tried disabling both the logs module and the bandwidth monitor and neither of these help. This does not happen 100% of the time but is a majority of the time it seems.

Has anyone had this issue or know of a solution?

All components are up to date as well as all ubuntu updates.

Thanks!

[Sam Graf]

I can confirm your description of the problem (the numbers were different for me but the substance of the problem was the same). I can't give you a solution, since nothing I tried stopped it.

[djedge]

yeah, it's a real bummer. 3.0 is so much better than 2.2 as far as features, UI, etc. But with this kinda of bug i may have to revert back to 2.2-2 ... Clients are already complaning.

[Sam Graf]

Thankfully I'm under no pressure to deploy. I think the developers will sort the issue as quickly as possible, hopefully soon enough that you can hang on to your 3.0 installations.

[christian]

Given the huge amount of in-depth changes in term of architecture, I'm not surprised to face some glitches with 3.0
I should not say that but rushing to go live in prod with this release, even if this is no more RC, is a bit optimistic 

No doubt Zentyal team is working very hard (days and night for what I understand) to solve all bug but it will take some time in any case. If you can't afford it, stay with 2.2 for a couple of week, at least from my standpoint.
e.g. yesterday night I updated my 3.0 test platform: after system update, GUI became unresponsive and I had to restart Zentyal from CLI.

[djedge]

I have not had any problems with the GUI and i'm quite happy with that aspect.

The only two problems i've had were the cpu usage / memory issues with manage-logs process..  and with samba / file sharing refusing to configure (Which i gave up on for now).

[Marcus]

Hello.

You high CPU load could also come from your swap.

You should check your swappiness...

Swappiness is a property for the Linux kernel that changes the balance between swapping out runtime memory, as opposed to dropping pages from the system page cache. Swappiness can be set to values between 0 and 100 inclusive. A low value means the kernel will try to avoid swapping as much as possible where a higher value instead will make the kernel aggressively try to use swap space. The default value is 60, and for desktop systems, setting it to 100 may affect the overall performance, whereas setting it lower (even 0) may improve interactivity (decreasing response latency.)

Short story;
First, check it out using a shell cmd:

Code: [Select]

cat /proc/sys/vm/swappiness
It should return "60"

If that is the case, you have a hardware/software bad configuration. You'll have to either increase your RAM or stop/tweak some services. If you can't or want a quick fix, please keep reading.

Please run this cmd in your shell:

Code: [Select]

sudo sysctl vm.swappiness=10
Following this cmd, executing the previous one should return "10"

Now let's make it persistant (following a reboot);
Paste this bloc in your shell:

Code: [Select]

sudo cat >> /etc/sysctl.conf <<-SWAPPINESS
cat vm.swappiness = 10
SWAPPINESS
Now we must restart the swap in order to empty it;
Paste this cmd in your shell:

Code: [Select]

sudo swapon -av && sudo swapoff -av(it may takes few seconds/minutes so be patient)

Hope this will help

Marcus

[christian]

Marcus,

Look at swap activity is a good point.

I though however that swappiness value was only reflecting how Ubuntu will be prone to swap or not but not reflecting whenever some swap occurs 
Furthermore swappiness = 60 is the standard value for desktop. Is it suitable for Ubuntu server ?

[Sam Graf]

Swap comes into play but as effect, not cause, in this particular case, I think. As djedge noted, we have a process or two (in my case MySQL along with manage-logs) that is consuming significant resources for long periods. And in my case, the longer the uptime, the longer manage-logs hogged resources. Eventually, it ran continually.

[djedge]

I have noticed that mysql is also a culprit...   Sometimes mysql is the main culprit , sometimes it's manage-logs.  This of course leads me to believe manage-logs is running tons of mysql queries as mysql might be 90% cpu while manage-logs is 10-20% at the same time.

I can easily change the swappiness value but in reference to the previous post i thought 60 was recommended for server installs on ubuntu... (Which is why it's the default).

Lowering the swappiness might delay performance degradation but the long term effect is still the same, mysql / manage-logs will thrash eventually.

Also , I currently have 2gigs allocated to the zentyal VM (running on esxi 5.1). I can increase the ram easily however 2 gig should be more than sufficient for just the firewall / gateway . I'm not running office or communication aspects of zentyal.

[Marcus]

Hello,

swappiness = 60 is the standard value

i thought 60 was recommended for server installs on ubuntu... (Which is why it's the default).

It is much more the "default" than the "recommended" setting.  IMO, "Recommended" is what you need it to be depending your computer vocation. 

I though however that swappiness value was only reflecting how Ubuntu will be prone to swap or not but not reflecting whenever some swap occurs

As far that my understanding of it goes, yes, that is totally correct.

According to the official Ubuntu Website:

The default setting in Ubuntu is swappiness=60. Reducing the default value of swappiness will probably improve overall performance for a typical Ubuntu desktop installation. A value of swappiness=10 is recommended, but feel free to experiment.

ref:
https://help.ubuntu.com/community/SwapFaq

With that said;
For desktops or database servers, IMO, swappiness = 10 is the configuration that work the best (try running Chrome or Firefox while swapping -> Only do it if you like "grayed, static, non-responsive windows" ).

We must keep in mind that swap = drive use.  HDD and SSD are slow memory. RAM is much more fast.

IMO, caching a database in RAM is a smarter aproach since it is less using/putting pressure on physical drives.  This also comes from the fact that logs requires a lot of read/write on a physical device.  The selected media used mathers.

Short story;
Load balancing the hardware ressources simply makes everything run faster.


Some other thinking;

This of course leads me to believe manage-logs is running tons of mysql queries as mysql might be 90% cpu while manage-logs is 10-20% at the same time.

When CPU is toping up, usually, this mean that there is "things to be done" waiting in the CPU queue.  If you are swapping, this is an even more common scenario (hard drives are busy handing or writing things.  Either on the swap partition, in your logs or in your DBs.).

That multi-tasking philosophy is driving a lot of server administrators to dedicate a full HDD or SSD for the swap.

Swap comes into play but as effect, not cause, in this particular case, I think.   

Well, yes and no...  When swap comes into play, things will just go worst and worst and worst and ... (hard drives are kinda slowing their response time down so getting less things done - downward spiral scenario)

One last thing;
Try checking your logging policies.  Being to much of a "big brother" will slow you down.

[Off Topic]

And in my case, the longer the uptime, the longer manage-logs hogged resources. Eventually, it ran continually.

You can do better - garanteed! 
My Zentyal 2.0 multi tasking home server build on a 4X WD Green 5400RPM soft RAID10/RAID1 (the full 2.0 setup except the PBX side that was replaced by FreePBX + much hacking on top of it like suPHP and a rethinked FTP module) had only very few reboots (once every power failure). The Zarafa indexer is the only thing that chews up 50% of my AMD 5400 dual core.  And on top of that, it does have a total of 2GB of RAM.
[/Off Topic]

I hope this will help

Best,

Marcus

[Sam Graf]

Marcus, while I don't disagree with your general point, this is something 1) new to 3.0, and 2) not consistent even in 3.0, as far as I can tell. I have never seen MySQL (say, in a LAMP configuration) just run constantly. I'm suspicious of the log process itself (as I already mentioned, I was logging events and DHCP with one attached client--not even close to a Big Brother) because the logs themselves are behaving strangely, as noted by other users.

[eisveldt]

Hello, just want to confirm the issue, i have it to.

I did a strace on the manage-logs process and i looks like a loop of some sort and it cant find some path etc.

Its a good concept, and a good framework with slow but good looking GUI.
But!! Over all Z3.0 feels unstable and i would not recommend it in production environment.
My HW is Intel dual core and 2Gig memory and i test it for routed proxy with AV and content filtering.
Some other issues is Zentyal apache is stoping preventing me from administrating, OK i can start it up.
Other processes stoped randomly like HTTP proxy and network services etc.

Keep up the good work and i hope we soon can see a reel RC 3.0.

Best regards Jimmy
 

[Sam Graf]

Hi, and welcome!

I agree that Zentyal 3.0 seems to have some rough edges, probably largely because it's a major update ( the first Zentyal release built on top of Ubuntu 12.04 LTS). I think the developer's effort to leverage new Samba features has probably contributed to some of the rough edges.

That said, in my experience the Zentyal developers work pretty hard at making their software a quality product. Even if 3.0 final was released prematurely, I have no doubt that they're concerned about fixing things as quickly as possible and having a very stable product. We can help them by continuing to file and monitor bug reports for reproducible issues and discuss possible problems (constructively) here in the forums.

[omata]

Hello.

You high CPU load could also come from your swap.

You should check your swappiness...

Swappiness is a property for the Linux kernel that changes the balance between swapping out runtime memory, as opposed to dropping pages from the system page cache. Swappiness can be set to values between 0 and 100 inclusive. A low value means the kernel will try to avoid swapping as much as possible where a higher value instead will make the kernel aggressively try to use swap space. The default value is 60, and for desktop systems, setting it to 100 may affect the overall performance, whereas setting it lower (even 0) may improve interactivity (decreasing response latency.)

Short story;
First, check it out using a shell cmd:

Code: [Select]

cat /proc/sys/vm/swappiness
It should return "60"

If that is the case, you have a hardware/software bad configuration. You'll have to either increase your RAM or stop/tweak some services. If you can't or want a quick fix, please keep reading.

Please run this cmd in your shell:

Code: [Select]

sudo sysctl vm.swappiness=10
Following this cmd, executing the previous one should return "10"

Now let's make it persistant (following a reboot);
Paste this bloc in your shell:

Code: [Select]

sudo cat >> /etc/sysctl.conf <<-SWAPPINESS
cat vm.swappiness = 10
SWAPPINESS
Now we must restart the swap in order to empty it;
Paste this cmd in your shell:

Code: [Select]

sudo swapon -av && sudo swapoff -av(it may takes few seconds/minutes so be patient)

Hope this will help

Marcus

Hi,

I posted the very same issue at Spanish forum, so in my case 'manage-log' is hogging both, memory and CPU, then browsing tasks at workstations are stopped, all clients get an 102 error and the system almos freeze because of resource consumption levels. My server is an AMD Sempron based system, using 4 GB of RAM; my previous device with Z2.2 has only 1 GB and worked a lot better even without use all the RAM neither CPU.

Then, I tried the directions for change and turn permanent the swappiness thing, so when I pu the following in the shell:

Code: [Select]

sudo cat >> /etc/sysctl.conf <<-SWAPPINESS
cat vm.swappiness = 10
SWAPPINESS

I get:

Code: [Select]

-bash: /etc/sysctl.conf: Permission denied

So, I just want to know, where I can find the installation information related to 2.2 version, because repository info was removed from installation page. In the other hand 'dansguardian' isn't behaving like 2.2 too, it is blocking update process when an ubuntu workstation run apt-get update and apt-get upgrade, the only thing that let me run without issue the update command in ubuntu/debian workstations is disable dansguardian/web-filtering, squid works fine.

Best regards,

Oscar.