High-Load Diagnosis help-- zentyal 2.2

[Marcus]

Hello,

I didn't experienced any memory leak on any of the Zentyal's software collection... 

Also, if you pay attention to the "top" I posted earlier, you'll see that the server that I used for this exemple is up for over 146 days and that it is running with a little 2GB of ram.

I'm telling you, if you don't do at least a "top" to find out what process is going nut, you'll never know what is really going on...  You'll only be guessing and that doesn't worth a penny. 

The other trade off is that you'll probably never know what was wrong - you'll be blaming X or Y without any solid proof and end up frustrated saying that the world is crap!

A lot of problems I had in the past were fixed by starting with a simple "top" investigation.

Best,

Marcus

[vshaulsk]

I am not saying it is X or Y and I perfectly understand that the best way to figure it out is during the situation ....look at "top" or have monitoring software on another machine.

I will post my "top" output late this evening.  All I am saying is that for the last year and a half I have never had this happen.  Maybe it will never happen again.... I don't know.  When I look at "top" I don't see currently any system taking up either a lot of ram or using a lot of cpu process by itself..... maybe combined all the process take up a lot of ram (not sure have not added it up).  The only thing I notice currently (or always about my system) is that I never have much free ram and my cache is approaching 4 gigs.  I would think with 8 gigs of available ram for the host OS I would several gigs left free.

Maybe this is not an issue because even if my system would start using swap .... I don't think this would have made my CPU's start using 100%...... but the RAM thing is just a question based off the only strange thing I see about my system (at least when compared to my windows experience)

[Sam Graf]

@Marcus: Top isn't going to tell us where the free memory is going. I agree that it would help solve a high load problem if the machine can be viewed while the problem is occurring.

As for this "memory leak"-- I agree it probably isn't a true memory leak. More likely it's event-driven. Nevertheless, something does happen to free RAM that seems unpleasant if nothing else.

[christian]

No I never really knew what happened and never had time to investigate. I noticed this, not while I was looking at log but when Zentyal log modules was started. Once stopped, everything was OK. I suspect something with Postgres DB unavailability but this is only suspicion.

Be cautious with memory usage on Unix/Linux systems. What really matters is not that much memory usage but swap, if any. Depending on the OS, it could allocate all the available memory just because it's there and available for processes to run and load data and if there is no specific request claiming for additional memory, there is no memory released. This is not an issue. You start to face memory shortage however when there is no enough memory and some memory parts are written on disk.

[Sam Graf]

That's the unpleasant part. The system "problem" as I'm describing it involves a modest amount of swap space usage--296K in the case of the above system. So seemingly regardless of the amount of installed RAM, at some point affected systems "bottom out" and end up maintaining use of a small amount of swap.

[vshaulsk]

This is what top is showing me.  My system is currently using swap for some reason... even though when you look at all the things running it is not 16 gigs worth of stuff.

[vshaulsk]

I restarted the system ..... and within a half hour it was once again using all of my system ram according to "top".

I have not been able to replicate the highload issue, but the ram usage comes back every time. 

So far once I restarted I have not had the system start using swap, but it basically fills up the cache until I only have about 200mb of system ram left.  To me this seems like a problem..... but maybe I am wrong.

Why would the system cache just keep building up.... and why would it not reset later???

[christian]

I don't understand why this must be an issue.
Linux memory usage is different from Windows. As a result Linux tries to cache as much as possible.
Look at the "cached" figure in picture you posted.

Again, you will start having problems if system starts to swap. This could be something to look at closely (is Zentyal showing such report in monitoring? I don't remember and can't test right now).
If there is no swapping, there is no memory issue.

in case it helps:
http://tldp.org/LDP/tlk/mm/memory.html

Some additional links:
http://linux-mm.org/LinuxMM
http://www.linux-tutorial.info/modules.php?name=MContent&pageid=260

[vshaulsk]

Christian, Marcus, Sam

I have been doing some reading ..... I have to admit I did not know that linux cached ram in the manner that it does.  Currently the system is showing about 10 gigs of cache and about 5mb of swap.  However the system is very responsive and seems to be functioning.  So I guess I will drop the ram issue.

The high load I will keep looking to see if it happens .... perhaps next time I will be able to figure out what process caused the issue.

[christian]

cool 

For those still having some doubt, this is an interesting link:
http://www.linuxatemyram.com/

[Sam Graf]

It would help, though, even just as an educational exercise, if someone had said up front, "It is perfectly normal for a Linux system to maintain a small amount of swap usage. This is only a problem when swap sizes are large and/or growing."

"How much physical memory do I need in my Zentyal server?" is a completely fair question that is complicated a little by the behavior we see. If we can't answer that question by simply watching how RAM is used, we wonder about how best to answer it. Educators call those teachable moments, and whether there is a problem or not (and I've already said I don't think we have a real problem), it would be nice to have a simple explanation of what we actually are seeing--which has nothing to do with Windows, as far as I can tell. Maybe then we would have a clear idea of how to know if physical memory needs increasing or not.

[vshaulsk]

Yes I definitely have learned a lot in the last 48 hours. 

[Sam Graf]

A problem is inconsistent information. Take this paragraph, for example:

No, disk caching only borrows the ram that applications don't currently want. It will not use swap. If applications want more memory, they just take it back from the disk cache. They will not start swapping.

Yet our systems do use swap. Yet it isn't a problem. And people wonder why we get a little confused ...

[christian]

What really matters is the swap rate more than swap size itself (although one may assume that swap size will increase in case requirement for free memory is higher than available memory).

vmstat and free commands will help a lot here to check  swap in and out and available memory.

[Marcus]

Hello,

By default Ubuntu set the swappiness level to 60.  This means that you'll start using your swap when ram reaches/goes over 40%.

In your case, you are always using over 40% of ram (cache doesn't count).

So, I would recommend you to lower your swappiness level (10 would be my recommendation in your case).


This is how you can do it;

Check swapinness level:

Code: [Select]

cat /proc/sys/vm/swappiness
Change swappiness level on the fly (going from 60 to 10):

Code: [Select]

sysctl vm.swappiness=10**This will revert back to it's initial value after a server reboot**

Make change permanent:
1) Edit /etc/sysctl.conf

Code: [Select]

nano /etc/sysctl.conf
2) Search for vm.swappiness and change its value as desired.
**If vm.swappiness does not exist, add it to the end of the file like so:

Code: [Select]

vm.swappiness=10
Tweak tips;
*Try to not go over 75% of real ram usage

*Clearing swap (be careful to have enough ram free)

Code: [Select]

swapoff -a && swapon -a
*You may use a USB key/Flash to IDE adaptor(with a CF)/Different Hard Drive for your swap drive instead of using the same hard drive that your system use.  This will avoid your regular hard drives to slow down.

Off topic but still related
*It can be use on Ubuntu desktop in order to speed it up (actually it will just prevent to use swap so your desktop will be "more reactive" after weeks of operation without a single reboot).

Best,

Marcus