Author Topic: General Linux Server Question  (Read 2344 times)

vshaulsk

  • Zen Samurai
  • ****
  • Posts: 477
  • Karma: +9/-1
    • View Profile
General Linux Server Question
« on: March 14, 2012, 01:31:12 pm »
Hello Everyone!!

I have a general linux question about how ram and swap are used in zentyal/ubuntu

First my setup:
Server running latest zentyal 2.2.6 with everything up to date.
Server hardware:
2x xenon E5606 processors (4 cores each)
10 gigs DDR3-1600 ram (5 slots with 2 gig sticks.... non ECC ram)
I have two 500 gig drives which I have setup in software raid1 using MDADM
I created 3 partions.  One for the root file system (30 gigs) EXT4, one for home (450 gigs) EXT4 and one I use for swap (20 gigs)
I also have 7 x 2TB seagate 5900RPM drives running in a software raid 6.

One top of my core zentyal installation I also run Virtualbox.  Which has two virtual servers running (One is a webserver and the other I am testing and Enterprise Resource Planning System).  The webserver has about 3 gigs or RAM allocated to it and the ERP system has about 1.5 gigs RAM allocated to it. 

This leaves the main system about 5.5 gigs of RAM. 

Now here is my question.  I have noticed that my system is using 2.84 gigs of swap and according to what the munin graphs are telling me I have seen a maximum of 524 pages/second of swap out.

I am trying to figure out why my system is using so much swap???  Is this normal??? 

Last night I changes the stripe_cache_size for my raid6 array from default 256 to 8192.  This has improved my write and read speeds to about 120 Mb/s (fully maximizes my gigabit lan).  However now I see my load jump during large file transfers and see a lot more swap being used in the beginning of the transfer.  As the transfer continues I see load go down. 

When I look to see what is causing the high load I always notice a lot of swap activity at the same time.  Why am I using swap when I have 5.5 gigs of RAM ????

christian

  • Guest
Re: General Linux Server Question
« Reply #1 on: March 14, 2012, 02:22:42 pm »
When I look to see what is causing the high load I always notice a lot of swap activity at the same time.  Why am I using swap when I have 5.5 gigs of RAM

According to literature, memory used is close to "stripe_cache_size * 4K * number-of-drives". This should not be that huge...



vshaulsk

  • Zen Samurai
  • ****
  • Posts: 477
  • Karma: +9/-1
    • View Profile
Re: General Linux Server Question
« Reply #2 on: March 14, 2012, 02:39:05 pm »
Yes Christian I found this formula as well.

stripe_cache_size * 4K * number-of-drives

8192 * 4k * 7 = 229376 = 229.376 MB   You are correct not that much

I guess in general when I look at the zentyal monitor it always shows free memory almost 0 while page cache is using the rest.  I know we have talked about this before and you told me this was because ubuntu tries to use all of your memory resources for caching info which speeds things up.  I am ok with this..... I also know that some swap usage is normal, but should be very little.  In my case I see something like 2.8 gigs...... I just don't know why.....as I stated when looking at munin graphs during my large file transfers I see the swap page/second spike.

Thank you for the help !!  I am just trying to understand my server behavior..... 

christian

  • Guest
Re: General Linux Server Question
« Reply #3 on: March 14, 2012, 02:50:05 pm »
swap should even - ideally - be 0 (zero), especially with such amount of memory.
What's the output of vmstat ?

You could also try "top" to look at processes consuming a lot of memory but I believe you did it already.
What does "buffers/cache" contain when you type
Code: [Select]
free -mYou may also try something like
Code: [Select]
ps -A --sort -rss -o comm,pmem | head -n 15 to identify such processes.

vshaulsk

  • Zen Samurai
  • ****
  • Posts: 477
  • Karma: +9/-1
    • View Profile
Re: General Linux Server Question
« Reply #4 on: March 14, 2012, 03:20:06 pm »
vmstat =
swpd = 2227744
free = 214176
buff = 164648
cache = 732712

free -m

mem: total = 10013; used 9803; free 209
-/+ buffers/cache: used = 8921; free 1092
swap: total = 19071; used = 2172; free 16899

ps -A --sort -rss -o comm,pmem | head -n 15
                %mem
virtualbox     31.1
I7-filter        17.2
virtualbox      16
dansguardian 1.4
dansguardian 1.3
dansguardian 1.3
dansguardian 1.3
dansguardian 1.3
dansguardian 1.3
dansguardian 1.3
dansguardian 1.3





« Last Edit: March 14, 2012, 03:21:57 pm by vshaulsk »

christian

  • Guest
Re: General Linux Server Question
« Reply #5 on: March 14, 2012, 03:47:30 pm »
For sure there is some swap.
At the time you took this snapshot, 75% of memory was used by virtualbox + L7 filter + proxy. This is significant.
"free" memory for new process is "only" 1Gb (look at buffers/cache), one third being used when you launch heavy file transfer. It leave little room for additional stuff on such system.

vshaulsk

  • Zen Samurai
  • ****
  • Posts: 477
  • Karma: +9/-1
    • View Profile
Re: General Linux Server Question
« Reply #6 on: March 14, 2012, 04:03:31 pm »
Christian, to clarify.... you said "one third being used when you launch heavy file transfer"... you mean because of the stripe cache * 4K * 7 disks = 229 mb.... so only leaves about 771 mb for other process.

I guess I have any option to just add more ram.....

However I am not sure if that will solve my problem.... My first question is why with 10 gigs of ram.... almost no users is my system using so much ram.

If I lets say put in 24 gigs of ram.... what is stopping the system from using all of it up again.  I guess I don't really understand how linux works when it comes to ram/swap  .... buffer/cache.   It says that buffer is 8.9 gigs and one gig for cache....

When I first restart my system... the swap is 0 and I have a lot of free ram.... however as the system gets used the free ram goes down.  Swap starts getting used and buffer and cache fills up most of it.  I have to be honest that I don't have a lot of understanding in this area, but this just does not seem right.

christian

  • Guest
Re: General Linux Server Question
« Reply #7 on: March 14, 2012, 04:36:29 pm »
First have a look at this:
http://www.linuxatemyram.com/
No technical explanation there but I like the way they present it  ;)

Concern is definitely not the fact that amount of "free" memory is low but the fact that you are consuming swap and may have some processes slow because of I/O wait (WA). But in any case this is not due to Linux caching mechanism. This one uses only "free" memory.

vshaulsk

  • Zen Samurai
  • ****
  • Posts: 477
  • Karma: +9/-1
    • View Profile
Re: General Linux Server Question
« Reply #8 on: March 14, 2012, 04:47:51 pm »
Right.... I think I have seen this explanation before...

In my case however it shows free of only 1 gig so what happened to the other 9 ???  Even if 75% was used by virtualbox + proxy + filtering ... I should still have 2.5 gigs.

Also why would the system start using swap and so much of it ????  over 2 gigs in swap.....

I do notice during heavy file transfers whether its to the physical raid6 or to the virtual servers..... I start getting heavier loads and also the response time goes way down.

Do you think adding more ram will solve my problem???  How would I plan ahead of time with regards to how much ram I would need.....????

robb

  • Guest
Re: General Linux Server Question
« Reply #9 on: March 14, 2012, 05:10:47 pm »
Quote
have some processes slow because of I/O wait (WA).

I would look for this. Adding more spindles would lower your I/O wait. I know this is not based on hard facts, but more on my best guess.

Can you in any way check if this is the case, by either adding or removing disks from the array and then check if you see an increase or decrease of swap use?

Sam Graf

  • Guest
Re: General Linux Server Question
« Reply #10 on: March 14, 2012, 05:46:10 pm »
How would I plan ahead of time with regards to how much ram I would need.....????
That's the "million dollar" question. As we've talked over before, a Zentyal server seems naturally to reduce free RAM to the point where at least some swap spaced is used. In other words, it seems not possible to use the the existence of some swap usage to answer the "how much" question.

I've attached a typical scenario, as shown by the Monitor, following a reboot.

vshaulsk

  • Zen Samurai
  • ****
  • Posts: 477
  • Karma: +9/-1
    • View Profile
Re: General Linux Server Question
« Reply #11 on: March 14, 2012, 06:29:38 pm »
robb, I may try subtracting some disks to see if this changes anything.

- I have no problem with linux using free ram as disk cache.  I just don't understand why my system only has 1 gig of disk cache (according to free -m) and why I am using so much swap.

Perhaps my issues have to do more with virtualbox since one of the VM disks for the webserver is located on the raid6 array......

I am afraid that when multiple people are transfering large files ..... the system will come to a crawl because of swap usage.

vshaulsk

  • Zen Samurai
  • ****
  • Posts: 477
  • Karma: +9/-1
    • View Profile
Re: General Linux Server Question
« Reply #12 on: March 16, 2012, 01:38:00 pm »
In a couple of days I am going to try to increase my system ram and see what happens.  I have another server which has 16 gigs (It currently has xenserver on it and I am using it as test bed).  I will use the ram from it and see what happens.

With 16 gigs of ram.... it should leave a lot of head room.  However I have a feeling the system after a week or so of operation will once again start using just as much swap.

Perhaps the high use of swap is caused by virtualbox because I see the highest loads and slowest system response when I transfer large files from one of my virtual servers. The virtualbox itself along with the ubuntu VM is installed on my main OS drive (2x500gig disks in raid1).  However the virtual disk from which I am pulling the data from is actually located on my raid6 array.... its a 100gig virtualdisk......(Hope that makes sense)LOL