Author Topic: Zentyal 4.2.2 samba high memory usage  (Read 2195 times)

allitsk

  • Zen Apprentice
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Zentyal 4.2.2 samba high memory usage
« on: October 05, 2016, 12:17:08 pm »
Hi all,
we have a problem on one production server zentyal 4.2.2 running as PDC, VPN server and file server (no exchange) with high memory usage of one samba process (not smbd). Memory usage builds up and takes around 4-5 days till it eats all memory (2GB RAM and 2GB swap) and then the server freeezes. Before it has enough memory, #service zentyal samba restart helps sometimes and sometimes helps only kill -9.

it looks like this in top:
 
Code: [Select]
PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2913 root      20   0 2305860 1,027g   3692 D   0,3 52,6 111:09.30 samba
..................

When I checked #lsof |grep 2913
I see many open pipes:
Code: [Select]
         ............
samba      2913                   root 1022w     FIFO                0,9       0t0    4045559 pipe
samba      2913                   root 1023w     FIFO                0,9       0t0    4045570 pipe
         ............

999 according to wc vs 8 in other samba process

I couldn't find any error in /var/log/samba/samba.log but these lines are occuring every minute:
Code: [Select]
[2016/10/05 11:56:31.676466,  2] ../lib/util/modules.c:196(do_smb_load_module)
  Module 'samba4' loaded
[2016/10/05 11:56:31.880586,  2] ../lib/util/modules.c:196(do_smb_load_module)
  Module 'acl_xattr' loaded
[2016/10/05 11:56:31.892361,  2] ../lib/util/modules.c:196(do_smb_load_module)
  Module 'dfs_samba4' loaded
[2016/10/05 11:56:31.892425,  2] ../source3/modules/vfs_acl_xattr.c:193(connect_acl_xattr)
  connect_acl_xattr: setting 'inherit acls = true' 'dos filemode = true' and 'force unknown acl user = true' for service IPC$

I did ps tree to see the rest samba processes:
Code: [Select]
#ps auxf|less
root      2903  0.0  0.2 538740  4468 ?        Ss   okt02   0:00 samba -D
root      2905  0.0  0.1 538740  3464 ?        S    okt02   0:00  \_ samba -D
root      2908  0.0  0.1 523692  3372 ?        Ss   okt02   0:09  |   \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root      2921  0.0  0.0 507312  1344 ?        S    okt02   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root      2927  0.0  0.3 523728  6380 ?        S    okt02   0:02  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root      4886  0.0  0.1 546664  2840 ?        S    okt02   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root      6054  0.0  0.2 546768  4964 ?        S    okt02   0:01  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root      6327  0.0  0.3 551412  7988 ?        S    okt03   0:01  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root      3837  0.0  0.1 546808  3488 ?        S    03:59   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     21028  0.0  0.2 551348  4936 ?        S    07:21   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     26144  0.0  0.4 550960  8220 ?        S    08:31   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     26599  0.0  0.3 551348  7948 ?        S    08:38   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
michael+ 26967  0.0  0.4 552408  8504 ?        S    08:44   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     27697  0.0  0.4 552396  8752 ?        S    08:55   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     27842  0.0  0.2 546808  5716 ?        S    08:55   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     27947  0.0  0.2 551384  6108 ?        S    08:56   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     27973  0.0  0.3 551348  6356 ?        S    08:57   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     27986  0.0  0.4 551348  8536 ?        S    08:57   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     28953  0.0  0.4 551360  8592 ?        S    09:10   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     29401  0.0  0.4 552408  8992 ?        S    09:16   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
marek.c+  4371  0.0  0.6 552408 14108 ?        S    11:03   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root      7195  0.0  0.6 547196 12636 ?        S    11:44   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root      2906  0.0  1.0 549600 21148 ?        S    okt02   1:22  \_ samba -D
root      2907  0.0  0.2 542892  5444 ?        S    okt02   0:09  \_ samba -D
root      2909  0.0  0.1 538740  3492 ?        S    okt02   0:00  \_ samba -D
root      2910  0.5  3.6 574872 74656 ?        S    okt02  21:29  \_ samba -D
root      2911  0.0  0.4 538732 10000 ?        S    okt02   0:13  \_ samba -D
root      2912  0.0  0.8 549160 17616 ?        S    okt02   0:54  \_ samba -D
root      2913  2.8 55.0 2311432 1128168 ?     D    okt02 111:17  \_ samba -D
root      2914  0.0  0.4 538732  9056 ?        S    okt02   0:00  \_ samba -D
root      2915  0.0  0.8 542884 16964 ?        S    okt02   0:46  \_ samba -D
root      2916  0.0  0.4 538740  9040 ?        S    okt02   0:07  \_ samba -D
root      2917  0.0  0.1 538740  3532 ?        S    okt02   0:00  \_ samba -D
root      2918  0.0  0.3 487780  6340 ?        Ss   okt02   0:08      \_ /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
root      2923  0.0  0.3 487224  7692 ?        S    okt02   0:04          \_ /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
root      2925  0.0  0.3 487764  6544 ?        S    okt02   0:04          \_ /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
root      2926  0.0  0.4 487780  9796 ?        S    okt02   0:09          \_ /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground

And as you can see, problematic is just one process (pid 2913) rest of them are ok.

Zentyal is regularly updated and configuration is almost default, it is running zentyal 4.2.2 with samba --version: Version 4.3.4-Zentyal on Ubuntu 14.04.1 and kernel 3.19.0-65-generic
Code: [Select]
#cat /proc/version
Linux version 3.19.0-65-generic (buildd@lgw01-01) (gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3) ) #73~14.04.1-Ubuntu SMP Wed Jun 29 21:05:22 UTC 2016
We have more zentyal servers on different sites, set up similarly (just the companies are different) and we have problem just with this one. Right now we have to restart service every 3-4 days. Do you have any other solution or experience?
« Last Edit: October 10, 2016, 04:24:13 pm by allitsk »

mcoa

  • Zen Apprentice
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: Zentyal 4.2.2 samba high memory usage
« Reply #1 on: October 05, 2016, 07:10:41 pm »
Hello,
I've the same behavior with Samba (PDC). The service run ok some days and after it uses all memory ram the service is
degraded. The solution is restart samba service (service zentyal samba restart)

Quote
Zentyal versión 4.2.2
Samba versión Version 4.3.4-Zentyal
Kernel version 3.19.0-65-generic
Server ram 8GB
VCPU 8

segelfreak

  • Zen Monk
  • **
  • Posts: 70
  • Karma: +4/-0
    • View Profile
Re: Zentyal 4.2.2 samba high memory usage
« Reply #2 on: October 09, 2016, 10:43:06 am »
See here https://forum.zentyal.org/index.php/topic,27428.0.html

I have the same issue, even with newest kernel 4.x. Just over night and with no clients online, memory sums up to over 2GB.

This is the status only after 16 hours running only as mail server:
Code: [Select]
root     30492  0.0  2.1 559028 76752 ?        Ss   09:28   0:00 samba -D
root     30506  0.0  1.5 559028 53932 ?        S    09:29   0:00  \_ samba -D
root     30509  0.0  2.2 546544 81656 ?        Ss   09:29   0:00  |   \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     30523  0.0  1.4 530164 53656 ?        S    09:29   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     30538  0.0  1.5 546544 54720 ?        S    09:29   0:00  |       \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root     30507  0.0  1.9 690384 69244 ?        S    09:29   0:00  \_ samba -D
root     30508  0.0  1.7 563180 61924 ?        S    09:29   0:00  \_ samba -D
root     30510  0.0  1.5 559028 53932 ?        S    09:29   0:00  \_ samba -D
root     30511  0.5  2.0 565316 73184 ?        S    09:29   1:12  \_ samba -D
root     30512  0.0  1.5 559028 54524 ?        S    09:29   0:00  \_ samba -D
root     30513  0.0  1.8 569500 66020 ?        S    09:29   0:01  \_ samba -D
root     30514  7.0 15.6 1062944 563008 ?      S    09:29  16:31  \_ samba -D
root     26426  0.0  0.0      0     0 ?        R    13:22   0:00  |   \_ [samba]
root     30515  0.0  1.5 559028 53932 ?        S    09:29   0:00  \_ samba -D
root     30516  0.0  2.0 563180 74952 ?        S    09:29   0:08  \_ samba -D
root     30517  0.0  1.5 559028 57140 ?        S    09:29   0:00  \_ samba -D
root     30518  0.0  1.5 559028 53932 ?        S    09:29   0:00  \_ samba -D
root     30520  0.0  2.1 498160 76836 ?        Ss   09:29   0:01      \_ /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
root     30524  0.0  1.7 510128 61972 ?        S    09:29   0:00          \_ /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
root     30527  0.0  1.6 498160 58204 ?        S    09:29   0:00          \_ /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
root     30529  0.0  1.5 498160 54504 ?        S    09:29   0:00          \_ /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground

It's pid 30514 now already at 15% of the memory. I have samba 4.3.4 running with 4.4.0-38 kernel, so it's still a persistent problem.
« Last Edit: October 09, 2016, 01:32:43 pm by segelfreak »

segelfreak

  • Zen Monk
  • **
  • Posts: 70
  • Karma: +4/-0
    • View Profile
Re: Zentyal 4.2.2 samba high memory usage
« Reply #3 on: October 09, 2016, 04:24:42 pm »
allitsk,

Can you please check your samba.log if you find this error:

Code: [Select]
/usr/sbin/samba_dnsupdate: dns_tkey_negotiategss: TKEY is unacceptable
In case, please check /var/lib/samba/private/dns.keytab if it root:named / 644?
In my case it was root:root 640.
After correcting this, error is gone and - although I still need to check a longer period, it seems the samba process does not grow larger than roughly 300k (used) after 7 hrs.

Code: [Select]
root      8549  5.9  8.6 810420 310324 ?       S    14:18   7:25  \_ samba -D

UDATE: OK, maybe I was way to optimistic. It's now slowly building up again, already at >700K. :-(
« Last Edit: October 09, 2016, 09:53:15 pm by segelfreak »

allitsk

  • Zen Apprentice
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: Zentyal 4.2.2 samba high memory usage
« Reply #4 on: October 10, 2016, 04:21:41 pm »
Hi segelfreak, all
it is great that we are not alone with this issue. As I stated we have more similar zentyal servers (same zentyal version, same samba version, same kernel different users) and just one has this problem.

I checked the dns.keytab and error in logs, but i didn't find any of this error and my dns.keytab looks fine:
Code: [Select]
-rw-r----- 1 root bind     797 okt 28  2015 dns.keytab
I did further investigation what is different with these processes and found util pmap (which lists memory usage of process), but i am not any wiser:
It looks like this:
Code: [Select]
# pmap -X 15098
15098:   samba -D
         Address Perm   Offset Device   Inode    Size    Rss    Pss Referenced Anonymous Swap Locked Mapping
    5609461e2000 r-xp 00000000  fd:01 1193099      60     60      4         60         0    0      0 samba
    5609463f1000 r--p 0000f000  fd:01 1193099       4      4      0          0         4    0      0 samba
    5609463f2000 rw-p 00010000  fd:01 1193099       4      0      0          0         0    4      0 samba
    56094698d000 rw-p 00000000  00:00       0     132     72     35         48        72   60      0 [heap]
    5609469ae000 rw-p 00000000  00:00       0   28724  26780  25686      25992     26780 1944      0 [heap]
    5609485bb000 rw-p 00000000  00:00       0  752248 752228 752228     719872    752228    0      0 [heap]
                    ...................

and problematic is the heap which uses actually 752MB RAM out of 1,3GB of whole process and getting bigger. When I used pmap -XX i didn't get more info, and with pmap -x i just received info that it is anonymous and "dirty". It looks like samba is using memory for something and doesn't cleanup. I can't get any more info.

If you have any tips which tools use to troubleshoot, i am open to any suggestions. Till that, we need to restart the samba service every 3 days.
« Last Edit: October 10, 2016, 04:34:58 pm by allitsk »

segelfreak

  • Zen Monk
  • **
  • Posts: 70
  • Karma: +4/-0
    • View Profile
Re: Zentyal 4.2.2 samba high memory usage
« Reply #5 on: October 12, 2016, 06:27:58 pm »
I wonder why this cannot be found at any of the official samba mailing groups, yet?

If it were only every 3 days, I would be more relaxed, but the system im running this on has only 4GB and 50% of it is gone within a single day, sometimes even less.

This really keeps me from using zentyal on any additional machines, although I find it a nice piece of cake in general.
It's not that one could blame zentyal as such for it, but at the end, it's not reliable and therefore not usable in a professional condition. It's very sad.

Frankly, I'm already spending too much time with this issue, it's annoying.

segelfreak

  • Zen Monk
  • **
  • Posts: 70
  • Karma: +4/-0
    • View Profile
Re: Zentyal 4.2.2 samba high memory usage
« Reply #6 on: October 12, 2016, 06:34:06 pm »
As I stated we have more similar zentyal servers (same zentyal version, same samba version, same kernel different users) and just one has this problem.

Actually I also tried installation on different systems, such as an AMD on an HP platform, Intel dual Core on Lenovo IBM and now finally Pentium D on an spare HP Proliant server. Certainly completely different chip sets, however, always the same core installation (I just switched the disk).
I found this on all three machines, so I was pretty sure that it should be not hardware related. Following your statement, I wonder what would happen if one makes a completely new installation and then uploads a backup to see if it can be tracked down to an installation issue?

segelfreak

  • Zen Monk
  • **
  • Posts: 70
  • Karma: +4/-0
    • View Profile
Re: Zentyal 4.2.2 samba high memory usage
« Reply #7 on: October 12, 2016, 06:54:49 pm »
One other thing I wonder about:

Why does zentyal build its own samba package?

Code: [Select]
samba --show-build
Samba version: 4.3.4-Zentyal
Build environment:
   Build host:  Linux ci01.hq.zentyal.com 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Is that necessary and what is the difference from the ubuntu stock package provided?
Actually I'm wondering because this problem seems somehow limited to this relatively small "zentyal world".

What if.... this package contains some errors?

What would happen if one would install a stock package over the zentyal one?


Ubuntu official version of samba package seems to be 4.3.11 now, right?
« Last Edit: October 12, 2016, 07:12:44 pm by segelfreak »

mcoa

  • Zen Apprentice
  • *
  • Posts: 7
  • Karma: +0/-0
    • View Profile
Re: Zentyal 4.2.2 samba high memory usage
« Reply #8 on: October 14, 2016, 10:06:27 pm »
Hello,
Search in the Samba list but i dont found similar case. In the next thread people have similar problem (memory, cpu)  with the Samba 4.x an kernel 3.19.x.x and the solution is downgrade the kernel to 3.19.0-43 https://forum.zentyal.org/index.php/topic,27362.60.html .

Regards.

segelfreak

  • Zen Monk
  • **
  • Posts: 70
  • Karma: +4/-0
    • View Profile
Re: Zentyal 4.2.2 samba high memory usage
« Reply #9 on: October 14, 2016, 10:18:55 pm »
I know this thread well, but it's something completely different. There was a soft lock on a core, here we look at a memory issue.
I can also confirm the softlock is gone using the up-to-date kernel.

segelfreak

  • Zen Monk
  • **
  • Posts: 70
  • Karma: +4/-0
    • View Profile
Re: Zentyal 4.2.2 samba high memory usage
« Reply #10 on: October 15, 2016, 01:25:03 pm »
I did further investigation what is different with these processes and found util pmap (which lists memory usage of process), but i am not any wiser:
It looks like this:
Code: [Select]
# pmap -X 15098
15098:   samba -D
    Address Perm   Offset Device   Inode    Size    Rss    Pss Referenced Anonymous Swap Locked Mapping
    5609461e2000 r-xp 00000000  fd:01 1193099      60     60      4         60         0    0      0 samba
    5609463f1000 r--p 0000f000  fd:01 1193099       4      4      0          0         4    0      0 samba
    5609463f2000 rw-p 00010000  fd:01 1193099       4      0      0          0         0    4      0 samba
    56094698d000 rw-p 00000000  00:00       0     132     72     35         48        72   60      0 [heap]
    5609469ae000 rw-p 00000000  00:00       0   28724  26780  25686      25992     26780 1944      0 [heap]
    5609485bb000 rw-p 00000000  00:00       0  752248 752228 752228     719872    752228    0      0 [heap]
                    ...................

and problematic is the heap which uses actually 752MB RAM out of 1,3GB of whole process and getting bigger. When I used pmap -XX i didn't get more info, and with pmap -x i just received info that it is anonymous and "dirty". It looks like samba is using memory for something and doesn't cleanup. I can't get any more info.

If you have any tips which tools use to troubleshoot, i am open to any suggestions. Till that, we need to restart the samba service every 3 days.


Well, I can confirm this finding.
Here's my output after roughly 24 hrs running.

Code: [Select]
Address Perm   Offset    Device   Inode       Size     Rss     Pss Referenced Anonymous Shared_Hugetlb Private_Hugetlb   Swap SwapPss Locked Mapping
55db84809000 r-xp 00000000  fc:00  7608422      60      56       4         56         0              0               0      0       0      0 samba
55db84a18000 r--p 0000f000  fc:00  7608422       4       0       0          0         0              0               0      4       0      0 samba
55db84a19000 rw-p 00010000  fc:00  7608422       4       0       0          0         0              0               0      4       0      0 samba
55db84ad9000 rw-p 00000000  00:00        0     132      60      20         48        60              0               0     72       8      0 [heap]
55db84afa000 rw-p 00000000  00:00        0   48424   45320   21474      31304     45320              0               0   3104     779      0 [heap]
55db87a44000 rw-p 00000000  00:00        0 2358428 2037716 1023042    1336160   2037716              0               0 320712  160368      0 [heap]

      Address Perm   Offset Device    Inode    Size     Rss     Pss Shared_Clean Shared_Dirty Private_Clean Private_Dirty Referenced Anonymous AnonHugePages Shared_Hugetlb Private_Hugetlb   Swap SwapPss KernelPageSize MMUPageSize Locked                   VmFlagsMapping
55db87a44000 rw-p 00000000  00:00        0 2361952 1999132 1999132            0            0          8904       1990228    1339564   1999132        382976              0               0 362820  362820              4           4      0    rd wr mr mw me ac sd  [heap]
   

The last line was extracted form using -XX to get more information, but I'm afraid this is way too far beyond my skills.
However, I understand that samba seems to build an internal data collection (heap), which is not limiting itself and constantly grows.

I read that there was a serious vulnerability in versions before 4.3.2, which also was related to problematic heap buffer memory overflow. But after this, there's no "heap" mentioned in any later change log.
« Last Edit: October 15, 2016, 01:52:59 pm by segelfreak »

segelfreak

  • Zen Monk
  • **
  • Posts: 70
  • Karma: +4/-0
    • View Profile
Re: Zentyal 4.2.2 samba high memory usage
« Reply #11 on: October 15, 2016, 05:44:25 pm »
Just an idea:

Does your server change its behavior if you'd re-condition the AD, without changing anything else...?

I was just thinking if the heap data may be linked to DNS updates, causing error. (2nd DC, dead DC's, etc)
So, what would happen with a freshly configured and mainly empty AD?

update:

I did some trials and set up a new (virtual) server on a separate machine. Installed all the software, modules, etc and set up a new AD. Memory didn't show any increase.

After one day I made a recovery from a backup. Same machine, same software releases. And... memory increase is back.

This really seems to turn out to be an AD related issue.


Update2:
Cleaned up the AD, using LDAP Admin tool Professional (trial).
Deleted all dead DC's, related replication settings, etc. In short: Any sign of non-existing other DC's.
after 2 hours, it looks like memory stays perfectly low.

Update3: After one full day, memory still good. I consider this solved.
« Last Edit: October 17, 2016, 08:06:22 pm by segelfreak »

Maekar

  • Zen Apprentice
  • *
  • Posts: 40
  • Karma: +0/-0
    • View Profile
Re: Zentyal 4.2.2 samba high memory usage
« Reply #12 on: October 20, 2016, 12:51:04 pm »
Update2:
Cleaned up the AD, using LDAP Admin tool Professional (trial).
Deleted all dead DC's, related replication settings, etc. In short: Any sign of non-existing other DC's.
after 2 hours, it looks like memory stays perfectly low.

Update3: After one full day, memory still good. I consider this solved.

Hi,

I have exactly the same problem but in a Zentyal 3.0 version.

Can you describe the steps to do that in LDAP Admin Tool? I have searched the hostname of my old DC with "cn" option and I've deleted 3 entries in the tree, but I still can select it on the top right corner as a Domain Controller. I'm not sure if its enough or if I'm missing something.

EDIT: I think it's solved. Thanks a lot!
« Last Edit: October 20, 2016, 04:02:04 pm by Maekar »