Zentyal Forum, Linux Small Business Server
Zentyal Server => Directory and Authentication => Topic started by: allitsk on October 05, 2016, 12:17:08 pm
-
Hi all,
we have a problem on one production server zentyal 4.2.2 running as PDC, VPN server and file server (no exchange) with high memory usage of one samba process (not smbd). Memory usage builds up and takes around 4-5 days till it eats all memory (2GB RAM and 2GB swap) and then the server freeezes. Before it has enough memory, #service zentyal samba restart helps sometimes and sometimes helps only kill -9.
it looks like this in top:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2913 root 20 0 2305860 1,027g 3692 D 0,3 52,6 111:09.30 samba
..................
When I checked #lsof |grep 2913
I see many open pipes:
............
samba 2913 root 1022w FIFO 0,9 0t0 4045559 pipe
samba 2913 root 1023w FIFO 0,9 0t0 4045570 pipe
............
999 according to wc vs 8 in other samba process
I couldn't find any error in /var/log/samba/samba.log but these lines are occuring every minute:
[2016/10/05 11:56:31.676466, 2] ../lib/util/modules.c:196(do_smb_load_module)
Module 'samba4' loaded
[2016/10/05 11:56:31.880586, 2] ../lib/util/modules.c:196(do_smb_load_module)
Module 'acl_xattr' loaded
[2016/10/05 11:56:31.892361, 2] ../lib/util/modules.c:196(do_smb_load_module)
Module 'dfs_samba4' loaded
[2016/10/05 11:56:31.892425, 2] ../source3/modules/vfs_acl_xattr.c:193(connect_acl_xattr)
connect_acl_xattr: setting 'inherit acls = true' 'dos filemode = true' and 'force unknown acl user = true' for service IPC$
I did ps tree to see the rest samba processes:
#ps auxf|less
root 2903 0.0 0.2 538740 4468 ? Ss okt02 0:00 samba -D
root 2905 0.0 0.1 538740 3464 ? S okt02 0:00 \_ samba -D
root 2908 0.0 0.1 523692 3372 ? Ss okt02 0:09 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 2921 0.0 0.0 507312 1344 ? S okt02 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 2927 0.0 0.3 523728 6380 ? S okt02 0:02 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 4886 0.0 0.1 546664 2840 ? S okt02 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 6054 0.0 0.2 546768 4964 ? S okt02 0:01 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 6327 0.0 0.3 551412 7988 ? S okt03 0:01 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 3837 0.0 0.1 546808 3488 ? S 03:59 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 21028 0.0 0.2 551348 4936 ? S 07:21 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 26144 0.0 0.4 550960 8220 ? S 08:31 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 26599 0.0 0.3 551348 7948 ? S 08:38 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
michael+ 26967 0.0 0.4 552408 8504 ? S 08:44 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 27697 0.0 0.4 552396 8752 ? S 08:55 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 27842 0.0 0.2 546808 5716 ? S 08:55 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 27947 0.0 0.2 551384 6108 ? S 08:56 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 27973 0.0 0.3 551348 6356 ? S 08:57 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 27986 0.0 0.4 551348 8536 ? S 08:57 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 28953 0.0 0.4 551360 8592 ? S 09:10 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 29401 0.0 0.4 552408 8992 ? S 09:16 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
marek.c+ 4371 0.0 0.6 552408 14108 ? S 11:03 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 7195 0.0 0.6 547196 12636 ? S 11:44 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 2906 0.0 1.0 549600 21148 ? S okt02 1:22 \_ samba -D
root 2907 0.0 0.2 542892 5444 ? S okt02 0:09 \_ samba -D
root 2909 0.0 0.1 538740 3492 ? S okt02 0:00 \_ samba -D
root 2910 0.5 3.6 574872 74656 ? S okt02 21:29 \_ samba -D
root 2911 0.0 0.4 538732 10000 ? S okt02 0:13 \_ samba -D
root 2912 0.0 0.8 549160 17616 ? S okt02 0:54 \_ samba -D
root 2913 2.8 55.0 2311432 1128168 ? D okt02 111:17 \_ samba -D
root 2914 0.0 0.4 538732 9056 ? S okt02 0:00 \_ samba -D
root 2915 0.0 0.8 542884 16964 ? S okt02 0:46 \_ samba -D
root 2916 0.0 0.4 538740 9040 ? S okt02 0:07 \_ samba -D
root 2917 0.0 0.1 538740 3532 ? S okt02 0:00 \_ samba -D
root 2918 0.0 0.3 487780 6340 ? Ss okt02 0:08 \_ /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
root 2923 0.0 0.3 487224 7692 ? S okt02 0:04 \_ /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
root 2925 0.0 0.3 487764 6544 ? S okt02 0:04 \_ /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
root 2926 0.0 0.4 487780 9796 ? S okt02 0:09 \_ /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
And as you can see, problematic is just one process (pid 2913) rest of them are ok.
Zentyal is regularly updated and configuration is almost default, it is running zentyal 4.2.2 with samba --version: Version 4.3.4-Zentyal on Ubuntu 14.04.1 and kernel 3.19.0-65-generic
#cat /proc/version
Linux version 3.19.0-65-generic (buildd@lgw01-01) (gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3) ) #73~14.04.1-Ubuntu SMP Wed Jun 29 21:05:22 UTC 2016
We have more zentyal servers on different sites, set up similarly (just the companies are different) and we have problem just with this one. Right now we have to restart service every 3-4 days. Do you have any other solution or experience?
-
Hello,
I've the same behavior with Samba (PDC). The service run ok some days and after it uses all memory ram the service is
degraded. The solution is restart samba service (service zentyal samba restart)
Zentyal versión 4.2.2
Samba versión Version 4.3.4-Zentyal
Kernel version 3.19.0-65-generic
Server ram 8GB
VCPU 8
-
See here https://forum.zentyal.org/index.php/topic,27428.0.html (https://forum.zentyal.org/index.php/topic,27428.0.html)
I have the same issue, even with newest kernel 4.x. Just over night and with no clients online, memory sums up to over 2GB.
This is the status only after 16 hours running only as mail server:
root 30492 0.0 2.1 559028 76752 ? Ss 09:28 0:00 samba -D
root 30506 0.0 1.5 559028 53932 ? S 09:29 0:00 \_ samba -D
root 30509 0.0 2.2 546544 81656 ? Ss 09:29 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 30523 0.0 1.4 530164 53656 ? S 09:29 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 30538 0.0 1.5 546544 54720 ? S 09:29 0:00 | \_ /usr/sbin/smbd -D --option=server role check:inhibit=yes --foreground
root 30507 0.0 1.9 690384 69244 ? S 09:29 0:00 \_ samba -D
root 30508 0.0 1.7 563180 61924 ? S 09:29 0:00 \_ samba -D
root 30510 0.0 1.5 559028 53932 ? S 09:29 0:00 \_ samba -D
root 30511 0.5 2.0 565316 73184 ? S 09:29 1:12 \_ samba -D
root 30512 0.0 1.5 559028 54524 ? S 09:29 0:00 \_ samba -D
root 30513 0.0 1.8 569500 66020 ? S 09:29 0:01 \_ samba -D
root 30514 7.0 15.6 1062944 563008 ? S 09:29 16:31 \_ samba -D
root 26426 0.0 0.0 0 0 ? R 13:22 0:00 | \_ [samba]
root 30515 0.0 1.5 559028 53932 ? S 09:29 0:00 \_ samba -D
root 30516 0.0 2.0 563180 74952 ? S 09:29 0:08 \_ samba -D
root 30517 0.0 1.5 559028 57140 ? S 09:29 0:00 \_ samba -D
root 30518 0.0 1.5 559028 53932 ? S 09:29 0:00 \_ samba -D
root 30520 0.0 2.1 498160 76836 ? Ss 09:29 0:01 \_ /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
root 30524 0.0 1.7 510128 61972 ? S 09:29 0:00 \_ /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
root 30527 0.0 1.6 498160 58204 ? S 09:29 0:00 \_ /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
root 30529 0.0 1.5 498160 54504 ? S 09:29 0:00 \_ /usr/sbin/winbindd -D --option=server role check:inhibit=yes --foreground
It's pid 30514 now already at 15% of the memory. I have samba 4.3.4 running with 4.4.0-38 kernel, so it's still a persistent problem.
-
allitsk,
Can you please check your samba.log if you find this error:
/usr/sbin/samba_dnsupdate: dns_tkey_negotiategss: TKEY is unacceptable
In case, please check /var/lib/samba/private/dns.keytab if it root:named / 644?
In my case it was root:root 640.
After correcting this, error is gone and - although I still need to check a longer period, it seems the samba process does not grow larger than roughly 300k (used) after 7 hrs.
root 8549 5.9 8.6 810420 310324 ? S 14:18 7:25 \_ samba -D
UDATE: OK, maybe I was way to optimistic. It's now slowly building up again, already at >700K. :-(
-
Hi segelfreak, all
it is great that we are not alone with this issue. As I stated we have more similar zentyal servers (same zentyal version, same samba version, same kernel different users) and just one has this problem.
I checked the dns.keytab and error in logs, but i didn't find any of this error and my dns.keytab looks fine:
-rw-r----- 1 root bind 797 okt 28 2015 dns.keytab
I did further investigation what is different with these processes and found util pmap (which lists memory usage of process), but i am not any wiser:
It looks like this:
# pmap -X 15098
15098: samba -D
Address Perm Offset Device Inode Size Rss Pss Referenced Anonymous Swap Locked Mapping
5609461e2000 r-xp 00000000 fd:01 1193099 60 60 4 60 0 0 0 samba
5609463f1000 r--p 0000f000 fd:01 1193099 4 4 0 0 4 0 0 samba
5609463f2000 rw-p 00010000 fd:01 1193099 4 0 0 0 0 4 0 samba
56094698d000 rw-p 00000000 00:00 0 132 72 35 48 72 60 0 [heap]
5609469ae000 rw-p 00000000 00:00 0 28724 26780 25686 25992 26780 1944 0 [heap]
5609485bb000 rw-p 00000000 00:00 0 752248 752228 752228 719872 752228 0 0 [heap]
...................
and problematic is the heap which uses actually 752MB RAM out of 1,3GB of whole process and getting bigger. When I used pmap -XX i didn't get more info, and with pmap -x i just received info that it is anonymous and "dirty". It looks like samba is using memory for something and doesn't cleanup. I can't get any more info.
If you have any tips which tools use to troubleshoot, i am open to any suggestions. Till that, we need to restart the samba service every 3 days.
-
I wonder why this cannot be found at any of the official samba mailing groups, yet?
If it were only every 3 days, I would be more relaxed, but the system im running this on has only 4GB and 50% of it is gone within a single day, sometimes even less.
This really keeps me from using zentyal on any additional machines, although I find it a nice piece of cake in general.
It's not that one could blame zentyal as such for it, but at the end, it's not reliable and therefore not usable in a professional condition. It's very sad.
Frankly, I'm already spending too much time with this issue, it's annoying.
-
As I stated we have more similar zentyal servers (same zentyal version, same samba version, same kernel different users) and just one has this problem.
Actually I also tried installation on different systems, such as an AMD on an HP platform, Intel dual Core on Lenovo IBM and now finally Pentium D on an spare HP Proliant server. Certainly completely different chip sets, however, always the same core installation (I just switched the disk).
I found this on all three machines, so I was pretty sure that it should be not hardware related. Following your statement, I wonder what would happen if one makes a completely new installation and then uploads a backup to see if it can be tracked down to an installation issue?
-
One other thing I wonder about:
Why does zentyal build its own samba package?
samba --show-build
Samba version: 4.3.4-Zentyal
Build environment:
Build host: Linux ci01.hq.zentyal.com 3.13.0-77-generic #121-Ubuntu SMP Wed Jan 20 10:50:42 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Is that necessary and what is the difference from the ubuntu stock package provided?
Actually I'm wondering because this problem seems somehow limited to this relatively small "zentyal world".
What if.... this package contains some errors?
What would happen if one would install a stock package over the zentyal one?
Ubuntu official version of samba package seems to be 4.3.11 now, right?
-
Hello,
Search in the Samba list but i dont found similar case. In the next thread people have similar problem (memory, cpu) with the Samba 4.x an kernel 3.19.x.x and the solution is downgrade the kernel to 3.19.0-43 https://forum.zentyal.org/index.php/topic,27362.60.html .
Regards.
-
I know this thread well, but it's something completely different. There was a soft lock on a core, here we look at a memory issue.
I can also confirm the softlock is gone using the up-to-date kernel.
-
I did further investigation what is different with these processes and found util pmap (which lists memory usage of process), but i am not any wiser:
It looks like this:
# pmap -X 15098
15098: samba -D
Address Perm Offset Device Inode Size Rss Pss Referenced Anonymous Swap Locked Mapping
5609461e2000 r-xp 00000000 fd:01 1193099 60 60 4 60 0 0 0 samba
5609463f1000 r--p 0000f000 fd:01 1193099 4 4 0 0 4 0 0 samba
5609463f2000 rw-p 00010000 fd:01 1193099 4 0 0 0 0 4 0 samba
56094698d000 rw-p 00000000 00:00 0 132 72 35 48 72 60 0 [heap]
5609469ae000 rw-p 00000000 00:00 0 28724 26780 25686 25992 26780 1944 0 [heap]
5609485bb000 rw-p 00000000 00:00 0 752248 752228 752228 719872 752228 0 0 [heap]
...................
and problematic is the heap which uses actually 752MB RAM out of 1,3GB of whole process and getting bigger. When I used pmap -XX i didn't get more info, and with pmap -x i just received info that it is anonymous and "dirty". It looks like samba is using memory for something and doesn't cleanup. I can't get any more info.
If you have any tips which tools use to troubleshoot, i am open to any suggestions. Till that, we need to restart the samba service every 3 days.
Well, I can confirm this finding.
Here's my output after roughly 24 hrs running.
Address Perm Offset Device Inode Size Rss Pss Referenced Anonymous Shared_Hugetlb Private_Hugetlb Swap SwapPss Locked Mapping
55db84809000 r-xp 00000000 fc:00 7608422 60 56 4 56 0 0 0 0 0 0 samba
55db84a18000 r--p 0000f000 fc:00 7608422 4 0 0 0 0 0 0 4 0 0 samba
55db84a19000 rw-p 00010000 fc:00 7608422 4 0 0 0 0 0 0 4 0 0 samba
55db84ad9000 rw-p 00000000 00:00 0 132 60 20 48 60 0 0 72 8 0 [heap]
55db84afa000 rw-p 00000000 00:00 0 48424 45320 21474 31304 45320 0 0 3104 779 0 [heap]
55db87a44000 rw-p 00000000 00:00 0 2358428 2037716 1023042 1336160 2037716 0 0 320712 160368 0 [heap]
Address Perm Offset Device Inode Size Rss Pss Shared_Clean Shared_Dirty Private_Clean Private_Dirty Referenced Anonymous AnonHugePages Shared_Hugetlb Private_Hugetlb Swap SwapPss KernelPageSize MMUPageSize Locked VmFlagsMapping
55db87a44000 rw-p 00000000 00:00 0 2361952 1999132 1999132 0 0 8904 1990228 1339564 1999132 382976 0 0 362820 362820 4 4 0 rd wr mr mw me ac sd [heap]
The last line was extracted form using -XX to get more information, but I'm afraid this is way too far beyond my skills.
However, I understand that samba seems to build an internal data collection (heap), which is not limiting itself and constantly grows.
I read that there was a serious vulnerability in versions before 4.3.2, which also was related to problematic heap buffer memory overflow. But after this, there's no "heap" mentioned in any later change log.
-
Just an idea:
Does your server change its behavior if you'd re-condition the AD, without changing anything else...?
I was just thinking if the heap data may be linked to DNS updates, causing error. (2nd DC, dead DC's, etc)
So, what would happen with a freshly configured and mainly empty AD?
update:
I did some trials and set up a new (virtual) server on a separate machine. Installed all the software, modules, etc and set up a new AD. Memory didn't show any increase.
After one day I made a recovery from a backup. Same machine, same software releases. And... memory increase is back.
This really seems to turn out to be an AD related issue.
Update2:
Cleaned up the AD, using LDAP Admin tool Professional (trial).
Deleted all dead DC's, related replication settings, etc. In short: Any sign of non-existing other DC's.
after 2 hours, it looks like memory stays perfectly low.
Update3: After one full day, memory still good. I consider this solved.
-
Update2:
Cleaned up the AD, using LDAP Admin tool Professional (trial).
Deleted all dead DC's, related replication settings, etc. In short: Any sign of non-existing other DC's.
after 2 hours, it looks like memory stays perfectly low.
Update3: After one full day, memory still good. I consider this solved.
Hi,
I have exactly the same problem but in a Zentyal 3.0 version.
Can you describe the steps to do that in LDAP Admin Tool? I have searched the hostname of my old DC with "cn" option and I've deleted 3 entries in the tree, but I still can select it on the top right corner as a Domain Controller. I'm not sure if its enough or if I'm missing something.
EDIT: I think it's solved. Thanks a lot!