Zentyal 4.2 - BUG: soft lockup - CPU #1, after latest update

[phaidros]

This kernel helped me: linux-image-generic-lts-xenial.

Code: [Select]

apt-get install linux-image-generic-lts-xenial
Running 4.4.0.13.7 since ~2 weeks with no crashes.

hth,
.phai

[LaM]

xenial (oldlibs): Generic Linux kernel image (dummy transitional package), 4.4.0.16.17: amd64 i386
Which can be seen here http://packages.ubuntu.com/xenial/linux-image-generic-lts-xenial

@phaidros, with which kernel?

Thx btw

L

[pcready.cl]

Kernel
linux-image-3.19.0-56-generic

S0x, too many kernel panic.

Kernel
linux-image-3.19.0-58-generic

Solved the problems to me! 

[Andreas Wirth]

Hi Carlos,

is this for sure and officially confirmed?
Has somebody else positive feedback towards the version: linux-image-3.19.0-58-generic?
I realized the kernel-version was delivered to our productive system at the 06.04.2016. But for me this bug always occurred after a couple of days uptime, and not immediately recognizable.

I mean there are other Debian/Ubuntu forked distributions, who suffered the same issue.
But there at least the official fix was delivered relatively promptly:
E.g. https://forge.univention.org/bugzilla/show_bug.cgi?id=40558

Cheers,
Andreas

[jwilliams1976]

FYI
This is a kernel bug https://bugs.launchpad.net/ubuntu/+source/linux-lts-utopic/+bug/1514785. You can test whether it's fixed by running the command: 'ip rule show' It should just spit out the rules and exit but on any versions with the bug it just loops and never exits. Zentyal must use this command somewhere and after a while it eats up all CPU and memory resources and results in the CPU soft hang.

Quick way to test it instead of waiting a week for Zentyal to crap out.

[hotsummer55]

FYI
This is a kernel bug https://bugs.launchpad.net/ubuntu/+source/linux-lts-utopic/+bug/1514785. You can test whether it's fixed by running the command: 'ip rule show' It should just spit out the rules and exit but on any versions with the bug it just loops and never exits. Zentyal must use this command somewhere and after a while it eats up all CPU and memory resources and results in the CPU soft hang.

Quick way to test it instead of waiting a week for Zentyal to crap out.

Not sure about this .I tested this against know bad kernel linux-image-3.19.0-49-generic.And it did not produce any problems when running ip rule show.
What kernel are you running now

[pcready.cl]

FYI
This is a kernel bug https://bugs.launchpad.net/ubuntu/+source/linux-lts-utopic/+bug/1514785. You can test whether it's fixed by running the command: 'ip rule show' It should just spit out the rules and exit but on any versions with the bug it just loops and never exits. Zentyal must use this command somewhere and after a while it eats up all CPU and memory resources and results in the CPU soft hang.

Quick way to test it instead of waiting a week for Zentyal to crap out.

Not sure about this .I tested this against know bad kernel linux-image-3.19.0-49-generic.And it did not produce any problems when running ip rule show.
What kernel are you running now

use the command on the kernel  linux-image-3.19.0-56-generic and nothing happened, and that is an affected version according to the forums... 

[Andreas Wirth]

@jwilliams1976:
Na' sorry, I don't think, that your mentioned bug https://bugs.launchpad.net/ubuntu/+source/linux-lts-utopic/+bug/1514785 has got anything to do with it.

• There's nowhere mentioned, that a CPU soft lockup is occurring
• There's only mentioned, that it messes up the rules table, which of course might be fatal and messing up the system's operational status as well

It might be a bug to keep an eye on, hopefully we don't get affected as well. (Don't need another one!)

I do believe this bug is related to samba (smbd) in combination with the kernel. (I bet if you turn off smbd, the bug disappears)
But it is occurring in and affecting obviously several kernel versions:
E.g. for the kernel 3.13.0-77:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1543980

But also for the kernel in UCS, what I mentioned before, for the kernel 4.1.16 in this bug:
https://forge.univention.org/bugzilla/show_bug.cgi?id=40558

So I still better stick to 3.19.0.47 in Zentyal for the moment, which seems to do the job for now... until somebody confirms that 3.19.0-58 is working properly for him/her.
Or the proper quick test to confirm, that the bug is gone. Like I mentioned before, for me it always took a couple of days, 6 usually in average, until the system crashed.
And running with 3.19.0.47, I realise, that the system frees memory from time to time (e.g. over night), instead of putting continuously on top, until this CPU lockup occurs and the killing of processes starts.
(Sorry our system is productive, and I can't mess around with it... anymore)

But please keep your experiences up2date here in this thread, if you've got a test system running, that reproduces this bug.
Have much thanks to everybody in advance...

@Carlos: Is your system still running alright with 3.19.0-58? Please keep us up2date...

[update]
Obviously Fedora 23 with kernel 4.4.3 runs into the same bug, reported by this user running on a cubietruck system:
http://www.cubieforums.com/index.php?topic=4076.0
But he or she restricts its occurrence to a high network IO in general via 'smb, scp, or rsync over ssh', but on the opposite the CPU lockup is always logged towards a smbd process.

[LaM]

Hey guys,

have anyone found a way to force the issue?

I'm running fine on the only updated machine which runs the .56 kernel....quite strange (now that I've said that hell will run on that machine  )
uname -a
Linux dccharlie 3.19.0-56-generic #62~14.04.1-Ubuntu SMP Fri Mar 11 11:03:15 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
uptime
 08:49:24 up 14 days, 10:45,  1 user,  load average: 0.25, 0.17, 0.15

Thx

L

[pcready.cl]

@jwilliams1976:
Na' sorry, I don't think, that your mentioned bug https://bugs.launchpad.net/ubuntu/+source/linux-lts-utopic/+bug/1514785 has got anything to do with it.

• There's nowhere mentioned, that a CPU soft lockup is occurring
• There's only mentioned, that it messes up the rules table, which of course might be fatal and messing up the system's operational status as well

It might be a bug to keep an eye on, hopefully we don't get affected as well. (Don't need another one!)

I do believe this bug is related to samba (smbd) in combination with the kernel. (I bet if you turn off smbd, the bug disappears)
But it is occurring in and affecting obviously several kernel versions:
E.g. for the kernel 3.13.0-77:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1543980

But also for the kernel in UCS, what I mentioned before, for the kernel 4.1.16 in this bug:
https://forge.univention.org/bugzilla/show_bug.cgi?id=40558

So I still better stick to 3.19.0.47 in Zentyal for the moment, which seems to do the job for now... until somebody confirms that 3.19.0-58 is working properly for him/her.
Or the proper quick test to confirm, that the bug is gone. Like I mentioned before, for me it always took a couple of days, 6 usually in average, until the system crashed.
And running with 3.19.0.47, I realise, that the system frees memory from time to time (e.g. over night), instead of putting continuously on top, until this CPU lockup occurs and the killing of processes starts.
(Sorry our system is productive, and I can't mess around with it... anymore)

But please keep your experiences up2date here in this thread, if you've got a test system running, that reproduces this bug.
Have much thanks to everybody in advance...

@Carlos: Is your system still running alright with 3.19.0-58? Please keep us up2date...

[update]
Obviously Fedora 23 with kernel 4.4.3 runs into the same bug, reported by this user running on a cubietruck system:
http://www.cubieforums.com/index.php?topic=4076.0
But he or she restricts its occurrence to a high network IO in general via 'smb, scp, or rsync over ssh', but on the opposite the CPU lockup is always logged towards a smbd process.

Code: [Select]

root@servet:~# dmidecode | grep "^System Information" -A8
System Information
        Manufacturer: HP
        Product Name: ProLiant ML150 G6
        Version: 1.0
        Serial Number: MXS108003W
        UUID: 745FC10B-XXXX-DF11-XXXX-C192EAA48B93
        Wake-up Type: Power Switch
        SKU Number: 466132-001
        Family: ProLiant Server

root@servet:~# uname -a
Linux servet 3.19.0-58-generic #64~14.04.1-Ubuntu SMP Fri Mar 18 19:05:43 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

root@servet:~# uptime
 12:35:34 up 3 days, 20:19,  1 user,  load average: 0,03, 0,10, 0,08

Code: [Select]

root@servpcr-fw:~# dmidecode | grep "^System Information" -A8
System Information
        Manufacturer: HP
        Product Name: ProLiant ML110 G5
        Version:      NA
        Serial Number: MX2014011G
        UUID: 44F48208-XXXX-5606-XXXX-560649F92209
        Wake-up Type: Power Switch
        SKU Number: AT040A
        Family: 1234567890

root@servpcr-fw:~# uname -a
Linux servpcr-fw 3.19.0-56-generic #62~14.04.1-Ubuntu SMP Fri Mar 11 11:03:15 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

root@servpcr-fw:~# uptime
 12:40:31 up 5 days, 12:40,  1 user,  load average: 0,16, 0,36, 0,31

[spott]

pcready.cl - does you have virtualized servers? Mainly have her problems when Zentyal is running in VPS - at least my server is virtualized.

[pcready.cl]

pcready.cl - does you have virtualized servers? Mainly have her problems when Zentyal is running in VPS - at least my server is virtualized.

They are all dedicated servers.

But I have them running Windows virtual machines with VirtualBox 5.0.16.

The failure is random, at least .58 are the server kernel and has had no problems.

Instead the other server with .56 kernel has never presented me problems.

The truth is not like nor reproduce the problem that is caused, both servers are in production.

.56 Kernel which has no active users samba, only used as Firewall, perhaps why it has not failed.

The server currently has the .58 kernel before the kernel had .56 and had active users in samba, about 15 concurrent users. And he had problems once a week, once twice a day.

Since the upgrade to version .58 I have not had more problems, so I think the .56 kernel without users samba does not fail, but when you already have access samba presents problems.

If the kernel fails .58 it will report immediately, greetings!

[LaM]

Nice...so 58 looks stable...

But waiting for the issue to come...isn't there a way to force the issue?

L

[pcready.cl]

Nice...so 58 looks stable...

But waiting for the issue to come...isn't there a way to force the issue?

L

Wait to see if one of my servers fails in version .58 or .56 and you commented how it goes fails, I think it is best to wait at least a week.

But the truth is not as forcing or reproduce the error in order to deliver a more concrete report.

I will report this way. Regards!

[LaM]

That's my point. I would like to find a way to reproduce the issue in order to be sure that is gone from the installed kernel.
Waiting is not the correct option imo. It doesn't give You the assurance that the kernel is bug-free
E.g. mine run with .51 and .56 and had been well for days...more than a week (and then one started to crush...)

Honestly I'm still trying to figure how to reproduce it. Looks latched to some concurrency with samba's calls...but i'm not sure.

I'll update You all asa i've more infos...

L