Zentyal 4.2 - BUG: soft lockup - CPU #1, after latest update

[LaM]

I have created a bug on bugtraker
https://tracker.zentyal.org/issues/4977

Kudos! 

Keep us up-2-date plz!

[hotsummer55]

please vote up bug if interested.may get better responce on this.

https://tracker.zentyal.org/issues/4977

[kinetica]

Server DELL
Virtual environment latest Proxmox 4.1 up to date

VM:
Zentyal 4.2.2 up to date used only as SAMBA SERVER
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
Kernel: 3.19.0-56-generic
Linux zensamba 3.19.0-56-generic #62~14.04.1-Ubuntu SMP Fri Mar 11 11:03:15 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux


PROBLEM DESCRIPTION:
zensamba kernel: [49848.028036] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [smbd:1724]
VM Stack we need to Stop and restart


Mar 23 16:37:41 zensamba kernel: [85320.064038] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [smbd:1696]
Mar 23 16:37:41 zensamba kernel: [85320.064038] Modules linked in: iptable_nat nf_nat_ipv4 nf_nat iptable_filter xt_mac xt_mark nf_conntrack_ipv4 nf_defrag_ipv4 xt_connmark nf_conntrack iptable_mangle

ip_tables x_tables quota_v2 quota_tree cirrus ttm drm_kms_helper drm joydev syscopyarea sysfillrect sysimgblt i2c_piix4 ppdev mac_hid shpchp serio_raw 8250_fintek parport_pc lp parport hid_generic

usbhid hid floppy psmouse pata_acpi
Mar 23 16:37:41 zensamba kernel: [85320.064038] CPU: 3 PID: 1696 Comm: smbd Not tainted 3.19.0-56-generic #62~14.04.1-Ubuntu
Mar 23 16:37:41 zensamba kernel: [85320.064038] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
Mar 23 16:37:41 zensamba kernel: [85320.064038] task: ffff8802337875c0 ti: ffff880219448000 task.ti: ffff880219448000
Mar 23 16:37:41 zensamba kernel: [85320.064038] RIP: 0010:[<ffffffff8105b976>]  [<ffffffff8105b976>] native_safe_halt+0x6/0x10
Mar 23 16:37:41 zensamba kernel: [85320.064038] RSP: 0018:ffff88021944bd78  EFLAGS: 00000206
Mar 23 16:37:41 zensamba kernel: [85320.064038] RAX: 000000000000003b RBX: 00000000000000b0 RCX: 0000000000000001
Mar 23 16:37:41 zensamba kernel: [85320.064038] RDX: 0000000000000000 RSI: 0000000000008550 RDI: ffff88023fff20c0
Mar 23 16:37:41 zensamba kernel: [85320.064038] RBP: ffff88021944bd78 R08: 000000000000023c R09: ffff88021944bc14
Mar 23 16:37:41 zensamba kernel: [85320.064038] R10: ffff88021944bee2 R11: 0000000000000004 R12: ffffffff811fa3eb
Mar 23 16:37:41 zensamba kernel: [85320.064038] R13: ffff88021944bd18 R14: 0000000000000006 R15: 00000000ffffff9c
Mar 23 16:37:41 zensamba kernel: [85320.064038] FS:  00007fe6cd43f780(0000) GS:ffff88023fd80000(0000) knlGS:0000000000000000
Mar 23 16:37:41 zensamba kernel: [85320.064038] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 23 16:37:41 zensamba kernel: [85320.064038] CR2: 000055f937453000 CR3: 000000021944c000 CR4: 00000000000006e0
Mar 23 16:37:41 zensamba kernel: [85320.064038] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 23 16:37:41 zensamba kernel: [85320.064038] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Mar 23 16:37:41 zensamba kernel: [85320.064038] Stack:
Mar 23 16:37:41 zensamba kernel: [85320.064038]  ffff88021944bdc8 ffffffff8105b47b 0000000000000088 0000855035a1fc48
Mar 23 16:37:41 zensamba kernel: [85320.064038]  ffff88021944be48 ffff8800a9c1f800 ffff88021944bec0 ffff88008c0ca580
Mar 23 16:37:41 zensamba kernel: [85320.064038]  0000000000000027 ffff8800bac7e000 ffff88021944be48 ffffffff8105a721
Mar 23 16:37:41 zensamba kernel: [85320.064038] Call Trace:
Mar 23 16:37:41 zensamba kernel: [85320.064038]  [<ffffffff8105b47b>] kvm_lock_spinning+0xbb/0x1b0
Mar 23 16:37:41 zensamba kernel: [85320.064038]  [<ffffffff8105a721>] __raw_callee_save_kvm_lock_spinning+0x11/0x20
Mar 23 16:37:41 zensamba kernel: [85320.064038]  [<ffffffff817b8886>] ? _raw_spin_lock+0x56/0x60
Mar 23 16:37:41 zensamba kernel: [85320.064038]  [<ffffffff81749200>] ? unix_state_double_lock+0x60/0x70
Mar 23 16:37:41 zensamba kernel: [85320.064038]  [<ffffffff8174b713>] unix_dgram_connect+0x93/0x250
Mar 23 16:37:41 zensamba kernel: [85320.064038]  [<ffffffff8168fde7>] SYSC_connect+0xe7/0x120
Mar 23 16:37:41 zensamba kernel: [85320.064038]  [<ffffffff81690fce>] SyS_connect+0xe/0x10
Mar 23 16:37:41 zensamba kernel: [85320.064038]  [<ffffffff817b8c4d>] system_call_fastpath+0x16/0x1b
Mar 23 16:37:41 zensamba kernel: [85320.064038] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00

00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Mar 23 16:38:09 zensamba kernel: [85348.064036] NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [smbd:1696]
Mar 23 16:38:09 zensamba kernel: [85348.064036] Modules linked in: iptable_nat nf_nat_ipv4 nf_nat iptable_filter xt_mac xt_mark nf_conntrack_ipv4 nf_defrag_ipv4 xt_connmark nf_conntrack iptable_mangle

ip_tables x_tables quota_v2 quota_tree cirrus ttm drm_kms_helper drm joydev syscopyarea sysfillrect sysimgblt i2c_piix4 ppdev mac_hid shpchp serio_raw 8250_fintek parport_pc lp parport hid_generic

usbhid hid floppy psmouse pata_acpi
Mar 23 16:38:09 zensamba kernel: [85348.064036] CPU: 3 PID: 1696 Comm: smbd Tainted: G             L 3.19.0-56-generic #62~14.04.1-Ubuntu
Mar 23 16:38:09 zensamba kernel: [85348.064036] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
Mar 23 16:38:09 zensamba kernel: [85348.064036] task: ffff8802337875c0 ti: ffff880219448000 task.ti: ffff880219448000
Mar 23 16:38:09 zensamba kernel: [85348.064036] RIP: 0010:[<ffffffff8105b976>]  [<ffffffff8105b976>] native_safe_halt+0x6/0x10
Mar 23 16:38:09 zensamba kernel: [85348.064036] RSP: 0018:ffff88021944bd78  EFLAGS: 00000206
Mar 23 16:38:09 zensamba kernel: [85348.064036] RAX: 000000000000003b RBX: 00000000000000b0 RCX: 0000000000000001
Mar 23 16:38:09 zensamba kernel: [85348.064036] RDX: 0000000000000000 RSI: 0000000000008550 RDI: ffff88023fff20c0
Mar 23 16:38:09 zensamba kernel: [85348.064036] RBP: ffff88021944bd78 R08: 000000000000023c R09: ffff88021944bc14
Mar 23 16:38:09 zensamba kernel: [85348.064036] R10: ffff88021944bee2 R11: 0000000000000004 R12: ffffffff811fa3eb
Mar 23 16:38:09 zensamba kernel: [85348.064036] R13: ffff88021944bd18 R14: 0000000000000006 R15: 00000000ffffff9c
Mar 23 16:38:09 zensamba kernel: [85348.064036] FS:  00007fe6cd43f780(0000) GS:ffff88023fd80000(0000) knlGS:0000000000000000
Mar 23 16:38:09 zensamba kernel: [85348.064036] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 23 16:38:09 zensamba kernel: [85348.064036] CR2: 000055f937453000 CR3: 000000021944c000 CR4: 00000000000006e0
Mar 23 16:38:09 zensamba kernel: [85348.064036] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 23 16:38:09 zensamba kernel: [85348.064036] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Mar 23 16:38:09 zensamba kernel: [85348.064036] Stack:
Mar 23 16:38:09 zensamba kernel: [85348.064036]  ffff88021944bdc8 ffffffff8105b47b 0000000000000088 0000855035a1fc48
Mar 23 16:38:09 zensamba kernel: [85348.064036]  ffff88021944be48 ffff8800a9c1f800 ffff88021944bec0 ffff88008c0ca580
Mar 23 16:38:09 zensamba kernel: [85348.064036]  0000000000000027 ffff8800bac7e000 ffff88021944be48 ffffffff8105a721
Mar 23 16:38:09 zensamba kernel: [85348.064036] Call Trace:
Mar 23 16:38:09 zensamba kernel: [85348.064036]  [<ffffffff8105b47b>] kvm_lock_spinning+0xbb/0x1b0
Mar 23 16:38:09 zensamba kernel: [85348.064036]  [<ffffffff8105a721>] __raw_callee_save_kvm_lock_spinning+0x11/0x20
Mar 23 16:38:09 zensamba kernel: [85348.064036]  [<ffffffff817b8886>] ? _raw_spin_lock+0x56/0x60
Mar 23 16:38:09 zensamba kernel: [85348.064036]  [<ffffffff81749200>] ? unix_state_double_lock+0x60/0x70
Mar 23 16:38:09 zensamba kernel: [85348.064036]  [<ffffffff8174b713>] unix_dgram_connect+0x93/0x250
Mar 23 16:38:09 zensamba kernel: [85348.064036]  [<ffffffff8168fde7>] SYSC_connect+0xe7/0x120
Mar 23 16:38:09 zensamba kernel: [85348.064036]  [<ffffffff81690fce>] SyS_connect+0xe/0x10
Mar 23 16:38:09 zensamba kernel: [85348.064036]  [<ffffffff817b8c4d>] system_call_fastpath+0x16/0x1b
Mar 23 16:38:09 zensamba kernel: [85348.064036] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00

00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Mar 23 16:38:15 zensamba kernel: [85353.780039] INFO: rcu_sched self-detected stall on CPU { 3}  (t=15001 jiffies g=343192 c=343191 q=0)
Mar 23 16:38:15 zensamba kernel: [85353.780039] Task dump for CPU 3:
Mar 23 16:38:15 zensamba kernel: [85353.780039] smbd            R  running task        0  1696   1681 0x00000008
Mar 23 16:38:15 zensamba kernel: [85353.780039]  ffffffff81c56040 ffff88023fd83d78 ffffffff810a0296 0000000000000003
Mar 23 16:38:15 zensamba kernel: [85353.780039]  ffffffff81c56040 ffff88023fd83d98 ffffffff810a388d 0000000000000087
Mar 23 16:38:15 zensamba kernel: [85353.780039]  0000000000000004 ffff88023fd83dc8 ffffffff810d41a0 ffff88023fd94bc0
Mar 23 16:38:15 zensamba kernel: [85353.780039] Call Trace:
Mar 23 16:38:15 zensamba kernel: [85353.780039]  <IRQ>  [<ffffffff810a0296>] sched_show_task+0xb6/0x130
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff810a388d>] dump_cpu_task+0x3d/0x50
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff810d41a0>] rcu_dump_cpu_stacks+0x90/0xd0
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff810d805c>] rcu_check_callbacks+0x42c/0x670
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff810a48c1>] ? account_process_tick+0x61/0x180
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff810dcf99>] update_process_times+0x39/0x60
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff810ec4a5>] tick_sched_handle.isra.16+0x25/0x60
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff810ec524>] tick_sched_timer+0x44/0x80
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff810ddc57>] __run_hrtimer+0x77/0x1d0
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff810ec4e0>] ? tick_sched_handle.isra.16+0x60/0x60
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff810de037>] hrtimer_interrupt+0xe7/0x220
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff8104abe9>] local_apic_timer_interrupt+0x39/0x60
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff817bbcc5>] smp_apic_timer_interrupt+0x45/0x60
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff817b9cfd>] apic_timer_interrupt+0x6d/0x80
Mar 23 16:38:15 zensamba kernel: [85353.780039]  <EOI>  [<ffffffff8105b976>] ? native_safe_halt+0x6/0x10
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff8101e3a9>] ? sched_clock+0x9/0x10
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff8105b47b>] kvm_lock_spinning+0xbb/0x1b0
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff8105a721>] __raw_callee_save_kvm_lock_spinning+0x11/0x20
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff817b8886>] ? _raw_spin_lock+0x56/0x60
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff81749200>] ? unix_state_double_lock+0x60/0x70
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff8174b713>] unix_dgram_connect+0x93/0x250
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff8168fde7>] SYSC_connect+0xe7/0x120
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff81690fce>] SyS_connect+0xe/0x10
Mar 23 16:38:15 zensamba kernel: [85353.780039]  [<ffffffff817b8c4d>] system_call_fastpath+0x16/0x1b

Anyone with good news?
Cheers

[spott]

Change kernel to older one

[kinetica]

Hello Spott, thank you for your reply.

Could you please suggest wich kernel version we better use to avoid/fix this bug and, if possible, give some indication/procedure on how to install it?
Since this is a live running server, any help on reducing downtime for the users to minimum, will be very appreciated

Cheers!

[spott]

I am running more than 20 days now with GNU/Linux 3.19.0-43-generic x86_64 kernel.
Read this topic here are good step by step guide.
Install manualy the older kernel and then modify Grub to load only this kernel. Or choose in boot manually that kernel. When You are doing that - down down time is only minut or so for restart.

[kinetica]

Hello Spott, 
We downgraded to linux-image-3.19.0-43-generic.
Will update this post in case of issues
Waiting for a new patched kernel.... 
Thank you

[LaM]

I'm going to downgrade tomorrow evening, i'll keep You updated too

[peptoniET]

Ok. Let's asume 3.19.0-79 solves the problem.

Will "apt-get upgrade" update the kernel after we did a rollback...?  Were kernel updates freezed with the rollback we did...? How to force kernel update back to normal?

Cannot find anything...

Thanks.

not sure where you got  3.19.0-79  kernel i though they where only up to 3.19.0-51
apt-get upgrade will not upgrade kernels.
You need apt-get dist-upgrade  to upgrade kernel.
If you edited /etc/default/grub as in my previous post .that will keep booting by default the kernel you set there.
If you purged the faulty kernel then yes if you dist-upgrade then it will boot with newer kernel that may be installed
If you hold down the shift key when booting you should get the grub boot screen and be able to choose different kernels

I've setup a VM to test around this problem.  Cloned production server.
Apt-get dist-upgrade does nothing and reports no updates are available.  Kernel version stays the same after running the command and re-booting.
I did not edit /etc/default/grub, it's untouched.
Thanks.

[LaM]

Hi guys,

I've tryied to downgrade to kernel*47 but I wasn't able, server wasn't booting and was stuck. I was able to boot with the kernel*49...strange.
Have to investigate the issue.

Meanwhile I've updated one server to kernel 3.19.0-56-generic...let's see if something change.

L

[kinetica]

Hi LaM,
our server was up-to-date with kernel: 3.19.0-56-generic, when We start to notice this bug
For us was necessary to downgrade to linux-image-3.19.0-43-generic:

Our procedure was:
-- Check kernel running = uname -r
-- Check the firmware installed with =  dpkg --list | grep linux-image
-- Check file present in the partition =   /boot/
-- Install the old firmware =  apt-get install linux-image-3.19.0-43-generic
-- Check all firmware installed again =  dpkg --list | grep linux-image
-- Modify grab =  /etc/default/grub

# GRUB_DEFAULT=0
GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 3.19.0-43-generic"

-- Reboot
-- Check kernel running = uname -r

Since then Samba and server are running smooth 

[LaM]

Thank You Kinetica,

so You confirm that even kernel 3.19.0-56-generic is affected. Waaay bad, way bad.

I get that this is the developer edition but this is embarrassing...
I'll check with the step You gave me this saturday, hopefully I will be able to fix the only kernel that isn't working (and the only one I need...sigh)

Btw...why specifically linux-image-3.19.0-43-generic and not linux-image-3.19.0-47-generic?

Thanks again!!

L

[uro.sh]

Hi LaM,
our server was up-to-date with kernel: 3.19.0-56-generic, when We start to notice this bug
For us was necessary to downgrade to linux-image-3.19.0-43-generic:

Our procedure was:
-- Check kernel running = uname -r
-- Check the firmware installed with =  dpkg --list | grep linux-image
-- Check file present in the partition =   /boot/
-- Install the old firmware =  apt-get install linux-image-3.19.0-43-generic
-- Check all firmware installed again =  dpkg --list | grep linux-image
-- Modify grab =  /etc/default/grub

# GRUB_DEFAULT=0
GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux 3.19.0-43-generic"

-- Reboot
-- Check kernel running = uname -r

Since then Samba and server are running smooth 

Thank you kinetica for your suggestion, but when i tried this my server stuck at boot. On screen was zentyal 4.2 and 5 dots and waiting for network connection to became ready. What did i done wrong? Is there any news for kernel update?

[LaM]

Imo it's the kernel...the update must have modified something...

But I don't have any prove of this, it's just an opinion

[kinetica]

Sorry guys for the long delay in the answer.
@LaM = "Btw...why specifically linux-image-3.19.0-43-generic and not linux-image-3.19.0-47-generic?"
 Just because I followed a suggestion from the User Spott stating " I am running more than 20 days now with GNU/Linux 3.19.0-43-generic x86_64 kernel.", and if you read all these five pages if I am not wrong you'll discover that users with linux-image-3.19.0-47-generic had the same issue.
Yes I confirm our server was at kernel 3.19.0-56-generic and the bug was present and painful

@uro.sh
I am not sure what on your boot/server did wrong. Regarding the boot, I followed indications in this guide
https://help.ubuntu.com/community/Grub2/Submenus
After downgrade the kernel I did not upgrade Zentyal at all!

Regarding news from new kernel or solution for this bug unfortunately no update
https://tracker.zentyal.org/issues/4977