Author Topic: [SOLVED] Zentyal extremely slow  (Read 20157 times)

acesuares

  • Zen Apprentice
  • *
  • Posts: 35
  • Karma: +0/-0
    • View Profile
[SOLVED] Zentyal extremely slow
« on: December 13, 2010, 02:13:28 pm »
Please see http://www.suares.an/index.php?topic=zentyal-ears for the solution.

Dear people,

When I installed the OLD ebox/zentyal on Ubuntu 8.04, there where many issues with slowness.
Then I moved to the new Zentyal, on 10.04, and it worked all fine for a while. Since some time the slowness is back.

The symptoms are:
- both harddisk leds burn constantly
- everything is slow, loggin in trough the console as well as opening a folder through samba
- there's a very high io/wait when looking at top.
- when looking at iostat there is almost NO disk traffic

This is load picture of a day. there are 4 cpu's only 2 shown.



« Last Edit: February 11, 2011, 05:36:41 pm by acesuares »

Josep

  • Zen Samurai
  • ****
  • Posts: 255
  • Karma: +6/-0
    • View Profile
Re: Zentyal extremely slow
« Reply #1 on: December 13, 2010, 04:24:22 pm »
Are your logs too large?
Are you registering lots of information?
Are you using dispatchers for events?
Any suspicious process/number of processes?

Try installing iotop ("sudo aptitude install iotop"), and see if there is something obvious.

acesuares

  • Zen Apprentice
  • *
  • Posts: 35
  • Karma: +0/-0
    • View Profile
Re: Zentyal extremely slow
« Reply #2 on: December 13, 2010, 05:54:08 pm »
Are your logs too large?

- how can I find out?

Are you registering lots of information?

- I don't know, how to check it ?

Are you using dispatchers for events?

- I dont' think so, how can I know?

Any suspicious process/number of processes?

- no

I run iostat -d 1 m -z and there is very little disk activity. I just ran iotop and couldn't see anything except for a lot of zero values. Looks like little disk ativity
I write a file every minute wit some stats, here is an except:

             CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle

21:44:01     all    0.86    0.00    0.27   18.75    0.00    0.06    0.00    0.00   80.06
21:45:01     all    0.86    0.00    0.27   18.75    0.00    0.06    0.00    0.00   80.06
21:46:01     all    0.86    0.00    0.27   18.74    0.00    0.06    0.00    0.00   80.06
21:47:01     all    0.86    0.00    0.27   18.73    0.00    0.06    0.00    0.00   80.08
21:48:01     all    0.86    0.00    0.27   18.70    0.00    0.06    0.00    0.00   80.11

12:47:01     all    0.87    0.01    0.34   13.02    0.00    0.06    0.00    0.00   85.70
12:48:01     all    0.87    0.01    0.34   13.02    0.00    0.06    0.00    0.00   85.70
12:49:01     all    0.87    0.01    0.34   13.01    0.00    0.06    0.00    0.00   85.71
12:50:01     all    0.87    0.01    0.34   13.01    0.00    0.06    0.00    0.00   85.71
12:51:01     all    0.87    0.01    0.34   13.01    0.00    0.06    0.00    0.00   85.71
12:52:01     all    0.87    0.01    0.34   13.00    0.00    0.06    0.00    0.00   85.72
12:53:01     all    0.87    0.01    0.34   13.00    0.00    0.06    0.00    0.00   85.72

iotop -o -d 1
shows very little disk activity and only very few processes shortly active. Still io/wait is 55%
at the moment.

I had the exact same problem with 8.04/ebox but it went away with the 10.04/zentyal installation but now it's back.

I tested on different hardware too.

acesuares

  • Zen Apprentice
  • *
  • Posts: 35
  • Karma: +0/-0
    • View Profile
Re: Zentyal extremely slow
« Reply #3 on: December 13, 2010, 06:04:47 pm »
Hmm I got these in dmesg. Can't be good.

[259464.399149] type=1503 audit(1292256422.386:145):  operation="exec" pid=28432 parent=28431 profile="/usr/bin/freshclam" requested_mask="::x" denied_mask="::x" fsuid=105 ouid=0 name="/bin/dash"
[260880.464528] INFO: task kjournald:309 blocked for more than 120 seconds.
[260880.471707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[260880.486233] kjournald     D 000070c5     0   309      2 0x00000000
[260880.486241]  f6b21e68 00000046 f5778000 000070c5 00000000 c0849760 f6aedbec c0849760
[260880.486251]  ebe895aa 0000ed1e c0849760 c0849760 f6aedbec c0849760 c0849760 f24df400
[260880.486261]  00000000 0000ed1e f6aed940 c2408760 f6aed940 f6b21eb4 f6b21e78 c058b16a
[260880.486270] Call Trace:
[260880.486283]  [<c058b16a>] io_schedule+0x3a/0x60
[260880.486290]  [<c022d2c8>] sync_buffer+0x38/0x40
[260880.486295]  [<c058b90d>] __wait_on_bit+0x4d/0x70
[260880.486299]  [<c022d290>] ? sync_buffer+0x0/0x40
[260880.486303]  [<c022d290>] ? sync_buffer+0x0/0x40
[260880.486308]  [<c058b9db>] out_of_line_wait_on_bit+0xab/0xc0
[260880.486316]  [<c0167860>] ? wake_bit_function+0x0/0x50
[260880.486321]  [<c022d28e>] __wait_on_buffer+0x2e/0x30
[260880.486329]  [<c02bef23>] journal_commit_transaction+0x3b3/0xc20
[260880.486335]  [<c058cdcf>] ? _spin_lock_irqsave+0x2f/0x50
[260880.486341]  [<c015b8f8>] ? try_to_del_timer_sync+0x68/0xb0
[260880.486348]  [<c02c2525>] kjournald+0xb5/0x1e0
[260880.486352]  [<c0167810>] ? autoremove_wake_function+0x0/0x50
[260880.486357]  [<c02c2470>] ? kjournald+0x0/0x1e0
[260880.486361]  [<c0167584>] kthread+0x74/0x80
[260880.486365]  [<c0167510>] ? kthread+0x0/0x80
[260880.486373]  [<c0104087>] kernel_thread_helper+0x7/0x10
[260880.486377] INFO: task flush-9:0:344 blocked for more than 120 seconds.
[260880.493943] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[260880.509225] flush-9:0     D 00033198     0   344      2 0x00000000
[260880.509233]  f30d7d88 00000046 f6b71d88 00033198 00000000 c0849760 f7201c2c c0849760
[260880.509243]  36cebf99 0000ed22 c0849760 c0849760 f7201c2c c0849760 c0849760 f2992400
[260880.509252]  36cc0243 0000ed22 f7201980 c2508760 f7201980 f30d7dd4 f30d7d98 c058b16a
[260880.509261] Call Trace:
[260880.509266]  [<c058b16a>] io_schedule+0x3a/0x60
[260880.509273]  [<c01ca91d>] sync_page+0x3d/0x50
[260880.509277]  [<c058b7b7>] __wait_on_bit_lock+0x47/0x90
[260880.509281]  [<c01ca8e0>] ? sync_page+0x0/0x50
[260880.509285]  [<c01ca8ae>] __lock_page+0x7e/0x90
[260880.509289]  [<c0167860>] ? wake_bit_function+0x0/0x50
[260880.509298]  [<c01d2292>] write_cache_pages+0x282/0x360
[260880.509302]  [<c01d1110>] ? __writepage+0x0/0x40
[260880.509308]  [<c0266d30>] ? ext3_ordered_writepage+0x0/0x1f0
[260880.509313]  [<c01d2394>] generic_writepages+0x24/0x30
[260880.509317]  [<c01d23cd>] do_writepages+0x2d/0x40
[260880.509325]  [<c0226425>] writeback_single_inode+0xd5/0x380
[260880.509329]  [<c0226f29>] writeback_inodes_wb+0x389/0x530
[260880.509334]  [<c02271c7>] wb_writeback+0xf7/0x1a0
[260880.509339]  [<c022748e>] wb_do_writeback+0x15e/0x170
[260880.509344]  [<c02274db>] bdi_writeback_task+0x3b/0xb0
[260880.509349]  [<c01e1680>] ? bdi_start_fn+0x0/0xc0
[260880.509353]  [<c01e16dd>] bdi_start_fn+0x5d/0xc0
[260880.509356]  [<c01e1680>] ? bdi_start_fn+0x0/0xc0
[260880.509360]  [<c0167584>] kthread+0x74/0x80
[260880.509365]  [<c0167510>] ? kthread+0x0/0x80
[260880.509369]  [<c0104087>] kernel_thread_helper+0x7/0x10
[260880.509374] INFO: task rsyslogd:24131 blocked for more than 120 seconds.
[260880.517680] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[260880.534561] rsyslogd      D 000050f9     0 24131      1 0x00000000
[260880.534574]  f5779c54 00000086 f6b71d8c 000050f9 00000000 c0849760 f253e8ac c0849760
[260880.534600]  ebe8f61d 0000ed1e c0849760 c0849760 f253e8ac c0849760 c0849760 f288cc00
[260880.534625]  ebe895aa 0000ed1e f253e600 c2408760 f253e600 f5779ca0 f5779c64 c058b16a
[260880.534650] Call Trace:
[260880.534659]  [<c058b16a>] io_schedule+0x3a/0x60
[260880.534666]  [<c022d2c8>] sync_buffer+0x38/0x40
[260880.534672]  [<c058b90d>] __wait_on_bit+0x4d/0x70
[260880.534679]  [<c022d290>] ? sync_buffer+0x0/0x40
[260880.534686]  [<c022d290>] ? sync_buffer+0x0/0x40
[260880.534693]  [<c058b9db>] out_of_line_wait_on_bit+0xab/0xc0
[260880.534700]  [<c0167860>] ? wake_bit_function+0x0/0x50
[260880.534707]  [<c022d28e>] __wait_on_buffer+0x2e/0x30
[260880.534715]  [<c022e114>] sync_dirty_buffer+0x64/0xd0
[260880.534722]  [<c02bdf72>] journal_dirty_data+0x1c2/0x230
[260880.534730]  [<c02672bd>] ext3_journal_dirty_data+0x1d/0x50
[260880.534737]  [<c022cf6b>] ? __block_commit_write+0x7b/0xb0
[260880.534745]  [<c0267315>] journal_dirty_data_fn+0x25/0x30
[260880.534752]  [<c0266408>] walk_page_buffers+0x58/0x80
[260880.534761]  [<c026a43d>] ext3_ordered_write_end+0x8d/0x180
[260880.534768]  [<c02672f0>] ? journal_dirty_data_fn+0x0/0x30
[260880.534775]  [<c01c9f6e>] ? iov_iter_copy_from_user_atomic+0x5e/0x90
[260880.534783]  [<c01ca0b2>] generic_perform_write+0x112/0x190
[260880.534791]  [<c01cb04a>] generic_file_buffered_write+0x5a/0xe0
[260880.534802]  [<c021c7dd>] ? file_update_time+0xbd/0x130
[260880.534809]  [<c01cc290>] __generic_file_aio_write+0x1f0/0x510
[260880.534820]  [<c016ab7f>] ? __remove_hrtimer+0x2f/0xa0
[260880.534828]  [<c016bd56>] ? hrtimer_try_to_cancel+0x36/0xb0
[260880.534843]  [<c0178115>] ? futex_wait_queue_me+0xa5/0xd0
[260880.534850]  [<c01cc607>] generic_file_aio_write+0x57/0xc0
[260880.534864]  [<c0208374>] do_sync_write+0xc4/0x100
[260880.534871]  [<c016d5c5>] ? sched_clock_local+0xa5/0x180
[260880.534878]  [<c0167810>] ? autoremove_wake_function+0x0/0x50
[260880.534892]  [<c02f57b4>] ? security_file_permission+0x14/0x20
[260880.534900]  [<c0208514>] ? rw_verify_area+0x64/0xe0
[260880.534907]  [<c058af28>] ? schedule+0x638/0x840
[260880.534915]  [<c0208632>] vfs_write+0xa2/0x1a0
[260880.534922]  [<c02082b0>] ? do_sync_write+0x0/0x100
[260880.534930]  [<c0208f52>] sys_write+0x42/0x70
[260880.534937]  [<c01033ec>] syscall_call+0x7/0xb
[263065.473474] type=1503 audit(1292260023.462:146):  operation="exec" pid=1685 parent=1683 profile="/usr/bin/freshclam" requested_mask="::x" denied_mask="::x" fsuid=105 ouid=0 name="/bin/dash"
[263065.473838] type=1503 audit(1292260023.462:147):  operation="exec" pid=1686 parent=1684 profile="/usr/bin/freshclam" requested_mask="::x" denied_mask="::x" fsuid=105 ouid=0 name="/bin/dash"


Any ideas? RAM problems?

acesuares

  • Zen Apprentice
  • *
  • Posts: 35
  • Karma: +0/-0
    • View Profile
Re: Zentyal extremely slow
« Reply #4 on: December 13, 2010, 07:24:01 pm »
Did a memtest, no problems detected.

Found this thread: http://ubuntuforums.org/showthread.php?t=1566037
But after updating the kernel the problems persists.

I am at my wits end.

Is this a Zentyal problem, an Ubuntu problem, or a hardware problem?


Josep

  • Zen Samurai
  • ****
  • Posts: 255
  • Karma: +6/-0
    • View Profile
Re: Zentyal extremely slow
« Reply #5 on: December 14, 2010, 09:13:21 am »
By the looks of it, it looks like a filesystem problem. Now, this may be due to a hardware problem, and this would explain why it's back, despite having upgraded the OS.
Try the command fsck (look for help on this command).
Having a backup of your precious data seems like a good idea.

borislav

  • Zen Apprentice
  • *
  • Posts: 5
  • Karma: +0/-1
    • View Profile
Re: [SOLVED] Zentyal extremely slow
« Reply #6 on: October 12, 2012, 08:41:05 am »
Experiencing the same problem: extremely slow interface and server software response. Tried latest versions on several different machines with same result except 4-processor server with 16Gb RAM and SAS hard drives. But I'm not ready to use such server just to have quick web-interface. And this is not EARS HDD problem, because I don't have them. Clean Ubuntu runs fast on tested machines.

EddieA

  • Zen Monk
  • **
  • Posts: 70
  • Karma: +3/-0
    • View Profile
Re: [SOLVED] Zentyal extremely slow
« Reply #7 on: October 12, 2012, 09:20:45 pm »
This looks like another manifestation of this.

Except mine is under ESXi, so no "real" hardware involved.  And no other VMs are showing any strange symptoms, so I have nothe else to suggest the "real" hardware is a fault.

Cheers.

borislav

  • Zen Apprentice
  • *
  • Posts: 5
  • Karma: +0/-1
    • View Profile
Re: [SOLVED] Zentyal extremely slow
« Reply #8 on: March 06, 2013, 02:28:42 pm »
Aww, how cute to see this topic marked as "solved". This piece of sssoft is still slow being fully up to date.

acesuares

  • Zen Apprentice
  • *
  • Posts: 35
  • Karma: +0/-0
    • View Profile
Re: [SOLVED] Zentyal extremely slow
« Reply #9 on: April 08, 2013, 07:01:02 pm »
My problem exclusively was caused by the WD EARS formatting problem. In newer Zentyal versions it is solved...

half_life

  • Bug Hunter
  • Zen Hero
  • *****
  • Posts: 862
  • Karma: +57/-0
    • View Profile
Re: [SOLVED] Zentyal extremely slow
« Reply #10 on: April 09, 2013, 04:39:51 am »
Borislav,  did you have something constructive to add to the conversation?  Or were you just stopping by to snipe?  If so,  please find yourself to the nearest exit as you have no interest in Zentyal other to criticize. 

Have a legitimate question?  Start a thread and we will try and help.

acesuares

  • Zen Apprentice
  • *
  • Posts: 35
  • Karma: +0/-0
    • View Profile
Re: [SOLVED] Zentyal extremely slow
« Reply #11 on: April 09, 2013, 05:10:22 am »
Totally off-topic, but the web interface is extremely slow. Even on the console (FF).

Why is that? It's a bunch of perl scripts... shouldn't be that slow.

I really want to know WHY and no sniping involved.

Ace

half_life

  • Bug Hunter
  • Zen Hero
  • *****
  • Posts: 862
  • Karma: +57/-0
    • View Profile
Re: [SOLVED] Zentyal extremely slow
« Reply #12 on: April 09, 2013, 06:32:17 am »
Is it sporadic or is it pretty much constant?

belazir

  • Zen Apprentice
  • *
  • Posts: 4
  • Karma: +0/-0
    • View Profile
Re: [SOLVED] Zentyal extremely slow
« Reply #13 on: April 09, 2013, 04:04:22 pm »
Have to say, like acesuares

Totally off-topic, but the web interface is extremely slow. Even on the console (FF).

Why is that? It's a bunch of perl scripts... shouldn't be that slow.



I've also found the web interface constantly slow. Never what you'd call sharp or snappy... *click* *wait* *click*... It's never timed out on me while I'm waiting for pages etc, but it does seem very laggy considering the resources it has available.

half_life

  • Bug Hunter
  • Zen Hero
  • *****
  • Posts: 862
  • Karma: +57/-0
    • View Profile
Re: [SOLVED] Zentyal extremely slow
« Reply #14 on: April 09, 2013, 04:53:45 pm »
I have two active installs. The first one is for work and is 2.2 with 6g ram 4 cpus to support 150+ users. The second one is a home server running 3.0  with 2gb ram and 2 cpus.  Both systems run as virtual machines under Proxmox.  Neither system is laggy unless the system is being stressed by something.  Both have most modules running.

Do either of you have Zarafa installed?  What is typically the system load as shown on the dashboard?  To help me understand,  how many cpus and how much ram on each machine  (if in a virtual environment that would be nice to know)?