Zentyal Forum, Linux Small Business Server

Zentyal Server => Installation and Upgrades => Topic started by: acesuares on December 13, 2010, 02:13:28 pm

Title: [SOLVED] Zentyal extremely slow
Post by: acesuares on December 13, 2010, 02:13:28 pm
Please see http://www.suares.an/index.php?topic=zentyal-ears (http://www.suares.an/index.php?topic=zentyal-ears) for the solution.

Dear people,

When I installed the OLD ebox/zentyal on Ubuntu 8.04, there where many issues with slowness.
Then I moved to the new Zentyal, on 10.04, and it worked all fine for a while. Since some time the slowness is back.

The symptoms are:
- both harddisk leds burn constantly
- everything is slow, loggin in trough the console as well as opening a folder through samba
- there's a very high io/wait when looking at top.
- when looking at iostat there is almost NO disk traffic

This is load picture of a day. there are 4 cpu's only 2 shown.
(http://www.suares.an/foto/zentyal-load.png)


Title: Re: Zentyal extremely slow
Post by: Josep on December 13, 2010, 04:24:22 pm
Are your logs too large?
Are you registering lots of information?
Are you using dispatchers for events?
Any suspicious process/number of processes?

Try installing iotop ("sudo aptitude install iotop"), and see if there is something obvious.
Title: Re: Zentyal extremely slow
Post by: acesuares on December 13, 2010, 05:54:08 pm
Are your logs too large?

- how can I find out?

Are you registering lots of information?

- I don't know, how to check it ?

Are you using dispatchers for events?

- I dont' think so, how can I know?

Any suspicious process/number of processes?

- no

I run iostat -d 1 m -z and there is very little disk activity. I just ran iotop and couldn't see anything except for a lot of zero values. Looks like little disk ativity
I write a file every minute wit some stats, here is an except:

             CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle

21:44:01     all    0.86    0.00    0.27   18.75    0.00    0.06    0.00    0.00   80.06
21:45:01     all    0.86    0.00    0.27   18.75    0.00    0.06    0.00    0.00   80.06
21:46:01     all    0.86    0.00    0.27   18.74    0.00    0.06    0.00    0.00   80.06
21:47:01     all    0.86    0.00    0.27   18.73    0.00    0.06    0.00    0.00   80.08
21:48:01     all    0.86    0.00    0.27   18.70    0.00    0.06    0.00    0.00   80.11

12:47:01     all    0.87    0.01    0.34   13.02    0.00    0.06    0.00    0.00   85.70
12:48:01     all    0.87    0.01    0.34   13.02    0.00    0.06    0.00    0.00   85.70
12:49:01     all    0.87    0.01    0.34   13.01    0.00    0.06    0.00    0.00   85.71
12:50:01     all    0.87    0.01    0.34   13.01    0.00    0.06    0.00    0.00   85.71
12:51:01     all    0.87    0.01    0.34   13.01    0.00    0.06    0.00    0.00   85.71
12:52:01     all    0.87    0.01    0.34   13.00    0.00    0.06    0.00    0.00   85.72
12:53:01     all    0.87    0.01    0.34   13.00    0.00    0.06    0.00    0.00   85.72

iotop -o -d 1
shows very little disk activity and only very few processes shortly active. Still io/wait is 55%
at the moment.

I had the exact same problem with 8.04/ebox but it went away with the 10.04/zentyal installation but now it's back.

I tested on different hardware too.
Title: Re: Zentyal extremely slow
Post by: acesuares on December 13, 2010, 06:04:47 pm
Hmm I got these in dmesg. Can't be good.

[259464.399149] type=1503 audit(1292256422.386:145):  operation="exec" pid=28432 parent=28431 profile="/usr/bin/freshclam" requested_mask="::x" denied_mask="::x" fsuid=105 ouid=0 name="/bin/dash"
[260880.464528] INFO: task kjournald:309 blocked for more than 120 seconds.
[260880.471707] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[260880.486233] kjournald     D 000070c5     0   309      2 0x00000000
[260880.486241]  f6b21e68 00000046 f5778000 000070c5 00000000 c0849760 f6aedbec c0849760
[260880.486251]  ebe895aa 0000ed1e c0849760 c0849760 f6aedbec c0849760 c0849760 f24df400
[260880.486261]  00000000 0000ed1e f6aed940 c2408760 f6aed940 f6b21eb4 f6b21e78 c058b16a
[260880.486270] Call Trace:
[260880.486283]  [<c058b16a>] io_schedule+0x3a/0x60
[260880.486290]  [<c022d2c8>] sync_buffer+0x38/0x40
[260880.486295]  [<c058b90d>] __wait_on_bit+0x4d/0x70
[260880.486299]  [<c022d290>] ? sync_buffer+0x0/0x40
[260880.486303]  [<c022d290>] ? sync_buffer+0x0/0x40
[260880.486308]  [<c058b9db>] out_of_line_wait_on_bit+0xab/0xc0
[260880.486316]  [<c0167860>] ? wake_bit_function+0x0/0x50
[260880.486321]  [<c022d28e>] __wait_on_buffer+0x2e/0x30
[260880.486329]  [<c02bef23>] journal_commit_transaction+0x3b3/0xc20
[260880.486335]  [<c058cdcf>] ? _spin_lock_irqsave+0x2f/0x50
[260880.486341]  [<c015b8f8>] ? try_to_del_timer_sync+0x68/0xb0
[260880.486348]  [<c02c2525>] kjournald+0xb5/0x1e0
[260880.486352]  [<c0167810>] ? autoremove_wake_function+0x0/0x50
[260880.486357]  [<c02c2470>] ? kjournald+0x0/0x1e0
[260880.486361]  [<c0167584>] kthread+0x74/0x80
[260880.486365]  [<c0167510>] ? kthread+0x0/0x80
[260880.486373]  [<c0104087>] kernel_thread_helper+0x7/0x10
[260880.486377] INFO: task flush-9:0:344 blocked for more than 120 seconds.
[260880.493943] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[260880.509225] flush-9:0     D 00033198     0   344      2 0x00000000
[260880.509233]  f30d7d88 00000046 f6b71d88 00033198 00000000 c0849760 f7201c2c c0849760
[260880.509243]  36cebf99 0000ed22 c0849760 c0849760 f7201c2c c0849760 c0849760 f2992400
[260880.509252]  36cc0243 0000ed22 f7201980 c2508760 f7201980 f30d7dd4 f30d7d98 c058b16a
[260880.509261] Call Trace:
[260880.509266]  [<c058b16a>] io_schedule+0x3a/0x60
[260880.509273]  [<c01ca91d>] sync_page+0x3d/0x50
[260880.509277]  [<c058b7b7>] __wait_on_bit_lock+0x47/0x90
[260880.509281]  [<c01ca8e0>] ? sync_page+0x0/0x50
[260880.509285]  [<c01ca8ae>] __lock_page+0x7e/0x90
[260880.509289]  [<c0167860>] ? wake_bit_function+0x0/0x50
[260880.509298]  [<c01d2292>] write_cache_pages+0x282/0x360
[260880.509302]  [<c01d1110>] ? __writepage+0x0/0x40
[260880.509308]  [<c0266d30>] ? ext3_ordered_writepage+0x0/0x1f0
[260880.509313]  [<c01d2394>] generic_writepages+0x24/0x30
[260880.509317]  [<c01d23cd>] do_writepages+0x2d/0x40
[260880.509325]  [<c0226425>] writeback_single_inode+0xd5/0x380
[260880.509329]  [<c0226f29>] writeback_inodes_wb+0x389/0x530
[260880.509334]  [<c02271c7>] wb_writeback+0xf7/0x1a0
[260880.509339]  [<c022748e>] wb_do_writeback+0x15e/0x170
[260880.509344]  [<c02274db>] bdi_writeback_task+0x3b/0xb0
[260880.509349]  [<c01e1680>] ? bdi_start_fn+0x0/0xc0
[260880.509353]  [<c01e16dd>] bdi_start_fn+0x5d/0xc0
[260880.509356]  [<c01e1680>] ? bdi_start_fn+0x0/0xc0
[260880.509360]  [<c0167584>] kthread+0x74/0x80
[260880.509365]  [<c0167510>] ? kthread+0x0/0x80
[260880.509369]  [<c0104087>] kernel_thread_helper+0x7/0x10
[260880.509374] INFO: task rsyslogd:24131 blocked for more than 120 seconds.
[260880.517680] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[260880.534561] rsyslogd      D 000050f9     0 24131      1 0x00000000
[260880.534574]  f5779c54 00000086 f6b71d8c 000050f9 00000000 c0849760 f253e8ac c0849760
[260880.534600]  ebe8f61d 0000ed1e c0849760 c0849760 f253e8ac c0849760 c0849760 f288cc00
[260880.534625]  ebe895aa 0000ed1e f253e600 c2408760 f253e600 f5779ca0 f5779c64 c058b16a
[260880.534650] Call Trace:
[260880.534659]  [<c058b16a>] io_schedule+0x3a/0x60
[260880.534666]  [<c022d2c8>] sync_buffer+0x38/0x40
[260880.534672]  [<c058b90d>] __wait_on_bit+0x4d/0x70
[260880.534679]  [<c022d290>] ? sync_buffer+0x0/0x40
[260880.534686]  [<c022d290>] ? sync_buffer+0x0/0x40
[260880.534693]  [<c058b9db>] out_of_line_wait_on_bit+0xab/0xc0
[260880.534700]  [<c0167860>] ? wake_bit_function+0x0/0x50
[260880.534707]  [<c022d28e>] __wait_on_buffer+0x2e/0x30
[260880.534715]  [<c022e114>] sync_dirty_buffer+0x64/0xd0
[260880.534722]  [<c02bdf72>] journal_dirty_data+0x1c2/0x230
[260880.534730]  [<c02672bd>] ext3_journal_dirty_data+0x1d/0x50
[260880.534737]  [<c022cf6b>] ? __block_commit_write+0x7b/0xb0
[260880.534745]  [<c0267315>] journal_dirty_data_fn+0x25/0x30
[260880.534752]  [<c0266408>] walk_page_buffers+0x58/0x80
[260880.534761]  [<c026a43d>] ext3_ordered_write_end+0x8d/0x180
[260880.534768]  [<c02672f0>] ? journal_dirty_data_fn+0x0/0x30
[260880.534775]  [<c01c9f6e>] ? iov_iter_copy_from_user_atomic+0x5e/0x90
[260880.534783]  [<c01ca0b2>] generic_perform_write+0x112/0x190
[260880.534791]  [<c01cb04a>] generic_file_buffered_write+0x5a/0xe0
[260880.534802]  [<c021c7dd>] ? file_update_time+0xbd/0x130
[260880.534809]  [<c01cc290>] __generic_file_aio_write+0x1f0/0x510
[260880.534820]  [<c016ab7f>] ? __remove_hrtimer+0x2f/0xa0
[260880.534828]  [<c016bd56>] ? hrtimer_try_to_cancel+0x36/0xb0
[260880.534843]  [<c0178115>] ? futex_wait_queue_me+0xa5/0xd0
[260880.534850]  [<c01cc607>] generic_file_aio_write+0x57/0xc0
[260880.534864]  [<c0208374>] do_sync_write+0xc4/0x100
[260880.534871]  [<c016d5c5>] ? sched_clock_local+0xa5/0x180
[260880.534878]  [<c0167810>] ? autoremove_wake_function+0x0/0x50
[260880.534892]  [<c02f57b4>] ? security_file_permission+0x14/0x20
[260880.534900]  [<c0208514>] ? rw_verify_area+0x64/0xe0
[260880.534907]  [<c058af28>] ? schedule+0x638/0x840
[260880.534915]  [<c0208632>] vfs_write+0xa2/0x1a0
[260880.534922]  [<c02082b0>] ? do_sync_write+0x0/0x100
[260880.534930]  [<c0208f52>] sys_write+0x42/0x70
[260880.534937]  [<c01033ec>] syscall_call+0x7/0xb
[263065.473474] type=1503 audit(1292260023.462:146):  operation="exec" pid=1685 parent=1683 profile="/usr/bin/freshclam" requested_mask="::x" denied_mask="::x" fsuid=105 ouid=0 name="/bin/dash"
[263065.473838] type=1503 audit(1292260023.462:147):  operation="exec" pid=1686 parent=1684 profile="/usr/bin/freshclam" requested_mask="::x" denied_mask="::x" fsuid=105 ouid=0 name="/bin/dash"


Any ideas? RAM problems?
Title: Re: Zentyal extremely slow
Post by: acesuares on December 13, 2010, 07:24:01 pm
Did a memtest, no problems detected.

Found this thread: http://ubuntuforums.org/showthread.php?t=1566037
But after updating the kernel the problems persists.

I am at my wits end.

Is this a Zentyal problem, an Ubuntu problem, or a hardware problem?

Title: Re: Zentyal extremely slow
Post by: Josep on December 14, 2010, 09:13:21 am
By the looks of it, it looks like a filesystem problem. Now, this may be due to a hardware problem, and this would explain why it's back, despite having upgraded the OS.
Try the command fsck (look for help on this command).
Having a backup of your precious data seems like a good idea.
Title: Re: [SOLVED] Zentyal extremely slow
Post by: borislav on October 12, 2012, 08:41:05 am
Experiencing the same problem: extremely slow interface and server software response. Tried latest versions on several different machines with same result except 4-processor server with 16Gb RAM and SAS hard drives. But I'm not ready to use such server just to have quick web-interface. And this is not EARS HDD problem, because I don't have them. Clean Ubuntu runs fast on tested machines.
Title: Re: [SOLVED] Zentyal extremely slow
Post by: EddieA on October 12, 2012, 09:20:45 pm
This looks like another manifestation of this (http://forum.zentyal.org/index.php/topic,12139.msg49839.html#msg49839).

Except mine is under ESXi, so no "real" hardware involved.  And no other VMs are showing any strange symptoms, so I have nothe else to suggest the "real" hardware is a fault.

Cheers.
Title: Re: [SOLVED] Zentyal extremely slow
Post by: borislav on March 06, 2013, 02:28:42 pm
Aww, how cute to see this topic marked as "solved". This piece of sssoft is still slow being fully up to date.
Title: Re: [SOLVED] Zentyal extremely slow
Post by: acesuares on April 08, 2013, 07:01:02 pm
My problem exclusively was caused by the WD EARS formatting problem. In newer Zentyal versions it is solved...
Title: Re: [SOLVED] Zentyal extremely slow
Post by: half_life on April 09, 2013, 04:39:51 am
Borislav,  did you have something constructive to add to the conversation?  Or were you just stopping by to snipe?  If so,  please find yourself to the nearest exit as you have no interest in Zentyal other to criticize. 

Have a legitimate question?  Start a thread and we will try and help.
Title: Re: [SOLVED] Zentyal extremely slow
Post by: acesuares on April 09, 2013, 05:10:22 am
Totally off-topic, but the web interface is extremely slow. Even on the console (FF).

Why is that? It's a bunch of perl scripts... shouldn't be that slow.

I really want to know WHY and no sniping involved.

Ace
Title: Re: [SOLVED] Zentyal extremely slow
Post by: half_life on April 09, 2013, 06:32:17 am
Is it sporadic or is it pretty much constant?
Title: Re: [SOLVED] Zentyal extremely slow
Post by: belazir on April 09, 2013, 04:04:22 pm
Have to say, like acesuares

Totally off-topic, but the web interface is extremely slow. Even on the console (FF).

Why is that? It's a bunch of perl scripts... shouldn't be that slow.



I've also found the web interface constantly slow. Never what you'd call sharp or snappy... *click* *wait* *click*... It's never timed out on me while I'm waiting for pages etc, but it does seem very laggy considering the resources it has available.
Title: Re: [SOLVED] Zentyal extremely slow
Post by: half_life on April 09, 2013, 04:53:45 pm
I have two active installs. The first one is for work and is 2.2 with 6g ram 4 cpus to support 150+ users. The second one is a home server running 3.0  with 2gb ram and 2 cpus.  Both systems run as virtual machines under Proxmox.  Neither system is laggy unless the system is being stressed by something.  Both have most modules running.

Do either of you have Zarafa installed?  What is typically the system load as shown on the dashboard?  To help me understand,  how many cpus and how much ram on each machine  (if in a virtual environment that would be nice to know)?
Title: Re: [SOLVED] Zentyal extremely slow
Post by: christian on April 09, 2013, 05:09:58 pm
Access to Zentyal web interface on my 2.2 platform (running on HP N40L with 8Gb of memory) is definitely slow.
Server itself is running very smoothly but when I authenticate in order to apply admin changes, it always takes more than 30 seconds before dashboard is displayed and this is indeed slow.

Then using GUI is more or less OK but not what I would call fast.

3.0 is supposed to be faster but I didn't notice huge improvement from this standpoint during the few tests I made.
Title: Re: [SOLVED] Zentyal extremely slow
Post by: acesuares on April 11, 2013, 12:51:43 pm
half-life, thx for looking in to this.

The machines are all HP Microservers, http://h10010.www1.hp.com/wwpc/us/en/sm/WF06a/15351-15351-4237916-4237918-4237917-4248009.html?dnr=1

With either one or 2 GB.

Load is typically low,
 06:48:29 up 12 days, 20:54,  1 user,  load average: 0.16, 0.05, 0.06 from one machine
and  06:49:02 up 6 days, 20:29,  4 users,  load average: 0.49, 0.69, 0.75 from another
(I have 5 deployed and all with zentyal yayh!)

The web interface experience is like posted above, not at all snappy, even when on the console.
click...wait describes it well.

If you want, I can give you a login at a test machine :-)

In any case, I can work with it but it's not a good recommendation for to convince all those windows system managers to switch over

cheers
ace







Title: Re: [SOLVED] Zentyal extremely slow
Post by: acesuares on April 11, 2013, 12:53:35 pm
3.0 is not faster on the HP40NL!
Title: Re: [SOLVED] Zentyal extremely slow
Post by: christian on April 11, 2013, 01:16:29 pm
Running Zentyal 2.2 on HP N40L too (however without local client), I confirm that GUI is quite slow. However, I would not say it has significant impact on your decision to go or not to go for Zentyal.
I would appreciate to work with faster interface, of course, but given my hardware size, it means that I don't have so many users, therefore so many administrative tasks.
What really matters is that services are fast enough. From this standpoint, I'm pretty happy with my HP N40L. Notice that I'm almost not using Zentyal file sharing  ;) OMV is much faster  ;D
Title: Re: [SOLVED] Zentyal extremely slow
Post by: acesuares on April 11, 2013, 01:24:53 pm
> However, I would not say it has significant impact on your decision to go or not to go for Zentyal.

Really? Are we back in the days of linux 1.0 so that user experience doesnt' matter? Shame on you my friend ;-)

I myself am complaining about the UI, how would the MS lovers think about that?

Everything is fat (or normal, as you've come to learn the speed of GNU/Linux) except the UI.

There's got to be some thing that can be done about that...
Title: Re: [SOLVED] Zentyal extremely slow
Post by: christian on April 11, 2013, 01:57:56 pm
Really? Are we back in the days of linux 1.0 so that user experience doesnt' matter? Shame on you my friend ;-)

Did I ever wrote that user experience doesn't matter? I don't think so.
[end]user experience does matter indeed  :P
I agree that Zentyal GUI is slow. Again, I would appreciate faster one but again, this is not, to me a blocking point, especially if you are using, like me, HP N40L, meaning you have few users behind and therefore light admin tasks.

So, there is room for improvement, we all agree on this but admin making his decision only based on "GUI performance" criteria will be wrong, from my standpoint. That's where I disagree on you previous statement.

BTW, what do you mean with "Linux 1.0" ? I can't see any link between Linux in 90's and the fact that Zentyal GUI is slow ? 
Title: Re: [SOLVED] Zentyal extremely slow
Post by: acesuares on April 11, 2013, 03:08:08 pm
My dear Zen Hero

Maybe you have indeed so much Zen!

But what about having 10 HP40NL? what about 20?
Maybe then your 'decision' will be influenced differently!!

Ohmmmmmm
Title: Re: [SOLVED] Zentyal extremely slow
Post by: Sam Graf on April 11, 2013, 03:38:46 pm
You seem to be implying a couple of things:
I'm not sure either of the things you seem to be implying is true. Among those who have experienced a slow GUI, no one is truly disagreeing that it could benefit from improvement. If there is any disagreement, it would be over just what one can expect from "a bunch of Perl scripts" and a webserver. But we can't quantify that here.

Given the agreement on the key issue--the GUI is slow--I'm not sure what the point of the current "disagreement" is.
Title: Re: [SOLVED] Zentyal extremely slow
Post by: half_life on April 11, 2013, 04:15:47 pm
half-life, thx for looking in to this.

The machines are all HP Microservers, http://h10010.www1.hp.com/wwpc/us/en/sm/WF06a/15351-15351-4237916-4237918-4237917-4248009.html?dnr=1

With either one or 2 GB.

Load is typically low,
 06:48:29 up 12 days, 20:54,  1 user,  load average: 0.16, 0.05, 0.06 from one machine
and  06:49:02 up 6 days, 20:29,  4 users,  load average: 0.49, 0.69, 0.75 from another
(I have 5 deployed and all with zentyal yayh!)

The web interface experience is like posted above, not at all snappy, even when on the console.
click...wait describes it well.

If you want, I can give you a login at a test machine :-)

In any case, I can work with it but it's not a good recommendation for to convince all those windows system managers to switch over

cheers
ace

You didn't mention whether you use Zarafa or not.  Also how many users on these machines?
Title: Re: [SOLVED] Zentyal extremely slow
Post by: acesuares on April 14, 2013, 12:17:31 am
No Zarafa, no users, except 2 or 3 on samba in one case and 15 on samba in another case. I use it mainly as firewall/router and only use the file sharing bit - sometmes with sshfs and sometimes with netatalk and sometimes with samba.

Anyway, this discussion seems to lead nowhere - as a experieced and fanatic open source user, I am happy to use Zentyal for those clients that I can convince to let me do their management. But when I talk to windows sysops, there is already a lot of lack of knowledge - they keep talking about 'I don't know how the server is setup' when wat they really don't know is how windows filesharing and networking works.


I am just saying, the interface is slow, compared wiht untangle or gibraltar for instance.

That's all.

Cheers

ace
Title: Re: [SOLVED] Zentyal extremely slow
Post by: Sam Graf on April 14, 2013, 03:54:22 pm
Anyway, this discussion seems to lead nowhere…

There really is little place for a discussion to go, unless I'm missing something. As a community, we may differ on the possible causes or on the seriousness of the problem (obviously not everyone is experiencing it in the same way or objecting to it in the same way, and those experiences and responses aren't less relevant just because they are less severe), but we aren't disagreeing that there is, or at least often is, a long response time from the administration webserver.

If you were looking for the developers to say something, to admit a problem and to say they're working on it, even if that was the case they would have been more likely to do that in the past (and they actually have talked about this before here). Think in terms of community discussion about this in these forums, and if the developers engage it, better yet. But look to the bug tracker to more reliably interact with the developers.
Title: Re: [SOLVED] Zentyal extremely slow
Post by: patmagee1024 on May 05, 2013, 04:56:58 am
This has been happening for the last couple of versions. What I have done is remote in via ssh (or PuTTY) and type the following:
Code: [Select]
sudo service apache2 restart
Apache will restart and the console will respond much quicker for me. Since I don't have to change the settings all that often this is a decent workaround for me.
Title: Re: [SOLVED] Zentyal extremely slow
Post by: sspeed on May 27, 2016, 05:56:57 am
I realize this thread is very old, but it has relevance to anyone searching. I had one install of Zentyal that was very fast. I installed another that was extremely slow.  I couldn't figure out why, then realized I had installed the Antivirus module on the second.  I removed that and it's as snappy as the first now.  I hope this helps someone.
Title: Re: [SOLVED] Zentyal extremely slow
Post by: captain on August 29, 2016, 02:00:09 pm
Can u try this:

https://forum.zentyal.org/index.php/topic,28187.0.html

Thx