Author Topic: Critical(?) eBox 1rc2 () problems  (Read 4903 times)

zette

  • Zen Apprentice
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Critical(?) eBox 1rc2 () problems
« on: February 12, 2009, 08:40:15 am »
Hello
I've a week ago updated my ebox 0.12.3-1 box, to unstable 1rc2 version. Update process went seamlessly, but them problems started.
First of all dashboard does not work at all. On dashboard page I get two errors (I give You my own English translation since I use Polish as eBox language):
- Internal error occurred. You'll find more information in log files.
- Schema internal error occured. You'll find more information in log files.

But the worst thing is that now the system that was working for the last 6 months 24h/7 without any problems is hanging every few hours. Only hard reset helps to reinitialize system for the next few hours.


Currently I'm using only basic functionality - firewall, squid, traffic shaping, dns and dhcp.

Error logs are to big to be attached. Please let me know where to send those if You need it.

sixstone

  • Zentyal Staff
  • Zen Hero
  • *****
  • Posts: 1417
  • Karma: +26/-0
    • View Profile
    • Sixstone's blog
Re: Critical(?) eBox 1rc2 () problems
« Reply #1 on: February 12, 2009, 10:44:29 am »
May you enable debug by turning variable "debug" to "yes" on /etc/ebox/99ebox.conf?

This may help us to find the issue you have.
My secret is my silence...

javi

  • Zen Hero
  • *****
  • Posts: 1042
  • Karma: +0/-0
    • View Profile
Re: Critical(?) eBox 1rc2 () problems
« Reply #2 on: February 12, 2009, 10:59:26 am »
Can you  try to set the language to English and try again?

Could you also generate a bug report and send it privately to javi at warp dot es ? To download a bug report go to: https://YOUR_IP/ebox/EBox/Bug/

javi

  • Zen Hero
  • *****
  • Posts: 1042
  • Karma: +0/-0
    • View Profile
Re: Critical(?) eBox 1rc2 () problems
« Reply #3 on: February 12, 2009, 11:08:35 am »
As to the hanging issue, it would be nice if you monitor the machine with the command top to see if there's a process going crazy....

zette

  • Zen Apprentice
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: Critical(?) eBox 1rc2 () problems
« Reply #4 on: February 12, 2009, 09:01:31 pm »
I'll try english with debug turned on tommorow. System went down once again, and I do not have currently access to it.

As for tracking processes with top, I think that I saw that ebox logger was kind a memory and processor hog. But I'll confirm that also tommorow.

javi

  • Zen Hero
  • *****
  • Posts: 1042
  • Karma: +0/-0
    • View Profile
Re: Critical(?) eBox 1rc2 () problems
« Reply #5 on: February 12, 2009, 09:14:40 pm »
If you see that ebox-logger might be the culprit. Please try to disable it and see if the machine doesn't hang.

Cheers,

javi

zette

  • Zen Apprentice
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: Critical(?) eBox 1rc2 () problems
« Reply #6 on: February 13, 2009, 09:19:50 am »
I've disabled log module, but there still exists process that takes full working time of two processors and almost 2GB of ram. It is 90manageEBoxLog.
The good thing is that, the dashboard shows some more info after turning debug on.

When I try to generate debug report I get PageNotFound error. Can this be because of disabled log module?


Dashboar error:

No such daemon: ebox.postgrey
\n

\n$VAR1 = bless( {
                 '-stacktrace' => 'No such daemon: ebox.postgrey at /usr/share/perl5/EBox/Service.pm line 74
   EBox::Service::running(\'ebox.postgrey\') called at /usr/share/perl5/EBox/ServiceModule/ServiceInterface.pm line 277
   EBox::ServiceModule::ServiceInterface::_isDaemonRunning(\'EBox::Mail=HASH(0x9560f2c)\', \'ebox.postgrey\') called at /usr/share/perl5/EBox/ServiceModule/ServiceInterface.pm line 330
   EBox::ServiceModule::ServiceInterface::isRunning(\'EBox::Mail=HASH(0x9560f2c)\') called at /usr/share/perl5/EBox/SysInfo.pm line 69
   EBox::SysInfo::modulesWidget(\'EBox::SysInfo=HASH(0xa48236c)\', \'EBox::Dashboard::Widget=HASH(0xab76b74)\', \'undef\') called at /usr/share/perl5/EBox/Module.pm line 690
   EBox::Module::widget(\'EBox::SysInfo=HASH(0xa48236c)\', \'modules\') called at /usr/share/perl5/EBox/CGI/Dashboard/Index.pm line 56
   EBox::CGI::Dashboard::Index::masonParameters(\'EBox::CGI::Dashboard::Index=HASH(0xa3e779c)\') called at /usr/share/perl5/EBox/CGI/Base.pm line 508
   EBox::CGI::Base::_process(\'EBox::CGI::Dashboard::Index=HASH(0xa3e779c)\') called at /usr/share/perl5/EBox/CGI/Base.pm line 261
   EBox::CGI::Base::run(\'EBox::CGI::Dashboard::Index=HASH(0xa3e779c)\') called at /usr/share/perl5/EBox/CGI/Run.pm line 86
   EBox::CGI::Run::run(\'EBox::CGI::Run\', \'Dashboard/Index\') called at /usr/share/ebox/cgi/ebox.cgi line 19
   ModPerl::ROOT::ModPerl::Registry::usr_share_ebox_cgi_ebox_2ecgi::handler(\'Apache2::RequestRec=SCALAR(0x9005b38)\') called at /usr/lib/perl5/ModPerl/RegistryCooker.pm line 204
   eval {...} called at /usr/lib/perl5/ModPerl/RegistryCooker.pm line 204
   ModPerl::RegistryCooker::run(\'ModPerl::Registry=HASH(0x93e8148)\') called at /usr/lib/perl5/ModPerl/RegistryCooker.pm line 170
   ModPerl::RegistryCooker::default_handler(\'ModPerl::Registry=HASH(0x93e8148)\') called at /usr/lib/perl5/ModPerl/Registry.pm line 31
   ModPerl::Registry::handler(\'ModPerl::Registry\', \'Apache2::RequestRec=SCALAR(0x9005b38)\') called at -e line 0
   eval {...} called at -e line 0
',
                 '-file' => '/usr/share/perl5/EBox/Service.pm',
                 '-text' => 'No such daemon: ebox.postgrey',
                 '-line' => 74,
                 '-package' => 'EBox::Service'
               }, 'EBox::Exceptions::Internal' );

\n
\n
« Last Edit: February 13, 2009, 09:24:36 am by zette »

javi

  • Zen Hero
  • *****
  • Posts: 1042
  • Karma: +0/-0
    • View Profile
Re: Critical(?) eBox 1rc2 () problems
« Reply #7 on: February 18, 2009, 12:03:10 pm »
Hi!

Thanks a lot for this report. This is really useful to track down this issue :)

zette

  • Zen Apprentice
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: Critical(?) eBox 1rc2 () problems
« Reply #8 on: February 24, 2009, 11:22:11 am »
OK, I managed to run that host for a week without hang. Unfortunately it is because, I log in every few hours and kill 90manageEBoxLog processes.
This is temoprary solution, but better than nothing. On the other hand, this way I'm almost sure that there is something wrong with this module. Is there any way to manually disable start of this service, or is it critical and should not be touched?

javi

  • Zen Hero
  • *****
  • Posts: 1042
  • Karma: +0/-0
    • View Profile
Re: Critical(?) eBox 1rc2 () problems
« Reply #9 on: February 24, 2009, 12:45:04 pm »
You can safely move /etc/cron.hourly/90manageEBoxLogs to another location, and the process won't start.

We are still trying to figure out what's wrong with it.

Is there any chance you could let us take a look into your eBox logs database stored in postgres?

Thanks!

zette

  • Zen Apprentice
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: Critical(?) eBox 1rc2 () problems
« Reply #10 on: February 24, 2009, 02:01:39 pm »
Just let me know, what privilidges You need.

javi

  • Zen Hero
  • *****
  • Posts: 1042
  • Karma: +0/-0
    • View Profile
Re: Critical(?) eBox 1rc2 () problems
« Reply #11 on: February 26, 2009, 02:07:05 pm »
Hi,

Quote
Just let me know, what privilidges You need.

It's enough if i can get a dump of your eboxlogs by running:

Code: [Select]
sudo su postgres -c "pg_dump eboxlogs" | bzip2 > /tmp/eboxlogs.bz2

Thanks!!


pablodav

  • Zen Apprentice
  • *
  • Posts: 2
  • Karma: +0/-0
    • View Profile
Re: Critical(?) eBox 1rc2 () problems
« Reply #12 on: April 07, 2009, 02:40:01 am »
Hi all!
Here I have exactly same error with this 90manageEBoxLogs process in cron hourly. It happens in versión 1.0 stable also!!!! This is critical issue for production servers because each time when this process starts all the system stops responding, and load average gets number 20 or more in my system.

Temporal solution for my was move /etc/cron.hourly/90manageEBoxLogs to another place and disable logs module.

I think it is the same issue when I try to purge logs from ebox console.

Regards!

Pablo.

javi

  • Zen Hero
  • *****
  • Posts: 1042
  • Karma: +0/-0
    • View Profile
Re: Critical(?) eBox 1rc2 () problems
« Reply #13 on: April 07, 2009, 10:26:17 am »
Hi pablodav,

How did you install eBox? Did you upgrade from rc2 to 1.0 stable, or you  did a fresh install of 1.0 stable?

RoboJ1M

  • Zen Monk
  • **
  • Posts: 51
  • Karma: +0/-0
    • View Profile
Re: Critical(?) eBox 1rc2 () problems (1.0 STABLE)
« Reply #14 on: May 22, 2009, 11:23:25 am »
Hi,

We upgraded (hot swapped machine, not an apt-get upgrade) to latest stable version from repos last night.
Now suddenly 90manageEBoxLogs is using 2GiB of RAM. Except we only have 512MiB and machine has ground to a halt.
Anything we can provide to help fix?
Don't know if it's related but up until this point we've had to periodically restart ebox-firewall (or any routing machine between us and the internet, such as our Demon Cisco router) because eventually ping times go from 40 to 400 and half our bandwidth goes missing.

Regards,

James