Zentyal Forum, Linux Small Business Server
Zentyal Server => Installation and Upgrades => Topic started by: zette on February 12, 2009, 08:40:15 am
-
Hello
I've a week ago updated my ebox 0.12.3-1 box, to unstable 1rc2 version. Update process went seamlessly, but them problems started.
First of all dashboard does not work at all. On dashboard page I get two errors (I give You my own English translation since I use Polish as eBox language):
- Internal error occurred. You'll find more information in log files.
- Schema internal error occured. You'll find more information in log files.
But the worst thing is that now the system that was working for the last 6 months 24h/7 without any problems is hanging every few hours. Only hard reset helps to reinitialize system for the next few hours.
Currently I'm using only basic functionality - firewall, squid, traffic shaping, dns and dhcp.
Error logs are to big to be attached. Please let me know where to send those if You need it.
-
May you enable debug by turning variable "debug" to "yes" on /etc/ebox/99ebox.conf?
This may help us to find the issue you have.
-
Can you try to set the language to English and try again?
Could you also generate a bug report and send it privately to javi at warp dot es ? To download a bug report go to: https://YOUR_IP/ebox/EBox/Bug/
-
As to the hanging issue, it would be nice if you monitor the machine with the command top to see if there's a process going crazy....
-
I'll try english with debug turned on tommorow. System went down once again, and I do not have currently access to it.
As for tracking processes with top, I think that I saw that ebox logger was kind a memory and processor hog. But I'll confirm that also tommorow.
-
If you see that ebox-logger might be the culprit. Please try to disable it and see if the machine doesn't hang.
Cheers,
javi
-
I've disabled log module, but there still exists process that takes full working time of two processors and almost 2GB of ram. It is 90manageEBoxLog.
The good thing is that, the dashboard shows some more info after turning debug on.
When I try to generate debug report I get PageNotFound error. Can this be because of disabled log module?
Dashboar error:
No such daemon: ebox.postgrey
\n
\n$VAR1 = bless( {
'-stacktrace' => 'No such daemon: ebox.postgrey at /usr/share/perl5/EBox/Service.pm line 74
EBox::Service::running(\'ebox.postgrey\') called at /usr/share/perl5/EBox/ServiceModule/ServiceInterface.pm line 277
EBox::ServiceModule::ServiceInterface::_isDaemonRunning(\'EBox::Mail=HASH(0x9560f2c)\', \'ebox.postgrey\') called at /usr/share/perl5/EBox/ServiceModule/ServiceInterface.pm line 330
EBox::ServiceModule::ServiceInterface::isRunning(\'EBox::Mail=HASH(0x9560f2c)\') called at /usr/share/perl5/EBox/SysInfo.pm line 69
EBox::SysInfo::modulesWidget(\'EBox::SysInfo=HASH(0xa48236c)\', \'EBox::Dashboard::Widget=HASH(0xab76b74)\', \'undef\') called at /usr/share/perl5/EBox/Module.pm line 690
EBox::Module::widget(\'EBox::SysInfo=HASH(0xa48236c)\', \'modules\') called at /usr/share/perl5/EBox/CGI/Dashboard/Index.pm line 56
EBox::CGI::Dashboard::Index::masonParameters(\'EBox::CGI::Dashboard::Index=HASH(0xa3e779c)\') called at /usr/share/perl5/EBox/CGI/Base.pm line 508
EBox::CGI::Base::_process(\'EBox::CGI::Dashboard::Index=HASH(0xa3e779c)\') called at /usr/share/perl5/EBox/CGI/Base.pm line 261
EBox::CGI::Base::run(\'EBox::CGI::Dashboard::Index=HASH(0xa3e779c)\') called at /usr/share/perl5/EBox/CGI/Run.pm line 86
EBox::CGI::Run::run(\'EBox::CGI::Run\', \'Dashboard/Index\') called at /usr/share/ebox/cgi/ebox.cgi line 19
ModPerl::ROOT::ModPerl::Registry::usr_share_ebox_cgi_ebox_2ecgi::handler(\'Apache2::RequestRec=SCALAR(0x9005b38)\') called at /usr/lib/perl5/ModPerl/RegistryCooker.pm line 204
eval {...} called at /usr/lib/perl5/ModPerl/RegistryCooker.pm line 204
ModPerl::RegistryCooker::run(\'ModPerl::Registry=HASH(0x93e8148)\') called at /usr/lib/perl5/ModPerl/RegistryCooker.pm line 170
ModPerl::RegistryCooker::default_handler(\'ModPerl::Registry=HASH(0x93e8148)\') called at /usr/lib/perl5/ModPerl/Registry.pm line 31
ModPerl::Registry::handler(\'ModPerl::Registry\', \'Apache2::RequestRec=SCALAR(0x9005b38)\') called at -e line 0
eval {...} called at -e line 0
',
'-file' => '/usr/share/perl5/EBox/Service.pm',
'-text' => 'No such daemon: ebox.postgrey',
'-line' => 74,
'-package' => 'EBox::Service'
}, 'EBox::Exceptions::Internal' );
\n
\n
-
Hi!
Thanks a lot for this report. This is really useful to track down this issue :)
-
OK, I managed to run that host for a week without hang. Unfortunately it is because, I log in every few hours and kill 90manageEBoxLog processes.
This is temoprary solution, but better than nothing. On the other hand, this way I'm almost sure that there is something wrong with this module. Is there any way to manually disable start of this service, or is it critical and should not be touched?
-
You can safely move /etc/cron.hourly/90manageEBoxLogs to another location, and the process won't start.
We are still trying to figure out what's wrong with it.
Is there any chance you could let us take a look into your eBox logs database stored in postgres?
Thanks!
-
Just let me know, what privilidges You need.
-
Hi,
Just let me know, what privilidges You need.
It's enough if i can get a dump of your eboxlogs by running:
sudo su postgres -c "pg_dump eboxlogs" | bzip2 > /tmp/eboxlogs.bz2
Thanks!!
-
Hi all!
Here I have exactly same error with this 90manageEBoxLogs process in cron hourly. It happens in versión 1.0 stable also!!!! This is critical issue for production servers because each time when this process starts all the system stops responding, and load average gets number 20 or more in my system.
Temporal solution for my was move /etc/cron.hourly/90manageEBoxLogs to another place and disable logs module.
I think it is the same issue when I try to purge logs from ebox console.
Regards!
Pablo.
-
Hi pablodav,
How did you install eBox? Did you upgrade from rc2 to 1.0 stable, or you did a fresh install of 1.0 stable?
-
Hi,
We upgraded (hot swapped machine, not an apt-get upgrade) to latest stable version from repos last night.
Now suddenly 90manageEBoxLogs is using 2GiB of RAM. Except we only have 512MiB and machine has ground to a halt.
Anything we can provide to help fix?
Don't know if it's related but up until this point we've had to periodically restart ebox-firewall (or any routing machine between us and the internet, such as our Demon Cisco router) because eventually ping times go from 40 to 400 and half our bandwidth goes missing.
Regards,
James
-
Hi,
Now suddenly 90manageEBoxLogs is using 2GiB of RAM. Except we only have 512MiB and machine has ground to a halt.
Anything we can provide to help fix?
As an intermim solution you can move /etc/cron.hourly/90manageEBoxLogs to another place to disable it. And you should disable the log module on eBox too while we find out what happens. It would very helpful if you let us take a look at your logs. Running the following command you will dump the logs to /tmp/eboxlogs.bz2
sudo su postgres -c "pg_dump eboxlogs" | bzip2 > /tmp/eboxlogs.bz2
Thanks
-
Can't attach logs, too big (578k compressed)
Can I have an email address PM'd please?
Thanks,
J.
-
Sure,
juruen at ebox-platform dot com
-
Hy guys!
Any news about this matter? I'm having the same problem with 90manageEBoxLogs, since I upgraded from 0.25 to 1.0. I'm purging my logs to see if it is any help...
Thanks
-
Hello guys it's me again :D
Just wanted to say that after purging my logs (older that 15 days was enuff), 90manageEBoxLogs now is much lighter.
When I first tried to purge the logs apache2 process ran like crazy. Because I'm runing ebox on a VM and only attributed 128MB RAM to it, the apache2 process eventually got killed since the VM ran out of memory... same thing happened to 90manageEBoxLogs. The strangest thing, was that not even one log was purged.
To try and overcome the problem, I set the RAM of the VM to 512MB and voila, the logs where finally purged after 20 minutes or so...
After this, I reseted the RAM back to 128MB and 90manageEBoxLogs is much lighter now and is never killed...
In light of this, my advice to anyone having problems with 90manageEBoxLogs is to keep purgin the logs 90 days old, 30 days old and so on... When the purging process becomes quick, there's a big chance that the problem is solved. At least that's my view on this subject and it worked for me ::)
Cheers!
-
I have the same Problem with the Version 1.2.2
Purging the logs (90 days back) last one hour and more and it do not help. The other processes are nearly stopped. Sometimes it is not possible to login at console.
I have error/warning/info/debug messages in the ebox.log file:
WARN> ConfigureLogDataTable.pm:184 EBox::Logs::Model::ColnfigureLogDataTable::updateRowNotify - Domain: samba does not exist in logs
DEBUG> LogFiltering.pm:70 EBox::Events::Model::Watcher::LogFiltering::new - Missing argument: tableInfo
i moved the file 90manageEBoxLogs from /etc/cron.hourly to another location. Then this script is running no longer.
However i can start the eboxlogs purging at the webinterface. Then the computer is slowing down.
Is it possible to purge the logs manually?
Can sombody help me with this problem
Thanks a lot
Manfred
-
As luck would have it, I am now experiancing this issue on a production machine. Takes down 25 users.. :(
I told someone how to restart the machine just now after it happening the other day and I found the server with 100% utilization.
I'll try to post info when I get to the jobsite.. If there are more hints, ideas that got posted here beforehand, that'd be nice.. :)
Not enough time in the day...
-
Hi dragonslayr,
What eBox version do you have?
If you see your cpu usage at 100% try to identify the process. If it's ebox-loggerd try just:
sudo /etc/init.d/ebox logs restart
-
Hi dragonslayr,
What eBox version do you have?
If you see your cpu usage at 100% try to identify the process. If it's ebox-loggerd try just:
sudo /etc/init.d/ebox logs restart
Hi Javi!
ebox:
Installed: 1.2.9-1ubuntu1~ppa1~hardy1
This is the first chance I've had to get to the job site. The problem has not reared it's ugly head since my other message on here. However, the machine seems to run a few days before the problem happens again. Always in the log is the cron.hourly line and that's it..
I am exploring the system and hopefully discover something..
-
Because I'm runing ebox on a VM and only attributed 128MB RAM to it, the apache2 process eventually got killed since the VM ran out of memory... same thing happened to 90manageEBoxLogs. The strangest thing, was that not even one log was purged.
Is not so strange, because before the purge the last hour logs are parsed to create the reports by time period. There are only the last hour logs but maybe it is the straw that broke the camel back...
-
Is it possible to purge the logs manually?
Sure. You can enter the PostgreSQL shell with the following command:
su ebox -c 'psql eboxlogs' , executed as root.
Inside the shell you can use the command '\dt' to see the database's table. You can delete entries using standard SQL commands. For example if you want to delete all entries from the log of the mail module, the command is: DELETE FROM message Of course, you can use more surgical deletes using the features of the SQL language. (If you need help, ask here)
If you purge the logs manually, please let me know if you problem has gone away
-
One of the possible causes of this error is that we're having a race condition. To check this, I would be useful to check if there are two or more instances of 90manageLogs running at the same time in any computer which shows this problem. You can use the command ps aux to see what are the active process in the computer.
-
Same problem here, i have to move the 90manageEBoxLogs
(ps aux | grep perl)
Only one instance was running. (i kill -9 it)
Now the system is stable again.
I have a log error on the web interface some hours before.
It was something like "can't write /var/log/ebox/ebox.log"
so i change the permissions to the file to 777 (775 don't work)
Then i shutdown the server to insert an other ethernet card. (shutdown -h now) -> I have to do this because the web interface was not responding, but i don't know if was the 90manageEBoxLogs script or the Internet connection (there was an other person there to insert the card).
----
El mismo problema.
El script 90manageEBoxLogs se ha comido toda la ram, lo he tenido que matar (solo estaba ejecutando 1 vez) y moverlo a otro directorio.
Horas antes al intentar acceder vía web, me ha dado un mensaje de error de escritura en los logs, he tenido que dar permisos a ese fichero para poder acceder al interface web ya que solo aparecía el error en vez del login.
El acceso al equipo era para añadir una tarjeta de red, al apagar el servidor este no respondía vía web y lo he apagado vía ssh.
Al encenderlo de nuevo, al cabo de unas 4 horas el servidor ha dejado de responder.
-
Sorry for bringing this topic up, but it seems that i have the same problem on 2.0.21 core. 90ManageEboxLogs is going mad...
I tried to do whta javier says in post 25 but when i type /dt i just don't get anything else that the prompt of the shell, >.
Is there any solution for this problem?
Thnks