Zentyal Forum, Linux Small Business Server

Zentyal Server => Installation and Upgrades => Topic started by: thorsten on August 03, 2013, 04:52:28 pm

Title: Admin interface unreachable - help request
Post by: thorsten on August 03, 2013, 04:52:28 pm
Hi,

 I kindly ask for any help on unusuall behavioir of my Zental 3 server, which was running perfectly. the last thing i did was to change the directory for backup files. When I quit the admin interface, the was a claIm for unsaved changes, but I am shure there were none so I ignored this and closed the browser.

After a while, some dhcp cloents were not able to get an IP from Zentyal, so I tried to log in, BUT the login screen told me that I was logged off due to A time out. ok, I did a reboot.

Now it is not possible to get the logon screen of the admin interface at all, all webservices seem to be down, but apache seems to run. All other services are not available to clients (dhcp, dns, samba, whatever) A further reboot did not helü. During reboot I was able to see that mysql service failed to star, but I can not restart, e.g. By service mysql start.

Can somebody help. please?

Thx
Thorsten
Title: Re: Admin interface unreachable - help request
Post by: half_life on August 03, 2013, 05:37:13 pm
I have been where you are at on a 2.2 system.  Can you manually start dhcp?
Title: Re: Admin interface unreachable - help request
Post by: thorsten on August 03, 2013, 07:37:55 pm
Hi,

how do I restart dhcp within zentyal? Usuall way /etc/init.d/dhcp restart does not work - there is no dhcp3 or any dhcp file within that directory. service dhcp restart/start/stop or service dhcp3 restart/start/stop do not work eiter.

THX
Thorsten
Title: Re: Admin interface unreachable - help request
Post by: half_life on August 03, 2013, 07:47:15 pm
/etc/init.d/zentyal dhcp start
Title: Re: Admin interface unreachable - help request
Post by: half_life on August 03, 2013, 07:49:21 pm
If you would like to see what is available for you to control within the /etc/init.d/zentyal script just give it a bad parameter such as:

/etc/init.d/zentyal  test status


If you just call /etc/init.d/zentyal  (start or stop)  it calls all of the startup or shutdown services that zentyal controls.

Title: Re: Admin interface unreachable - help request
Post by: thorsten on August 03, 2013, 09:02:55 pm
Any command

/etc/init.d/zentyal dhcp start
/etc/inid.d/zentyal test status

results in:

Error in tempfile() using /var/lib/zentyal/tmp/XXXXXXXXXX: Could not create temp file /var/lib/zentyal/tmp/KwKLuDkhfP: No space left on device at /usr/share/perl5/EBox/Module/Base.pm line 999.

Just the file name behind /tmp/xxx changes....

Best regards
Thorsten
Title: Re: Admin interface unreachable - help request
Post by: half_life on August 03, 2013, 09:15:48 pm
Sounds like your root partition is full. 

Run
Code: [Select]
df -h
and paste the results back to here.
Title: Re: Admin interface unreachable - help request
Post by: thorsten on August 03, 2013, 10:31:41 pm
mmm... root itself looks fine, but var is full. But what is stored in there?

I do not use that - is there something I can whipe out? Also an increase by means of lvm would be possible...

filesystem                      Size  Used Avail Use% Mounted on
/dev/mapper/VG--01-LV--01_root   14G  6,1G  7,1G  47% /
udev                            3,9G  4,0K  3,9G   1% /dev
tmpfs                           1,6G  536K  1,6G   1% /run
none                             50M     0   50M   0% /run/lock
none                            3,9G     0  3,9G   0% /run/shm
cgroup                          3,9G     0  3,9G   0% /sys/fs/cgroup
/dev/mapper/VG--01-LV--03_tmp    23G  172M   22G   1% /tmp
/dev/mapper/VG--01-LV--04_srv    33G  413M   31G   2% /srv
/dev/mapper/VG--01-LV--02_var    33G   32G     0 100% /var
/dev/mapper/VG--02-LV--05_home  1,4T   80G  1,2T   7% /home
Title: Re: Admin interface unreachable - help request
Post by: half_life on August 03, 2013, 10:43:16 pm
Your /var mountpoint  is full. You can move all of the old logs off /var/log to gain some room.  I am suspecting that you use Zarafa?

Anyways, first things first.  You will need to create some free space on /var to return to normal operations.  Once you have gotten the machine back on its feet,  you will need to root cause this event.  I am guessing it will relate to the mysql server given how much room you have allotted for /var.  Thus my question about Zarafa.
Title: Re: Admin interface unreachable - help request
Post by: thorsten on August 03, 2013, 10:48:11 pm
Hi,

OK, the enemy is /var/cache/zentyal/duplicity with four directories and two of them with 9.9 GB ... is this what you suppose?
Title: Re: Admin interface unreachable - help request
Post by: half_life on August 03, 2013, 10:48:34 pm
I suggest
Code: [Select]
mv /var/log/*.gz /home to move all of the old logs.

I am curious as to how big your mysql installation is so try
Code: [Select]
du -sh /var/lib/mysql
Unless you feel like re-installing Zentyal today, don't randomly delete things stored under the /var directory.  Adding space via lvm is an option but there may be safer ways to approach this.
Title: Re: Admin interface unreachable - help request
Post by: half_life on August 03, 2013, 10:49:16 pm
That would be it!   Move those directories to /home and then see what is what.
Title: Re: Admin interface unreachable - help request
Post by: half_life on August 04, 2013, 01:41:32 am
Before I quit monitoring this thread for updates on the off chance you still need help,  have you gotten things straightened out?
Title: Re: Admin interface unreachable - help request
Post by: thorsten on August 04, 2013, 07:53:42 am
Sorry, I will have visitors today and can not work on the problem. /var/logs is quite smal, just some 100 MB compared to the LVM device size of 30 GB. I will move the files from /var/cache/zentyal/duplicity ..., is this any problem?

also:
/var/lib/mysql is just 1.2 GB.

THX in advance, will give you a PM in case I will need further help on this thread.

Thorsten
Title: Re: Admin interface unreachable - help request
Post by: half_life on August 04, 2013, 07:46:18 pm
I see no problem moving the two directories under /var/cache/zentyal/duplicity .   They are used to keep track of changes made to portions of the system that are backed up.  Removing them will trigger a full backup to the best of my knowledge. It should re-create one directory when it runs.  Longer term,  it probably would be best to make a little more room on the var mountpoint to avoid this issue
Title: Re: Admin interface unreachable - help request
Post by: thorsten on August 05, 2013, 08:55:48 am
Hi Half_life

this is a perfect explanation for an undiscovered failure where maloperation and system dullness goes hand in hand:

I had to recover files from system backup which is stored on an external USB-HDD mounted in /media/hdd_mobil/
For security purposes I did remove the HDD but I did not attach a new one. Additionally I was about to leave for a two weeks vaccation to a tiny tropical island. As the Island does not even have internet  8), there was no need to re-attach the backup HDD before end of holydays: The conserved data on the backup HDD were stored securely  :D.

But without the USB-HDD the scheduled Zentyal backups failed. I did not even think about that Zentyal would prepare a service directory that might grow up day by day - as there was no HDD, changes were noticed or "backed up" in /var/cache/zentyl/duplicity and nothing was cleaned or removed in the mean time. /var/ grow larger and larger.

Then, back at home after two weeks of sun and sea, I forgot to re-attach the USB-HDD. Washing of cloth and paying bills of two weeks was more important than working on (presumably unimportant) computer backup HDDs. On the fourth day after return, all Zentyal service stopped as described.

The root case - as you discoverd - was an overcrowed root (/var/) directory.

...

OK. Back to the problem: I added another 10 GB to respective logical volume mounted as /var/ via LVM. Cleaning of other cache areas by apt-get autoremove and autoclean gave another 4 GB. The system is now back on duty - execpt for user corner and jabber (I do not know why).

Also I will clean the backup by cleaning up the backup-HDD and reconfiguring the backup service. Additionally I will reduce size of /srv/ which is obviously to large for just my web-page php installations (1x drupal, 1x piwik + some redirected web-pages). This will give release enough space for future purposes within LVM if necessary.

I forgot to mention: I love LVM   :D

Best regards
Thorsten

PS: If you have any idea on the jabber / user corner issue, I would be glad, but this is not urgent.
Title: Re: Admin interface unreachable - help request
Post by: half_life on August 05, 2013, 07:04:17 pm
Have you tried /etc/init.d/zentyal usercorner stop

followed by /etc/init.d/zentyal usercorner start  ?


Title: Re: Admin interface unreachable - help request
Post by: thorsten on August 06, 2013, 09:51:52 am
No, just the restart via the admin interface.
Title: Re: Admin interface unreachable - help request
Post by: thorsten on August 06, 2013, 01:30:32 pm
No, no success - just some strange behaviour:

Code: [Select]
sudo /etc/init.d/zentyal usercorner start
Gives just:
Code: [Select]
* Restarting Zentyal module: usercorner
... done.

In contrast to this, admin interface still reports the service to be stopped and access via https://Zentyal-IP:8888

gives :
Code: [Select]
Forbidden
You don't have permission to access / on this server

Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle this request

Today evening (GMT+1) I will try to follow this
http://forum.zentyal.org/index.php?topic=10547.0
Anyway, first inspection of /etc/init.d/passwd seems to be OK, but I will have to try using a better (readable / usable) terminal besides my ipad    :o

Any other ideas?
Title: Re: Admin interface unreachable - help request
Post by: half_life on August 06, 2013, 04:35:52 pm
You could restart all zentyal processes to see if it is a timing issue.  This will take your server off line for a few minutes

Code: [Select]
/etc/init.d/zentyal  restart