Ran out of space, /var/lib/postgresql == 15GB!

[RoboJ1M]

Hi,

Our ebox has run out of disk space

An 18GB disk:

Code: [Select]

root@router-internal:/# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              18G   16G  1.8G  90% /
varrun                505M   56K  505M   1% /var/run
varlock               505M     0  505M   0% /var/lock
udev                  505M   44K  505M   1% /dev
devshm                505M     0  505M   0% /dev/shm
15GB in the postgresql database folder:

Code: [Select]

root@router-internal:/# du -h / | grep '[0-9]G\>'
du: cannot access `/proc/15946/task/15946/fd/3': No such file or directory
du: cannot access `/proc/15946/task/15946/fdinfo/3': No such file or directory
du: cannot access `/proc/15946/fd/3': No such file or directory
du: cannot access `/proc/15946/fdinfo/3': No such file or directory
14G     /var/lib/postgresql/8.3/main/base/16384
14G     /var/lib/postgresql/8.3/main/base
15G     /var/lib/postgresql/8.3/main
15G     /var/lib/postgresql/8.3
15G     /var/lib/postgresql
15G     /var/lib
15G     /var
15G     /
Some sort of clean up I can run?
Sadly we're still on ebox 1.0

J.

[J. A. Calvo]

Have you tried to purge the logs?

[RoboJ1M]

eBox Logs? No, just the /var/logs
I need to reboot it first but I'll try that next.

[J. A. Calvo]

Yes, eBox logs are stored in PostgreSQL, you should purge them if you need space.

[AlninlA]

never mentioned before. thanks calvo.

[RoboJ1M]

Hi,

Purging the logs didn't free up any space.

Will a VACUUM do the trick?
http://www.postgresql.org/docs/7.4/interactive/sql-vacuum.html

Not sure how I can do it though!

J.

[RoboJ1M]

trying:

Code: [Select]

sudo su postgres
vacuumdb --dbname eboxlogs --full --analyze
It's, er, running.
I guess this may take some time.

J.

[RoboJ1M]

df shows that the disk went from 99% full to 79% full after the vacuumdb.
/var/lib/postgresql/8.3/main/base/16384 contains 12GB.

from psql:

Code: [Select]

eboxlogs=# select count(*) from firewall;
   count
-----------
 103298239
(1 row)
100 million rows! Looks like purge didn't do much

I'm going to do DELETE FROM firewall.

For next time, is TRUNCATE safe?

J1M.

[RoboJ1M]

Solved, needed to do:

Code: [Select]

$ sudo su postgres
$ psql

\c eboxlogs
delete from firewall;
vacuum full analyse;
12GB of space freed up.

Regards,

J1M.

[naelq]

Yes, eBox logs are stored in PostgreSQL, you should purge them if you need space.

don't you think a 12GB of logs is too much?! shouldn't the system delete them automatically after a given period? or maybe have the option to archive/compress old logs?
with such massive growth of logs, any system is potentially unstable due to possible lack of free space!

nael

[J. A. Calvo]

http://doc.ebox-platform.com/en/logs.html#logs-configuration

[naelq]

great, but still, i don't see any size limit, which should be considered a failsafe.


nael

[Saturn2888]

I was just going to note that myself. Size-limits are very important if logs are going to get that big. I've always thought my 8GB logs were huge.

[cheesyking]

8GB is nothing.

I had a server set to keep proxy logs for a couple of months and noticed the free space on the drive was a lot less than I thought should be so I set the logs to purge after a week and thought nothing more of it.

a few more months later and bam, 100GB db file!

I assume there's some kind of problem where the purge command times out if the db is too big so it never purges and the problem just snowballs.

I'd agree with some way of limiting the maximum size of the logs stored in the database, ideally with some method of archiving out the old logs to a compressed file.

Postgesql's ability to keep on working until it ran out of disc space was pretty impressive though.