Zentyal Forum, Linux Small Business Server
Zentyal Server => Installation and Upgrades => Topic started by: Escorpiom on December 16, 2012, 08:45:58 am
-
So I did some reading on how to make Squid respond faster.
My Zentyal 2.2.7 server is running smooth, but I think it can be a bit better.
At this moment the whole installation sits on a WD raptor 10k harddisk. Squid cache is 10GB.
To obtain faster response times I would like to change the physical media and the way it is used.
These are some possible ideas.
- Put it all on SSD drive. Drawback is that it will wear out pretty soon.
- Leave Zentyal on the raptor 10k, add the 120GB SSD and use only for /var, lots of free space to do balanced writes and prolong SSD life
- Put in another 4GB of RAM and make a Linux ramdisk to use that as Squid cache-dir
- Use compact flash card for /var. Drawback that it can be damaged easily or pulled out of the system by someone
- Robb a bank and buy the SLC or eMLC based SSD that will last as a cache drive
And the software "tweaks" as you may call them:
- Use reiserFS for the cache dir as it performs better with small files (Ubuntu12 support reiserFS I lazily assume)
- Use aufs for populating the cache dir instead of ufs
- Use more memory for Squid so less physical storage is used (that would need hacking into config files)
- Go adventurous and install Zentyal 3.0 to get Squid 3.x that is not single threaded as opposed to Squid 2.x
Well that sums it up, a lot of possible solutions.
I would love to hear some comments on this, what you think is best or how do you have it setup at your turf.
Little remark: No raid setup please, goal is to make something more energy efficient and getting rid of spinners is preferred.
The number of users lies anywhere between 40 and 60.
Cheers.
-
This is a pretty interesting discussion.
Before trying to improve disk cache efficiency, I would start looking at "how cache is used?" because if cache usage is low, then improving it will give little benefit.
Are you sure Squid 3.x is always faster than 2.7 ?
-
Well, I'm no Squid expert at all, but from what I've read Squid 3.x can use multi core processors.
Squid 2.x still only uses one core, it is single threaded.
Imagine the scenario where we use SSD as a cache, latency due to disk access will be very low as compared to conventional harddisk.
So if we use Squid 3.x it can fetch multiple objects from the cache at once, taking advantage of the low SSD latency.
That translates directly to better performance.
How the cache is used is another topic on itself, and it will be different depending on the environment used and the level of Squid tuning you want to get involved with.
When you live in a country with very limited Internet connectivity you learn to appreciate any savings in bandwidth, even if the hit rate is between 10% and 20%.
Cheers.
-
Needless to say that I'm totally overwhelmed by the flood of ideas from my fellow community members.
Advanced Zentyal tuning may not come at the right time, now that we're into the 3.x discussion.
I still hope for some ideas later on.
Cheers and merry Christmas!
-
Indeed there is very few room fore creativity given constraints you express:
- no raid but either SSD or ramdisk
- different file system
- some Squid tweak ?
I don't think this will generate that much debate. You should rather give a try and let us know your feedback :)
The only way I would not go is Zentyal 3.0 if goal is to build production platform. I'm pretty sure Squid will run smoothly (faster ? I don"t know) but given what I see on this forum, there is still quite a lot of problems with 3.0
-
Have a look here (http://forum.zentyal.org/index.php/topic,13494.0/topicseen.html) too ;)
Is 3.0 really faster or is it adventurous ???
-
Thank you Christian, as always you are coloring the Zentyal forum.
Yes, I read the topic you linked to. But the topic starter has not implemented any tweaks to Squid.conf or improved the filesystem.
Neither does he inform about the hardware he uses.
I found one important tweak, "mem_cache" that is very important for Squid performance. As far as the user count, if claudiompjr has 200 users
he could take advantage implementing Squid.
I just think the tweaks he made are not the right ones, but before giving advice I have to test it myself.
That said, I run Zentyal 3.0.9 and Squid at a clients shop, few users so not really groundbreaking but it is good to gather information and to learn.
Until now I haven't run in to serious issues, but I agree with you Christian that Zentyal 3.x is not to be considered trouble free at this point.
So...The logical question for the Zentyal team, will Squid 3.x be made available for Zentyal 2.2.7 also or is it exclusively for 3.x?
I'm thinking to go this way: Buy two SSD drives. One small SLC /eMLC based drive for /var. Frequent writes from logging, Squid cache and www dir.
It should be around 20GB maximum.
The other drive could be a normal SSD drive for more static data so it will last a bit longer.
In that way I could optimize the /var to my likings, and two SSD will still consume less power and generate less heat than my raptor 10k.
Cheers.
-
Thank you Christian, as always you are coloring the Zentyal forum.
I hope my color choice is not too dark 8)
I'm thinking to go this way: Buy two SSD drives. One small SLC /eMLC based drive for /var. Frequent writes from logging, Squid cache and www dir.
It should be around 20GB maximum.
The other drive could be a normal SSD drive for more static data so it will last a bit longer.
In that way I could optimize the /var to my likings, and two SSD will still consume less power and generate less heat than my raptor 10k.
Although I didn't comment this part that much, I'm with you on the approach: for my own Zentyal, with other objectives in mind, I did split /var from other spindle, using SSD for "system" partition, mirroring for data (i.e. Samba and mail) and kind of trash disk for /var. I'm pretty convinced, even if I did not measure anything, that the way you organize your partitions on hard disks or SSD does matter.
-
gotta pull this topic up :)
- Use reiserFS for the cache dir as it performs better with small files (Ubuntu12 support reiserFS I lazily assume)
- Use aufs for populating the cache dir instead of ufs
- Use more memory for Squid so less physical storage is used (that would need hacking into config files)
These things I want to set up myself. So reiserfs in Zentyal 12.04 is really better than ext4 (with small files)? And also I should add noatime and notail options in fstab (where the reiserfs partition should be mounted on /var/spool/squid3) And changing ufs to aufs and increasing memory limit means changing Zentyal stub files. Now there comes a question. Under /usr/share/zentyal/stubs/squid there are squid.conf.mas and squid-external.conf.mas. Which is the right one for modifying cache_dir and memory limit? And which is the best way to restart/reconfigure squid so it can take the changes from the stub files?
-
I have changed only the "squid.conf.mas" file and can confirm it works.
You only have to restart the http proxy service from the Zentyal admin interface, but you may use the terminal if you feel comfortable.
I decided to go the two-SSD setup. When Zentyal 3.2 is released I plan to rebuild my server and use one consumer SSD for static files and one small SLC or eMLC based SSD for /var.
Still hope for a price drop.
Cheers.
-
Simply restart zentyal squid service and stubs take place? Nice!
Thanks! Will try out.
-
For me it was squid-external.mas.conf which after zentyal service restart then writes the changes into /etc/squid3/squid-external.conf where all the necessary lines were.
Now it looks like
admins@server:~$ sudo cat /etc/squid3/squid-external.conf
###################################################################################
http_port localhost:3130
visible_hostname (external)HOSTNAME
coredump_dir /var/spool/squid3
cache_effective_user proxy
cache_effective_group proxy
[b]cache_mem 768 MB[/b]
[b]cache_dir aufs /var/spool/squid3 39936 16 256[/b]
maximum_object_size 300 MB
access_log /var/log/squid3/external-access.log squid
cache_log /var/log/squid3/external-cache.log
cache_store_log /var/log/squid3/external-store.log
pid_filename /var/run/squid3-external.pid
[b]memory_pools off
buffered_logs on
client_db off[/b]
dns_nameservers 127.0.0.1 194.19.240.46 212.70.161.27
# refresh patterns
# windows updates
refresh_pattern http://.*\.windowsupdate\.microsoft\.com/ 0 80% 20160 reload-into-ims
refresh_pattern http://.*\.update\.microsoft\.com/ 0 80% 20160 reload-into-ims
refresh_pattern http://download\.microsoft\.com/ 0 80% 20160 reload-into-ims
refresh_pattern http://windowsupdate\.microsoft\.com/ 0 80% 20160 reload-into-ims
refresh_pattern http://.*\.download\.windowsupdate\.com/ 0 80% 20160 reload-into-ims
refresh_pattern http://office\.microsoft\.com/ 0 80% 20160 reload-into-ims
refresh_pattern http://w?xpsp[0-9]\.microsoft\.com/ 0 80% 20160 reload-into-ims
refresh_pattern http://w2ksp[0-9]\.microsoft\.com/ 0 80% 20160 reload-into-ims
# linux updates
refresh_pattern http://.*\.archive\.ubuntu\.com/ 0 80% 20160 reload-into-ims
refresh_pattern http://(ftp|http)[0-9]*\.[a-z]+\.debian\.org/ 0 80% 20160 reload-into-ims
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
# end refresh patterns
acl_uses_indirect_client on
# no cache domains acl
acl from_localhost src 127.0.0.0/8 ::1
acl to_localhost dst 127.0.0.0/8 ::1
acl manager url_regex -i ^cache_object:// +i ^https?://[^/]+/squid-internal-mgr/
acl SSL_ports port 443 # https, snews
acl SSL_ports port 873 # rsync
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 631 # cups
acl Safe_ports port 777 # multiling http
acl Safe_ports port 873 # rsync
acl Safe_ports port 901 # SWAT
acl CONNECT method CONNECT
acl purge method PURGE
follow_x_forwarded_for allow from_localhost
log_uses_indirect_client on
http_access allow manager to_localhost
http_access deny manager
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow from_localhost
# we use firewall to deny clients from the outside
http_access allow all
always_direct allow to_localhostLines between [ b][ /b] were modified, sorry for that.
Also modified the fs to reiserfs and mounted it with noatime and notail options. Transparent proxy seems to run more fluid to me now.
Another good find - if you sudo service zentyal squid stop than it immediately acts as a router and not forces the traffic to go trough proxy.
-
Yeah sorry about the external conf, I'm still running Zentyal 2.2.
The squid proxy in Zentyal 3.0 has a different setup.
Cheers.
-
Forgive me because I'm kinda new to this proxy tuning stuff. But only editing the external conf file and restarting the service (module) is enough in Zentyal 3.0+ ?
I'm asking that because on some other forums, not zentyal related, they had to recompile the package since squid only accepts UFS out of the box.
-
Editing this:
The Squid package for Zentyal (or better said Ubuntu) comes pre-compiled, and I wouldn't mess with compiling it myself.
I think you refer to aufs vs. ufs? Only way to find out is to try it.
Should work just fine though.
Edit again: Just looked it up, couldn't find any options regarding ufs/aufs at compile time, at least on Ubuntu.
Cheers.
-
A whole year later, and I'm still on the guest of improving Squid performance.
So far, the biggest performance increase came not from adding or changing hardware, but simply updating Zentyal
and thus getting an updated Squid 3.3.x in the process.
I must say, the difference as compared to Zentyal 2.2.x is huge, and it still runs from a 10k raptor drive.
As far as the SSD concerns, I've had plenty of time to read.
The conclusion is that SSD "wear" will be less of an issue when using quality SSD drives, even if they are MLC based.
For example, I've just ordered an Intel 2500 series 120GB SSD. Should be more reliable then other consumer SSD drives.
It's still a bit of an experiment, but at this point the SSD wear seems exaggerated.
It was mentioned that ramdisk would be the best solution.
My cache footprint would require 16GB of ram, and honestly I find those modules (2x8GB) to expensive. Maybe when RAM comes down in price, I'll give it a shot.
So far this little update.
Cheers.