Zentyal Forum, Linux Small Business Server

Zentyal Server => Installation and Upgrades => Topic started by: highjo on October 14, 2013, 09:08:32 am

Title: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: highjo on October 14, 2013, 09:08:32 am
Hello All,

Sorry for this trivial kind of question newbie here. I must admit that zentyal made me understand certain part of networking easily and made me achieve lots of things faster than I thought and Kudos for that. However what is left for me to do is something I feel like I lack information and knowledge about.

I would like to have 30k bandwidth for all machines connected to zentyal and allow 2 or 3 machines full access to any site besides porn sites(porn blocked 24/7). The rest of the machines would have social networking and video streaming available only 12 PM to 1PM and from 7 PM to the following 8 AM and porn blocked 24/7.

I thought I should have 2 objects say "full_access_group" and "partial_access_group". When adding the objects by IP address I realized that since I enabled zentyal DHCP server, IPs can change , I will need the DHCP to always give the same IPs the same machines. my questions are resumed below:

Question 1: How to set bandwidth to 30k for all connecting device to zentyal

Question 2: How to apply filters on connection for the range to time explained earlier

Question 3: How to configure DHCP to always give the same IPs to the same devices

Question 4: I am currently using a Wireless Router for the internal network where I DHCP functionality is disabled and forwarded to zentyal PC IP. if I change the wifi router to a switch do I have to do anything on the switch for the devices connected to find the zentyal DHCP server?

Thanks for reading this and for helping out :D
Title: Re: bandwith management and fiters: what to use firewall or proxy
Post by: robb on October 14, 2013, 09:20:46 am
-> moved to Server section
Title: Re: bandwith management and fiters: what to use firewall or proxy
Post by: BrettonWoods on October 14, 2013, 09:23:46 am
Firstly with the title the answer is both as you will find that proxy for http and http filtering is the place to go.

Then you have all the protocols and applications that might be direct.

If you want static ip's then its a matter of defining them as network objects with mac addresses so that the dhcp will hand out static addresses.

I would love to be able to apply traffic rules by vlan or subnet actually you can in a way.
Its how you plan your network objects.

With the router just plug and play
Title: Re: bandwith management and fiters: what to use firewall or proxy
Post by: highjo on October 14, 2013, 09:34:42 am
Apologies for posting in the wrong sub-forum. sorry.

Hi BrettonWoods, the thing is my IP range is from 100 to 254. it's not like I wanted static IPs per se. I have static IPs for servers only and they are bellow 100. I was just wondering how possible it can be done with zentyal because there was a Windows server 2008 attempt to the same thing I am trying achieve by someone else (temporarily). The DHCP server always on the windows gives the same IPs for all machines not having static IPs.

I am not sure I understood the quoted part below.
 
Quote
I would love to be able to apply traffic rules by vlan of subnet put vanilla Zentyal you can't.

Thanks for this blazing fast answer
Title: Re: bandwith management and fiters: what to use firewall or proxy
Post by: BrettonWoods on October 14, 2013, 09:43:31 am
if you have it setup right with dedicated switches or vlans then you can create different object groups.

I have never done it and you would have to provide routing or bridges so that each subnet or vlan could talk to each other.

Because you can set polices via network objects then objects that are on switch1 get treated in a different way than objects on switch two.

vlans are probably easier to work with.

That way you wouldn't have to define all the mac addresses and management could take place at the patch panel.

Christian is more up on the networking as I am saying it could be done but have not myself.
Title: Re: bandwith management and fiters: what to use firewall or proxy
Post by: robb on October 14, 2013, 09:44:07 am
You can simulate static IP's by extending DHCP leasetimes to <pick a VERY long time>
Or do it as you would do it on a Windows DHCP server: bind mac address to an IP address. Once you have all devices listed, the work should be manageable.
The Zentyal way would be to create network objects and add them as reservations in DHCP.

VLAN's (or separated physical switches) will need multiple internal interfaces and introduce extra routing, but you will be certain that a device will be in a specific VLAN. However, wireless clients will be another challenge to maintain.
Title: Re: bandwith management and fiters: what to use firewall or proxy
Post by: christian on October 14, 2013, 09:54:58 am
I was just wondering how possible it can be done with zentyal because there was a Windows server 2008 attempt to the same thing I am trying achieve by someone else (temporarily). The DHCP server always on the windows gives the same IPs for all machines not having static IPs.

I'm not 100% sure I understand what you mean but it looks like there is an urban legend those days that makes people thinking that Microsoft DHCP server is both dynamic and static at same time, allocating always same IP to machines. What's an improvement  ;D ;D

This doesn't exist (AKAK)

As described in DHCP protocol (have a look at this RFC (http://tools.ietf.org/html/rfc2131) and pay attention to T1 & T2), at 50% of lease duration, client will contact DHCP server and extend his lease, which means that unless lease already expired (very unlikely at 50% of the duration  ;)) every client, whatever DHCP server brand, will keep its already acquired IP address.

My $0.02
Title: Re: bandwith management and fiters: what to use firewall or proxy
Post by: christian on October 14, 2013, 10:05:57 am
I would like to have 30k bandwidth for all machines connected to zentyal and allow 2 or 3 machines full access to any site besides porn sites(porn blocked 24/7). The rest of the machines would have social networking and video streaming available only 12 PM to 1PM and from 7 PM to the following 8 AM and porn blocked 24/7.

I thought I should have 2 objects say "full_access_group" and "partial_access_group". When adding the objects by IP address I realized that since I enabled zentyal DHCP server, IPs can change , I will need the DHCP to always give the same IPs the same machines. my questions are resumed below:

Bandwidth management and HTTP content filtering are 2 different concepts. You may apply both or only one but these are not linked (although both exposed through "HTTP proxy" interface)

If, for some reason, you want DHCP to always allocate same address to same machine, the only way is to "reserve" IP per MAC address using network object group.

Quote
Question 1: How to set bandwidth to 30k for all connecting device to zentyal
Reading and applying this (http://doc.zentyal.org/en/proxy.html#bandwidth-throttling).

Quote
Question 2: How to apply filters on connection for the range to time explained earlier
Reading and applying this (http://doc.zentyal.org/en/proxy.html#access-rules) and this (http://doc.zentyal.org/en/proxy.html#filter-profiles).

Quote
Question 3: How to configure DHCP to always give the same IPs to the same devices
Reading and applying this (http://doc.zentyal.org/en/dhcp.html#dhcp-server-configuration-with-zentyal).


Quote
Question 4: I am currently using a Wireless Router for the internal network where I DHCP functionality is disabled and forwarded to zentyal PC IP. if I change the wifi router to a switch do I have to do anything on the switch for the devices connected to find the zentyal DHCP server?
1 - If you change Wifi access point for a switch, how will Wifi devices connect ?
2 - Switches are (most of the time) unmanaged. You can see it as panel connecting cables (unless you start dealing with VLAN or other very specific stuff
Title: Re: bandwith management and fiters: what to use firewall or proxy
Post by: robb on October 14, 2013, 10:06:54 am
[picky mode] IP addresses stay the same unless there comes a new device and there is not an unassigned IP address in the pool available[/picky mode]

explanation:
suppose you have a pool of 5 IP's and set a leasetime of.... 1 month
first device that connects gets IP 1
5th device that connects gets IP 5

Next day, all 5 devices connect again but not in same order. They still get same IP address since DHCP has leasetime of 1 month and in DHCP table the macaddresses of day before are still stored.

Day 3 a 6th device tries to connect while none of the others are connected: Device gets IP 1 since not all IP's are active. 4 other devices can connect and will be provided _if possible_ their previous IP with a max of 5 devices total.
Title: Re: bandwith management and fiters: what to use firewall or proxy
Post by: christian on October 14, 2013, 10:10:22 am
Very good point + you're perfectly right.

Valid lease but disconnected device + DHCP pool fully "allocated" will end up with IP address change.
This is not nitpicking but accuracy  8)  My fault for having been lazy  :-[ :-[ and thank you for maintaining clever reading  :-*
Title: Re: bandwith management and fiters: what to use firewall or proxy
Post by: BrettonWoods on October 14, 2013, 12:15:45 pm
I was trying to keep things in a similar mode to my mind which is unfortunately simple.

A lot of the zentyal network management involves the usuage of network objects.
The simplest is a single declared IP with it partnering mac address.

Its a bit of a catch-22 scenario when it comes to simple minds and lazy sysadmins such as this one.

If I dont want to do it by defining mac addresses then there is the next level of ranges or scopes.

The next level of object is a network scope where you don't assign all the mac's
So either you have a separate nic interface with its own dhcp and own subnet and the network object is that range.
You have one of these for each specific network object range.

Its sort of plug and play range grouping at the patch panel, where you patching will result in different network access rights.

Probably the easiest way is to get a vlan capable switch that way you can use a single nic but segregate by vlan.
Each vlan has a dhcp server scope and its own subnet and its own network object range.

That bit is fairly easy as you can have several network object ranges in which zentyal can apply various rules for in various modules.

Then you need to start thinking of shared resources and back at the server apply the routing to get these subnets to talk to each other.

Being simple I have never tried it out, but keep meaning to as it does allow much control over the network based on the owner of a desktop.

I have to say I wish Zentyal had a further level of abstraction from IP to users and groups, as maybe this would make things more digestible for my simple brain.

If your wireless router or lan is connected to a vlan compatible switch all should work as its the switch, that will do all the clever work.

So really I would say use vlans and the switch manufactures software for grouping and aggregation. I have a lovely cisco 2970 catalyst and from experience, 3com are much easier to use. Maybe one day when I am suicidaly bored I might read the cisco documentation and do it. :)

There are ways to do it with out a vlan capable switch and I guess you could virtualise http://openvswitch.org/ and as long as you have the nic logistics correct it should work.

Title: Re: bandwith management and fiters: what to use firewall or proxy
Post by: highjo on October 15, 2013, 01:07:47 pm
Thanks so much for all insight exposed here .Sorry I had to step out yesterday and couldn't have time to access the forum. Honestly I was delighted reading each one the posters opinion and all information exchanged here are just like tutorials for me. This community is really knowledgeable and warm. I am glad I chose zentyal for my first trial to replace windows server. I will read link suggested by Christian and revert on how it goes for me.

Thanks for all the input
Title: Re: bandwith management and fiters: what to use firewall or proxy
Post by: highjo on October 16, 2013, 10:08:51 pm
Hello Good People.

Let me bother you a little again.

About Question 1 : what does Maximum unlimited size per client means?

About Question 3: I have developers objects which members are added using both IP and Mac addresses . Supposing I have 4 PCs , I have added the IPs 101,102,103,104. Now In DHCP > Interface (eth1) > configuration, in the Fixed address section I have chosen the object "developers".  Because I needed to copy their MAC addresses , I initially set the range from 150 to 254 so they were having IPs above 150. After adding the members to developers object , I set the range from 100 to 254. I applied the change and restarted the client (linux) network-manager service but the machines didn't get the IPs I thought would be attached to their MAC addresses.

Is there anything am doing wrong ?

Question 4 : In response to Christian, I wanted to use the wifi as Access Point(AP) and pick the connection from the switch which would then be connected to the zentyal on the internal network interface. Apparently That didn't work so I put the wifi (still acting like AP) back into the Internal network Interface and put the Switch inside the Wifi port (using it as a switch). it's looks like this way. internal network machines > swicth > switch(wifi) > zentyal > (2 ISPs Gateway) > internet . This seems to work but don't know yet the implication on speed etc.


Thanks very much in advance
Title: Re: bandwith management and fiters: what to use firewall or proxy
Post by: BrettonWoods on October 16, 2013, 11:13:21 pm
Question 1 is that the proxy?

That it means the max file size in one download if I remember correctly.

Its always easier when you are in front of a working machine.

Question 3

static ips need to be out of the range of dhcp scopes. Fell for that one myself until christian told me.

Question 4.
I am just back from the pub :) think I might have to wait to answer it.

But from first glance not that much
Title: Re: bandwith management and fiters: what to use firewall or proxy
Post by: highjo on October 18, 2013, 05:31:46 pm
Hello, most of my concerns are being answered here. There could be a lot more discussion about the question 4 or about what I did which doesn't look so conventional to me but as far as everything is working . I am cool .Special thanks to Christian and BrettonWoods
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: astana on October 18, 2013, 05:45:03 pm
I would like to have 30k bandwidth for all machines connected to zentyal . The rest of the machines would have social networking and video streaming available only 12 PM to 1PM and from 7 PM to the following 8 AM

Ok, I know you've got all the answers to your questions as the guys answering are technically very good, but I'd like to ask WHY?
So for example you have 10 possible machines (for example) you're wanting to limit to 30kb/s each, so you're 'allocating' or 'reserving' 300kb/s bandwidth. Now what happens when Mr.X arrives early, is he 'allowed' all 300kb/s as no-one else is there? Your solution tells me no, he'll be offered 30k/s.
Then you mention streaming, at 30kb/s.

I'm really down on this idea as I tried to implement exactly the same thing and it was total fail. The results are easy to imagine: Permanently slow internet access and never using the full capacity of the pipe.

Trying to manage bandwidth with squid is like using only a hammer to build a house, slow, painful, job ain't pretty in the end and you'll not make any friends.

No matter which way you slice it, squid hasn't got the tools to manage bandwidth allocation in an intelligent way.

I ended up dropping squid management of bandwidth and using a better gateway to divide usage. Result: No more complaints about slow internet, usage almost doubled.
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: BrettonWoods on October 18, 2013, 05:53:19 pm
In response to question4. I am more able to answer now.

Firstly yes its a speed thing as you will be connecting on through a wifi router port which is most likely 100mbs.

It all depends on the switch that you plugged into the router. If you have 1gig going into a 100mbs port then you are only going to get 100mbs.

I wouldn't say its ideal, it will work.

You should be able to connect the wan port of the wifi router and turn off the internal dns and dhcp and just have it as an access point.

Its all specific to the hardware but most should work in this manner.

You might also want to provide another nic and have your wifi on this port so that you can have different rules for wifi and wired lan.
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: highjo on October 18, 2013, 07:47:13 pm
Hello Astana  can you please elaborate on the solution you used for bandwidth management?
Thank you verymuch
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: astana on October 19, 2013, 03:56:29 am
All I did was get rid our of old gateway (a windows server box with kerio winroute) and replace it with a pfSense box. pfSense at least is intelligent enough to spread the bandwidth equally and do the job well.
So in the example above Mr X would get 100% bandwidth until someone else starts using it, and at that point they will share 100% of the pipe.

In our school we had 5Mbit/s shared amongst 60 odd computers. With squid doing the bandwidth management is was really really bad (youtube wouldn't ever stream, downloads crawled etc). With pfSense doing the job it felt like our connection was 10x as big.

In our new school with a 15Mbit/s pipe we can sit there maxxing it out and I've not had 1 complaint that our internet is slow!

I'm sure other routers would deal with this as well as pfSense but I don't know them to say.

btw.. This is with a pfSense box basic configuration, just firewall and routing, basic intrusion detection etc, running on a low spec desktop.
 
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: christian on October 19, 2013, 07:39:25 am
I'm glad to notice you have solved your problem. That's the main good point.
This said, and although I do share that current Zentyal implementation in term of proxy might be slow, you are comparing stuff that you should ot compare because features are different.

As I wrote previously, filtering and bandwidth management are different stuff.

If you run pfSense without filtering proxy but only QOS, this is obviously faster.
And what yo describe doesn't tell much in term of bandwidth. You state it in term of performance only. Is internet fast or slow?

Bandwidth management is something different, potentially linked with QOS.

The idea behind QOS is to say:
- lets manage priority in case multiple clients want to share same resource (here internet access) that is limited.
- priority can be managed in term of protocol or in term of client
- when there is no conflict (e.g. one single user or device), then 100% of bandwidth can be allocated. doesn't mean you internet access will be fast BTW: if you apply content filtering, it might be quite slow bt if you store result in cache, it will be faster for next user requesting same page (I make the assumption we discuss about HTTP)

I hope I've clarified what I mean  ;)
Bottom line: QOS or bandwidth management "out-of-the-box" doesn't really exist except if implemented with default values like "HTTP will get higher priority" but in such case rules have to be exposed so that you know what happens.

Last but not least, what I understand from what you wrote is, aside this "QOS, bandwidth management" debate, "pfSense is faster than Zentyal"
I do not challenge this but would appreciate some technical feedback  :)
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: astana on October 19, 2013, 01:09:27 pm
Christian,

I wasn't even saying pfSense if faster than Zentyal. I was saying squid does not have the tools to do bandwidth management.
It really is that simple. Squid can limit to a known value which is a totally blunt instrument for managing capacity.

Unfortunately in Zentyal proxy has a bandwidth management tab which I think leads people astray to thinking it will help solve problems like 1 user taking all bandwidth and everyone else being starved. The end result of using this tab (ie using squid to control capacity) is a failed internet policy.

I've never used Zentyal as a gateway so I can't comment on it's QOS, nor did I imply that Zentyal was inferior.

Hope that makes my position clear.
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: BrettonWoods on October 19, 2013, 03:12:41 pm
Astana, what do you suggest instead and this is out of interest and not a criticism.

Its good that you have an opinion on this.

I always thought the Zentyal implimentation is a little blunt as it is supposed to be simple to use.

There are other methods of squid, but do you believe there are better ways to proxy manage?

http://www.enterprisenetworkingplanet.com/netos/article.php/3352971/Rein-In-Your-Bandwidth-Hogs-with-Squid-Proxying.htm

http://knowlinux.blogspot.co.uk/2006/04/bandwidth-throttling-using-squid.html

http://www.tldp.org/HOWTO/pdf/Bandwidth-Limiting-HOWTO.pdf

I do find it a bit worrying that if anybody has a differing opinion or offers a critique they are not seen as healthy discussion.

So I am up for a bit of squid bashing purely because I am unsure of other methods.

Its good that we have opinion as I use Zentyal as my gateway of choice because I find other features lacking in other packages.
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: astana on October 20, 2013, 04:26:28 am
Hi Mr Woods ;)

This isn't really about squid bashing or bashing anything. It's a simple discussion about what each software component offers on a technical basis.
Squids only method of bandwidth throttling is delay pools.
Once you read up and understand the basic principle of delay pools and then apply it to normal scenario network usage you realise it is really an incredibly blunt tool. It is impossible to spread the bandwidth fully and evenly amongst a variable number of users in time.
The only thing it can do is divide bandwidth and allocate that to each user. This might have niche and important uses, but for a lot of network setups (sb/home etc) it doesn't do your pipe any justice.

All the different ways of configuring squid delay pools boil down to the same technology (see above).

Lets work with a (possible typical) example: Company Office, 3 Departments: Admin/Dev/Sales

Proxy limiter: 1MB/s to each office.

Results when Dev are in crunch time and it's 8pm and they have to upload a 4GB Image, limited to 1MB/s for no good reason. Devs now hate Network administrator for life (they know you have a 3MB/s pipe).

Results without Limiting:
10am and Devs have to reupload their image (they found a bug at midnight). They use all available bandwidth until sales start skyping with clients, then bandwidth is automatically shared with voip getting priority. Result is everyone is happy (except for sleepless devs, but there's nothing you can do about that).

during this time admin are playing solitaire so don't enter into the equation (small joke).

----------------------------

My original comments weren't designed to hype an alternative solution or to bash Zentyal, just I was seeing someone who thought these zentyal options would give him the solution he required.

I have no experience of how Zentyal deals with traffic when acting as a gateway. Can someone enlighten me if it works like pfSense (fairly allocating all bandwidth through the pipe with highly configurable QoS etc)?

So as a conclusion I'd just like to say I'm really hard pushed to find any reason at all to using squid to manage bandwidth.

edit 2:
Maybe I also didn't make myself clear in my setup: I use Zentyal is proxy/filtering through an explicit proxy with authentication but do not use Zentyal as a gateway (this is obviously handled by pfSense).
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: christian on October 20, 2013, 07:01:15 am
Astana:

I previously read your post too fast (again) an though you were running either pfSense or Zentyal. That's clear now.

What you are looking for is QOS (Quality of Service): you want to define priority with rules that are applied only when you reach resource capacity. This results in traffic shaping. This does exist at routing level for protocols, meaning this is perhaps managed at pfSense level) but it doesn't exist at proxy level. Furthermore, your design choice introduces extra complexity (from this QOS standpoint) as you have split HTTP proxy and internet gateway, meaning even if it was possible, from psFsense, all requests come from proxy, with one single shaping rule.
On the other hand, I believe your design has some added value for you, I'm not challenging your design.

As you do understand, pool delaying is not QOS but a way to limit bandwidth once user reaches is "quota", which is very different.
QOS classifier (in pfSense like for almost all QOS implementations) is based on queuing and hierarchy per service (service here meaning protocol).
This is done at iptables level where the "account" concept is unknown.

Even if you were working at layer 7 level, this would not work with proxy because you don't know, when request is performed by proxy, if content if for one user or another.

What you could perhaps do is to duplicate proxies. One proxy for dev. One proxy for other users. at gateway level, allocate lower priority to Dev's proxy. When there is no congestion, they will get all the available bandwidth but will be limited in case there is some congestion. Still I don't know if pfSense permits this.
Side effect of such approach is that unless you set-up complex proxy network were each proxy can share its cache with other proxies, you impact proxy cache efficiency as same page might be stored twice in different proxy.

Does it make sense ?
 
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: astana on October 20, 2013, 07:14:23 am
Astana:

I previously read your post too fast (again) an though you were running either pfSense or Zentyal. That's clear now.

What you are looking for is QOS (Quality of Service): you want to define priority with rules that are applied only when you reach resource capacity. This results in traffic shaping. This does exist at routing level for protocols, meaning this is perhaps managed at pfSense level) but it doesn't exist at proxy level. agreed. QOS is for saying for example voip has higher priority over http which has higher priority over bittorrent.Furthermore, your design choice introduces extra complexity (from this QOS standpoint) as you have split HTTP proxy and internet gateway, meaning even if it was possible, from psFsense, all requests come from proxy, with one single shaping rule.However pfSense is intelligent enough to notice that the requests are coming from different clients ports and can automatically balance all HTTP traffic from the one zentyal proxy server (i.e. all clients are fairly balanced)!
On the other hand, I believe your design has some added value for you, I'm not challenging your design. This is the only design I found that allowed full use of the bandwidth for all users without throttling. As I said I already had a gateway so I have no idea if Zentyal performing as a gateway solves this problem

As you do understand, pool delaying is not QOS but a way to limit bandwidth once user reaches is "quota", which is very different.
QOS classifier (in pfSense like for almost all QOS implementations) is based on queuing and hierarchy per service (service here meaning protocol).
This is done at iptables level where the "account" concept is unknown.Correct, QOS is about different types of packet having different priorities, and nothing to do with throttling

Even if you were working at layer 7 level, this would not work with proxy because you don't know, when request is performed by proxy, if content if for one user or another. Actually this is false as each client connects on a random port to the proxy and has a different destination IP address, so yes you can differentiate that different users are connected to the one proxy host

What you could perhaps do is to duplicate proxies. One proxy for dev. One proxy for other users. at gateway level, allocate lower priority to Dev's proxy. When there is no congestion, they will get all the available bandwidth but will be limited in case there is some congestion. Still I don't know if pfSense permits this. pfSense allows the use of a proxy, but pretty much the same configuration as Zentyal.
Side effect of such approach is that unless you set-up complex proxy network were each proxy can share its cache with other proxies, you impact proxy cache efficiency as same page might be stored twice in different proxy.This is what cache digest is for. You can configure squid to check with it's neighbours to see if they already have the item in the cache.

Does it make sense ? It all makes sense, but I'm thinking you're still missing the point I'm trying to make.
Replies in red

Really the crux of the matter is: When Zentyal isn't acting as a gateway, how do you fairly distribute bandwidth to all users of the proxy without using squid throttling (which I'm sure you'll agree is a really bad tool for the aim of sharing a fixed pipe fairly).
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: christian on October 20, 2013, 07:40:10 am
Really the crux of the matter is: When Zentyal isn't acting as a gateway, how do you fairly distribute bandwidth to all users of the proxy without using squid throttling (which I'm sure you'll agree is a really bad tool for the aim of sharing a fixed pipe fairly).

As far as I understand technology, you can't. But I don't know everything and perhaps (for sure  ;)) someone else has better knowledge or idea.

Regarding layer 7 QOS: this is not matter of source but matter of content  ;) and back to proxy: real question (well, the one that really matters for you) with proxy is not who is connected and requesting (page might be stored in cache and results in no bandwidth usage) but how proxy, as network client, will consume bandwidth. That's why Squid "quote" makes some sense: limiting on external interface is extremely difficult, let's limit at the source side. But again this is not real QOS and priority management.

So, no I don't understand your point: either you know some technology that would permit to reach what you describe and then you could make a proposal for implementation or you are only looking to solution to your problem (which as a problem, is clear enough) and my answer is: too bad, AFAIK, it doesn't exist. Do you wonder why even pfSense doesn't provide something different from Zentyal ?

Regarding proxies sharing cache: I do know it can be done. This is not something I just invented for the beauty of providing additional entry to this thread  ;)  What I meant is that although it exists, you will not get it out of the box, AFAIK, with Zentyal. but I might be wrong as I'm not running 3.x
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: astana on October 20, 2013, 07:46:39 am
Simply put, I have no problem because my gateway handles the problem of starvation and sharing for me!
What I perceive as a problem for others, having already gone down that path and finding it totally lacking, is using squid to allocate bandwidth.
I'll repeat again, I don't use Zentyal as a gateway, so I have no knowledge if Zentyal performs the same way as pfSense of equally sharing the bandwidth across all clients (even if there is one client that is the zentyal proxy serves 100 users).
If Zentyal as a gateway doesn't do that, then I can propose a solution that does, but it does take another box.
Please note that on my gateway I'm not actively doing any layer 7 QoS, as all connections are queued correctly out of the box.

edits for correct grammar.
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: BrettonWoods on October 20, 2013, 08:09:17 pm
You make some really interesting points. I had never really thought about it as I am happy with the results I get. I have a habit of being generous in the bandwdith allocation.

This seems to work and it does balance traffic and I get the feeling and from memory pfsense doesn't do anything clever in fact it just doesn't do anything. Its just your router handing out packets evenly.

I get your argument though as why tie people down to specific limits if higher is available.

A lot of this is how and in what manner a sysadmin is going to control netwok access and bandwidth throttling. I guess its if needed, enable, if not then don't.

My personal opinion is that I would like more control and currently the zentyal proxy throttling could probably do with more control over the bucket system.
Most will just not enable the throttling and I am probably one of those. If I had a bit more control then I would probably enable it more often.

A lot of the Zentyal throttling is based on making essesntial services guaranteed.
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: astana on October 21, 2013, 03:10:38 am
You make some really interesting points. I had never really thought about it as I am happy with the results I get. I have a habit of being generous in the bandwdith allocation.Being happy is good, your users being happy is even better!

This seems to work and it does balance traffic and I get the feeling and from memory pfsense doesn't do anything clever in fact it just doesn't do anything. Its just your router handing out packets evenly. Saying it's doing nothing is disingenuous. It's actually doing a fantastic job transparently

I get your argument though as why tie people down to specific limits if higher is available.

A lot of this is how and in what manner a sysadmin is going to control netwok access and bandwidth throttling. I guess its if needed, enable, if not then don't. Absolutely, but nice to be aware what it will do and what it won't do

My personal opinion is that I would like more control and currently the zentyal proxy throttling could probably do with more control over the bucket system.
Most will just not enable the throttling and I am probably one of those. If I had a bit more control then I would probably enable it more often. The trouble is it's squids only method of control, so little more can be done with it.

A lot of the Zentyal throttling is based on making essesntial services guaranteed. This isn't good as it does confuse throttling with QoS, which as Christian rightly says are 2 very different beasts.
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: christian on October 21, 2013, 08:44:22 am
Astana,

I'm sorry but the more we discuss and the more I'm lost. Reading one post, I feel like "OK, I understand now what he means" and the next post makes me feel something different.

To summarize my current understanding of what you mean, as this discussion very confusing for me:
- you don't face any problem and you are very happy with pfSense as gateway and Zentyal as HTTP proxy
- you think Zentyal should propose something else than proxy only in order to provide QoS.

Am I correct ?

Assuming I am (although at this stage I'm totally lost), debate is somewhat truncated:
- Zentyal does provide QoS service (http://doc.zentyal.org/en/qos.html) that is not linked to proxy. Such service obviously works only when Zentyal is used as gateway because it works, as I explained previously, like other QoS implementations, at protocol level.
- I don't know any implementation of "QoS per user"
- I don't understand what you want to achieve (more) with Squid and I don't think we can have efficient discussion if we endlessly mix up everything
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: astana on October 21, 2013, 09:12:06 am
Astana,

I'm sorry but the more we discuss and the more I'm lost. Reading one post, I feel like "OK, I understand now what he means" and the next post makes me feel something different.

To summarize my current understanding of what you mean, as this discussion very confusing for me:
- you don't face any problem and you are very happy with pfSense as gateway and Zentyal as HTTP proxy Yes!
- you think Zentyal should propose something else than proxy only in order to provide QoS.No, I don't think Zentyal should provide any more, but I do think the squid method should be a bit clearer as I feel it's a trap for the unsuspecting

Am I correct ? Pretty much!

Assuming I am (although at this stage I'm totally lost), debate is somewhat truncated:
- Zentyal does provide QoS service (http://doc.zentyal.org/en/qos.html) that is not linked to proxy. Such service obviously works only when Zentyal is used as gateway because it works, as I explained previously, like other QoS implementations, at protocol level. This is why I explained I have never used Zentyal as a gateway, therefore I could not comment on that.
- I don't know any implementation of "QoS per user" Agreed, it does not exist!
- I don't understand what you want to achieve (more) with Squid and I don't think we can have efficient discussion if we endlessly mix up everything That is the whole point of this discussion. It is impossible to do more with squid (and following that with the Zentyal bandwidth tab in the proxy), which can lead to really poor results.

Answers in red
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: christian on October 21, 2013, 09:32:28 am
So what's your proposal and your point  ???
To remove HTTP proxy bandwidth throttling so that you (and other people) don't get confused ?
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: astana on October 21, 2013, 09:39:30 am
I was never making a proposal for changing the technology, I was giving the OP some information about the balancing that can be done outside of simple squid throttling.
And no, I wouldn't recommend removing the throttling as I'm sure there are legitimate reasons for implementing on a network, however if the reason is hogging/starvation then it's far too blunt a technology to use in a real world network.

I think if you re-read what I wrote carefully (I realise English may not be your first language), then you'll see the question is was asking is WHY and not that I was unhappy with any current options or technologies.
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: christian on October 21, 2013, 10:06:22 am
No matter which way you slice it, squid hasn't got the tools to manage bandwidth allocation in an intelligent way.

I ended up dropping squid management of bandwidth and using a better gateway to divide usage. Result: No more complaints about slow internet, usage almost doubled.

Indeed English is not my mother thong.
I'm reacting to above statement where you explain that Squid can't achieve something fitting your needs and ended up using better gateway, which I (perhaps poorly) understand as "doing it with another gateway [than Zentyal] worked better"

Therefore our confused (to me) and useless (for other) debate.

TTFN.

Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: astana on October 21, 2013, 10:12:06 am
I can see where the confusion came in, but in fact Zentyal has never been my gateway (as explained), and the gateway I had was dumb as a rock, and couldn't divide the bandwidth from my zentyal proxy. I thought I had explained that clearly, so apologies if that wasn't the case!
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: highjo on October 21, 2013, 06:55:17 pm
OK fellas, I have been reading all your inputs which has been very constructive to me but got a little lost with all the back and forth but was a little looking like a kind of challenge.

Now let's me give you a scenario for you to explain to me how best I can do my QoS. I have 2 connections:
connection 1: shared ,up to 4 M/s but very unstable
connection 2: dedicated , 1M/s a little more stable.

Connection 1 has weight 6 and connection has weight 1 on the gateways section.

I still have internet complains . Even though I applied the bandwidth throttling to 30k/s and set the Maximum unlimited size per client to 1000MB(1GB) to the "all_users" network group which is every IP from 192.168.0.5 to 192.168.0.254. I was surprised this morning to see that i could download via scp a db file of 2GB with speed doing to 200K/s .

It's obvious that I haven't done something right. I will also want to block torrents during office hours. With all that information how do you advise on a better QoS implementation on Zentyal.

I am totally unable to see from zentyal what is wrong with each connection when they are fooling.


Thanks very much
Title: Re: [SOLVED] bandwith management and fiters: what to use firewall or proxy
Post by: christian on October 21, 2013, 07:18:18 pm
Even though I applied the bandwidth throttling to 30k/s and set the Maximum unlimited size per client to 1000MB(1GB) to the "all_users" network group which is every IP from 192.168.0.5 to 192.168.0.254.

Can you confirm (or not) taht such setting has been applied at HTTP proxy level ?

Quote
I was surprised this morning to see that i could download via scp a db file of 2GB with speed doing to 200K/s .

Is this download done using HTTP proxy  ???  ;)

Quote
I will also want to block torrents during office hours.


Layer 7 filter should help you here.