Zentyal Forum, Linux Small Business Server

Zentyal Server => Installation and Upgrades => Topic started by: ccarpenter on August 24, 2012, 03:47:48 pm

Title: HTTP Proxy not authenticating
Post by: ccarpenter on August 24, 2012, 03:47:48 pm
I have setup my Zentyal to be an AD slave and I has successfully synced all users and groups but when I set the proxy to "Authorize and Filter" and create a filter policy that is only applied to my "Sales" group to deny a few sites it is not working at all. Is there something else I need to do to get it to authenticate?
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 27, 2012, 02:26:16 pm
Does anyone have any idea. I know my AD synchronization is working because I can change a users password and it'll sync it to zentyal. How do I get the users of my "sales" group to authenticate?
Title: Re: HTTP Proxy not authenticating
Post by: christian on August 27, 2012, 03:05:01 pm
You should perhaps:
- ensure that proxy is used
- FW rules do not permit direct access to internet
- ensure "transparent proxy" is not set
- ensure there is no profile authorizing free access (I mean w/o authentication)
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 27, 2012, 05:29:55 pm
I confirmed it is working for sure by setting my pc to go through the proxy. I setup a port forward rule to forward all 80 and 443 traffic to go to 3128. It seems though that the firewall changes don't ever take effect until after the firewall is rebooted. I added the rules and saved them and even manually restarted the firewall service and the rules haven't taken effect. I also had some other VOIP forwards that I had trouble getting working until I decided to reboot and then they were working. I can't reboot everytime I need to add a rule.
Title: Re: HTTP Proxy not authenticating
Post by: christian on August 27, 2012, 05:45:23 pm
Either I really don't understand what you are trying to achieve or my guess is correct and you are wrong with your approach. Let me explain:
Using proxy can be done either in transparent or explicit mode.

In transparent mode, browser (and user) is not aware of any proxy in the middle as this feature relies on network redirect. Because of this, authenticating is not an option.
In explicit mode, you have to configure your browser to use explicitly proxy or use auto discovery mode like WPAD (http://trac.zentyal.org/wiki/Documentation/Community/HowTo/SelectRightHTTPproxyDesign).

Back to transparent mode (although it doesn't match your requirement that is to apply profiling to some users): do not bother inventing rules like redirect at FW level. You can do it very easily by selecting "transparent" mode in HTTP proxy section.

Last but not least, do you mean that saving changes doesn't apply rules? I almost never reboot my server although I do change FW rules from time to time...  ???
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 27, 2012, 05:57:08 pm
Yes I have to have rules applied only to certain groups so I need authentication. and I will look into WPAD.
As far as the firewall settings I noticed it when I was setting up port forwarding for external ip phones to access our internal phone server. I setup all my ports and noticed it was not working. I even created a service and added all the ports and configured rules to allow them through and it still didn't work so I decided to restart the firewall late that night. Low and behold it started working right after a reboot. Now I was trying to force port 80 and 443 to go to 3128 through a port forward and it didn't work after a save, and I can't reboot in the middle of the day.
Title: Re: HTTP Proxy not authenticating
Post by: christian on August 27, 2012, 06:42:36 pm
Are you setting these port forwarding rules in order to offer HTTP proxy service to external users? If no, I don't why such rules are required. If yes, then your approach is, at least to me, wrong. You should either rely on VPN and use internal proxy or have your proxy listening on external interface.
In such case, notice that implementing WPAD for external users is a bit tricky  ;) and strange  ::)
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 27, 2012, 10:44:14 pm
OK I have followed the doc http://trac.zentyal.org/wiki/Documentation/Community/HowTo/SelectRightHTTPproxyDesign (http://trac.zentyal.org/wiki/Documentation/Community/HowTo/SelectRightHTTPproxyDesign)

and if I manually specify the pac file at http://wpad.mydomain.com/proxy.pac in my client browsers it works. I have also setup the DNS records for wpad, but I can't seem to get that to take effect I have restart my client (windows 7) a few times. It just wont seem to auto configure itself. Any ideas?
Title: Re: HTTP Proxy not authenticating
Post by: christian on August 27, 2012, 11:05:19 pm
You should rather try with wpad.dat instead of proxy.pac.
Then what happens if you type http://wpad (http://wpad) in your browser ?
What does your wpad.dat file contain ?
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 27, 2012, 11:14:55 pm
I had proxy.pac and wpad.dat in the root because I wasn't sure exactly which one the clients request. This may be my issue. I created a vitrual host on my zentyal install to go to wpad.mydomain.com. I put a test index file there to make sure it works and it resolves when I go to http://wpad.mydomain.com, but when I type http://wpad it resolves to the default first site on zentyal because the dns A record points to the ip of the zentyal server. So it would depend how clients request the autoconfiguration I guess.

The contents of the file are just this:
function FindProxyForURL(url, host)
{
   return "PROXY 10.20.20.1:3128";
}

Which is the IP of my zentyal server. Like I said before when I manually specify the wpad.dat or proxy.pac file on the client it will successfully block the sites that need blocked.
Title: Re: HTTP Proxy not authenticating
Post by: christian on August 27, 2012, 11:34:13 pm
Unfortunately, if you look at Microsoft technet here (http://technet.microsoft.com/en-us/library/cc713344.aspx), you will notice that Microsoft's implementation relies on http://wpad/wpad.dat  (http://wpad/wpad.dat) :-[
As a workaround, what I suggest is that you create one virtual IP attached to your internal interface and:
- create "wpad" virtual host listening on this IP
- ensure wpad.dat file is as the root of this web site
- set up DNS so that wpad A record points to this virtual IP
- then try http://wpad again  8)
Title: Re: HTTP Proxy not authenticating
Post by: christian on August 27, 2012, 11:43:40 pm
hint: are you sure your client fqdn matches host.yourdomain.something ?
I'm asking because DNS lookup is based on client's fqdn  ;)
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 28, 2012, 12:21:45 am
Yes my zentyal install and client and servers are all on the same domain. I created the virtual ip address and changed the DNS A record for wpad to be the new ip (10.20.20.2). Which I can ping. And I can ping wpad and wpad.mydomain.com and both resolve to 10.20.20.2. My problem is still my virtual host. I changed the virtualhost to be <VirtualHost 10.20.20.2:80> so it is listening on that address yet it still resolves to the default site of zentyal for http://wpad?
Title: Re: HTTP Proxy not authenticating
Post by: christian on August 28, 2012, 12:31:33 am
cache related issue ?
How is it that you resolve "wpad" as 10.20.20.2 but reach zentyal default server when accessing wpad?
Some typo somewhere or virtual host misconfiguration?
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 28, 2012, 12:37:00 am
I don't think there is a misconfiguration. I double checked all the setting I have described. How do I clear the cache? I didn't see a way to do it in the gui.
Title: Re: HTTP Proxy not authenticating
Post by: christian on August 28, 2012, 09:39:35 am
cache issue client side...  ;)
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 29, 2012, 02:52:43 pm
I am about ready to give up zentyal for another distribution because of all the problem with the proxy setup. I am testing a few computers manually configured to go though the proxy server. Some work exactly as they should while others have no internet access at all while they are configured to use the proxy server. I have configured it to use basic and ntlm authentication. On the computers that do have internet access ntlm is working because they are not prompted for a username or password and the sites that should be blocked are blocked. Yet when using firefox they are prompted for a username and password for basic authentication and the user credentials they use will not allow them to login. Has anyone experienced anything like this?
Title: Re: HTTP Proxy not authenticating
Post by: stuartiannaylor on August 29, 2012, 03:08:44 pm
Hi,

Yeah had the same with firefox. From memory you have to enable advanced option settings.
I agree that the proxy is a bit problematic but its more a history of M$ & ntlm.

https://sivel.net/2007/05/firefox-ntlm-sso/
Is just a quick google.

Don't give up as I use a couple of other distro's and you will find similar.
Ubuntu & Zentyal are as close to cutting edge as you will prob get.

Also 3.0 RC1 is out and to be honest that is the way to go.

NTLM isn't secure anymore as there are exploits. Kerberos is the way to go and is all singing and dancing in the new version.

The proxy seems much better in 3.0 I have been away from the community for a while but did notice group/user time restrictions all new goodies in the proxy.

http://www.zentyal.org/beta/ (http://www.zentyal.org/beta/)

I suggest running it on virtual box or other and have a look before you decide to go elsewhere.
If you need any help please post.

Stuart
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 29, 2012, 03:15:03 pm
Thanks for the quick reply, but what really gets me is why do some computers have no internet access at all while other work exactly as they should. I have rebooted the computers and ran ipconfig /flushdns to chear their dns cache from all the testing I've been doing, and even rebooted the firewall. I just don't get why some work and some don't.
Title: Re: HTTP Proxy not authenticating
Post by: christian on August 29, 2012, 03:17:45 pm
So far, it's a bit difficult, at least to me, to understand what you did and where.
e.g., at least using Zentyal GUI, I don't see where you can chose between basic and NTLM and enable it. Do you mean you tweaked squid conf  ???
I also don't understand whenever you tried to enable ntlm at Firefox level.
Last but not least, what do you mean with
Quote
the user credentials they use will not allow them to login
any error message ?

Did you look at some log?
Title: Re: HTTP Proxy not authenticating
Post by: stuartiannaylor on August 29, 2012, 03:23:49 pm
I am having a read through your posts and a bit of googling. Why some don't and some do is a hard one to explain. Almost as bad as intermittent faults that always fill me with dread when they arrive.

Is it a dns fault where you can ping the ip but not the FQDN.

Also are clients all the same? win version / os ?

I know clutching at straws  :-[

have a go at pinging some ip and have a look at the network settings maybe some where dns / default gateway ...

I would go back to basics and turn off the proxy.

Check all the clients.

Proxy on, no filters transparent.

Check all the clients.

Bring in your filters.

Check all the clients.

Will have a closer look at your post. If you find any oddities please post away.

If you need to chill that RC1 of 3.0 is ready for a virtual machine test :)
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 29, 2012, 03:29:04 pm
I found this to setup NTLM in the squid conf.
http://trac.zentyal.org/wiki/Documentation/Community/HowTo/ProxyWithNTLM

I configured IE in the connections tab to manually point it to the proxy server and left firefox to use system setting which should use what IE uses. At the loign box it gives for firefox I put in the user credentials they use to login to the computer with. All that happens is the login prompt goes away and comes right back up like it won't accept the credentials. It gives no error message.

Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 29, 2012, 03:34:45 pm
@stuartiannaylor

I can ping everything by ip and by FQDN so I know my dns shouldn't be an issue. All clients are using DHCP from the firewall. I have mixed windows xp/7/server 2003. On the server with proxy enabled I have no internet access at all. On two windows 7 pcs the proxy works, but on another windows 7 pc it also doesn't have internet access. Same for XP. One works as it should while the other has not internet access. Mind you that when I remove the proxy setting on all computers they all have internet access and all can resolve ip and FQDN.
Title: Re: HTTP Proxy not authenticating
Post by: stuartiannaylor on August 29, 2012, 03:34:58 pm
I would knock out the wpad stuff aswell. wpad and pac is just for auto proxy settings.

Do it the old fashioned way on a couple of clients and enter the proxy details.

Do I understand you are trying to authenticate to an internal hosted site on zentyal through apache?

As add the apache logs for that site onto the thread.

Also do you have dns module enabled?

I noticed one time with virtual hosts that I had to manually place the IP in the automatically created dns entry.
Damn my memory but when blank I think it was the assumption local host and I found that didn't work.

Also if you are like me its not anything to do with the virtual host where actually say www. is throwing you back to the default domain?
Title: Re: HTTP Proxy not authenticating
Post by: christian on August 29, 2012, 03:40:36 pm
Clearer for what concerns NTLM proxy side. You should have started with this info  ;) for remote people not aware of what you did. It might help for debugging purpose  8)
Regarding authentication "error" (at least in term of expected behaviour, I would suggest that you look at logs (squid and syslog) and search for any entry matching account.

Just curious: is this behaviour linked to specific account or to specific workstation  ???
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 29, 2012, 03:41:03 pm
@stuartiannaylor

I have removed all webserver functionality and wpad configurations to eliminate any issues I may be having with that. Sorry I forgot to post that. I was trying to eliminate anything that could be affecting the proxy.
Title: Re: HTTP Proxy not authenticating
Post by: stuartiannaylor on August 29, 2012, 03:43:46 pm
It would sound like the proxy isn't working maybe and some of the clients are not set right and not using the proxy.

When the proxy is off what route is the IP traffic taking to get access?

You haven't got a problem like two conflicting dhcp servers have ?

To be honest flummoxed, someone will help.

Keep posting and someone will help.

If you get the right details to christian he will have you sorted in a milli second.

I get the feeling there is more going on than remotely I am aware of.

Sorry for sounding condescending but go back to basics and build things up slowly.

Then at the point of loss forward us details

Apols Stuart
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 29, 2012, 03:47:25 pm
Regarding authentication "error" (at least in term of expected behaviour, I would suggest that you look at logs (squid and syslog) and search for any entry matching account.

Just curious: is this behaviour linked to specific account or to specific workstation  ???

Can you tell me the exact location of the squid logs? No it is effecting any account. For instance on my mac it is not part of the domain so it prompts me for basic authentication and i can use any number of domain login accounts that were synced via ldap and it just keeps re-prompting a login.
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 29, 2012, 03:50:02 pm
When the proxy is off what route is the IP traffic taking to get access?

What do you mean by this? The default gateway for the DHCP pool is to use the zentyal firewall.
Title: Re: HTTP Proxy not authenticating
Post by: stuartiannaylor on August 29, 2012, 03:50:12 pm
:) /var/logs all in there.

Always good to throw in the zentyal log and module specific log.

I usually use winscp as a freebie goodie to do it remotely.

What do you mean by this? The default gateway for the DHCP pool is to use the zentyal firewall. (http://What do you mean by this? The default gateway for the DHCP pool is to use the zentyal firewall.)

Sorry I got confused but I thought you said you couldn't get any access from the proxy machine?
In fact a little confused to be honest :)

Stay with it :) post some logs and check the basics and build without the proxy and all.

Might be a while as I have an invite for pub lunch and a beer.
Title: Re: HTTP Proxy not authenticating
Post by: christian on August 29, 2012, 04:03:29 pm
so it looks like your "basic" authentication settings do not work... are you 100% sure tweak you apply is correct? (I didn't look at the detail of what this "how to" suggests yet)
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 29, 2012, 04:03:53 pm
:) /var/logs all in there.

Always good to throw in the zentyal log and module specific log.

I usually use winscp as a freebie goodie to do it remotely.

What do you mean by this? The default gateway for the DHCP pool is to use the zentyal firewall. (http://What do you mean by this? The default gateway for the DHCP pool is to use the zentyal firewall.)

Sorry I got confused but I thought you said you couldn't get any access from the proxy machine?
In fact a little confused to be honest :)

Stay with it :) post some logs and check the basics and build without the proxy and all.

Might be a while as I have an invite for pub lunch and a beer.

I don't have internet access on some clients if they are setup to go through the proxy on port 3128, but on those same computers with the proxy setting removed I do have internet access through zentyal as the gateway as long as it's not being sent through squid.

Also I don't have permission to view /var/log/squid? Shouldn't I?
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 29, 2012, 04:07:48 pm
so it looks like your "basic" authentication settings do not work... are you 100% sure tweak you apply is correct? (I didn't look at the detail of what this "how to" suggests yet)

It seems that way. I removed the NTLM auth setting in squid and dansguardian just now and I am being prompted for login in IE and its the same as it was with firefox. Just keeps re-prompting for login even with credentials I know are correct.
Title: Re: HTTP Proxy not authenticating
Post by: christian on August 29, 2012, 04:15:39 pm
- access to squid log (although at this stage I don't think this is the highest priority): you need to sudo.
- basic authentication not working:  ??? ??? are these credential working when you, e.g., access user corner ?
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 29, 2012, 04:23:19 pm
- access to squid log (although at this stage I don't think this is the highest priority): you need to sudo.
- basic authentication not working:  ??? ??? are these credential working when you, e.g., access user corner ?

In the admin gui for user corner it says "User Corner is not supported on slave servers."
Title: Re: HTTP Proxy not authenticating
Post by: christian on August 29, 2012, 04:33:49 pm
hoops, I missed this point described in your  very first post  :-[
Would you mind posting your squid.conf file here?
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 29, 2012, 04:36:35 pm
hoops, I missed this point described in your  very first post  :-[
Would you mind posting your squid.cong file here?

/etc/squid/squid.conf
or
/usr/share/zentyal/stubs/squid/suid.conf.mas
Title: Re: HTTP Proxy not authenticating
Post by: christian on August 29, 2012, 04:38:09 pm
the one active, meaning /etc/squid/squid.conf
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 29, 2012, 05:02:04 pm
Here is the /etc/squid/squid.conf
I changed my domain and server name in the auth_param setting.


# <EBOX> TAG_HTTPORT #
http_port 3128
# END_TAG #

visible_hostname localhost
dns_nameservers 8.8.8.8 8.8.4.4
hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
no_cache deny QUERY

# refresh patterns

# windows updates
refresh_pattern http://.*\.windowsupdate\.microsoft\.com/ 0 80% 20160 reload-into-ims
refresh_pattern http://.*\.update\.microsoft\.com/ 0 80% 20160 reload-into-ims
refresh_pattern http://download\.microsoft\.com/ 0 80% 20160 reload-into-ims
refresh_pattern http://windowsupdate\.microsoft\.com/ 0 80% 20160 reload-into-ims
refresh_pattern http://.*\.download\.windowsupdate\.com/ 0 80% 20160 reload-into-ims
refresh_pattern http://office\.microsoft\.com/ 0 80% 20160 reload-into-ims
refresh_pattern http://w?xpsp[0-9]\.microsoft\.com/ 0 80% 20160 reload-into-ims
refresh_pattern http://w2ksp[0-9]\.microsoft\.com/ 0 80% 20160 reload-into-ims
# linux updates
refresh_pattern http://.*\.archive\.ubuntu\.com/ 0 80% 20160 reload-into-ims
refresh_pattern http://(ftp|http)[0-9]*\.[a-z]+\.debian\.org/ 0 80% 20160 reload-into-ims

refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern .               0       20%     4320
# end refresh patterns

coredump_dir /var/spool/squid
cache_effective_user proxy
cache_effective_group proxy
cache_mem 128 MB
maximum_object_size 300 MB
access_log /var/log/squid/access.log squid
pid_filename /var/run/squid.pid

cache_dir ufs /var/spool/squid 500 16 256



# <EBOX> TAG_ACL #
#auth_param basic realm Zentyal HTTP proxy
#auth_param basic program /usr/lib/squid/ldap_auth -v 3 -b  ou=Users,dc=rvw,dc=com  -u uid  -h ldap://127.0.0.1:389
auth_param ntlm program /usr/lib/squid/ntlm_auth -b mydomain.local/myserver
auth_param ntlm children 25

acl authorized  proxy_auth required




acl Sales proxy_auth my users synced from ldap
acl Information__Systems proxy_auth my users synced from ldap
     

     









# no cache domains acl

# END_TAG #
acl localhost src 127.0.0.0/8
acl localhostdst dst 127.0.0.0/8
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl to_localhost dst 127.0.0.0/8
acl SSL_ports port 443 563   # https, snews
acl SSL_ports port 873      # rsync
acl Safe_ports port 80      # http
acl Safe_ports port 21      # ftp
acl Safe_ports port 443 563   # https, snews
acl Safe_ports port 70      # gopher
acl Safe_ports port 210      # wais
acl Safe_ports port 1025-65535   # unregistered ports
acl Safe_ports port 280      # http-mgmt
acl Safe_ports port 488      # gss-http
acl Safe_ports port 591      # filemaker
acl Safe_ports port 777      # multiling http
acl Safe_ports port 631      # cups
acl Safe_ports port 873      # rsync
acl Safe_ports port 901      # SWAT
acl purge method PURGE
acl CONNECT method CONNECT

follow_x_forwarded_for allow localhost
log_uses_indirect_client off

http_access allow localhost

http_access deny manager
http_access deny purge
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_reply_access allow all



# <EBOX> TAG_DELAYPOOLS #


acl eboxlocalnets dst 10.20.20.0/22




# END_TAG

# <EBOX> TAG_HTTP_ACCESS #








http_access allow authorized all
http_access allow   Information__Systems

http_access allow   Sales





# default policy section


http_access allow authorized all
http_access allow all


# END_TAG #

# <EBOX> TAG SNMP #

# END_TAG #

always_direct allow localhostdst
Title: Re: HTTP Proxy not authenticating
Post by: christian on August 29, 2012, 05:06:00 pm
 ??? either you made a mistake or there is something really weird because this file looks like squid.conf.mas, not like squid.conf
Please check twice and confirm.
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 29, 2012, 05:19:22 pm
Sorry copied the wrong text file. Edited my last post.
Title: Re: HTTP Proxy not authenticating
Post by: christian on August 29, 2012, 05:27:57 pm
Cool.
Looking at your file, you are using NTLM authentication... so I'm lost now.
I was under the impression that you were facing an error with basic authentication and in order to investigate this, you reverted back to "basic authentication", still it was not working. However, when I look at this file, there is no basic but NTLM.
Could you please clarify this?
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 29, 2012, 05:31:59 pm
Could you please clarify this?

Yes I've switched it back and forth a couple times this morning. It just happen to be with NTLM when I copied it just now.
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 29, 2012, 05:48:15 pm
Also 3.0 RC1 is out and to be honest that is the way to go.

NTLM isn't secure anymore as there are exploits. Kerberos is the way to go and is all singing and dancing in the new version.

Installed 3.0 in a vm and I noticed the "users and groups" found my domain automatically. Do I still need to use the zentyal windows program on my server to synchronize users?
Title: Re: HTTP Proxy not authenticating
Post by: ccarpenter on August 29, 2012, 06:04:14 pm
Going through my syslog it is showing this:

Aug 29 12:00:01 firewall slapd[10359]: connection_read(19): no connection!
Aug 29 12:00:01 firewall slapd[10359]: connection_read(19): no connection!
Aug 29 12:00:01 firewall slapd[10359]: connection_read(13): no connection!
Aug 29 12:00:02 firewall slapd[10359]: last message repeated 11 times
Aug 29 12:00:02 firewall slapd[10359]: connection_read(19): no connection!
Aug 29 12:00:03 firewall slapd[10359]: last message repeated 3 times
Aug 29 12:00:03 firewall slapd[10359]: connection_read(13): no connection!
Aug 29 12:00:03 firewall slapd[10359]: connection_read(13): no connection!

looks like ldap is not working correctly?