Need some help understanding port forwading

[half_life]

I have the following scenario playing out at work. 

I have a remote device that needs to connect to a service on an internal machine.  I have setup port forwarding for this without checking replace source address.

The service receives the packet and tries to reply on another port back to the device.  Instead of sending the reply to the device it is sending it to the Zentyal machine (destination IP== my zentyal). 

The vendor is informing me that the packet arriving to them is from my zentyal machine (via wireshark) and not the device so their software is working correctly.  The rub is this is the second such setup I have done (same vendor different software) and the first one works correctly.

I will admit my understanding of iptables is not perfect but there does not seem to be a rewrite rule involved.  Is this normal behavior?  My belief is that the device ip should still be in the packet header.  Is this so?

Thanks

[christian]

Is there any reason why you replace source address?

[half_life]

Christian,  I did not enable source port replacement.

ETA:  I mean source address replacement ;-)

[christian]

Oops, sorry, I read it too fast.
So what you describe looks like source address is replaced even if you don't enable "source address replacement". Is that correct ?

[half_life]

It is what I am being told.  I haven't done any testing myself.

[Sam Graf]

  I'm probably just showing my ignorance here and not being helpful, but is Squid in the equation? I'm wondering about the behavior of X-forwarded-for.

[half_life]

Squid only comes into play on web traffic.  Closer inspection of the iptables rules shows me that the packets on the inbound side are DNATed which means only the destination address is re-written.  Outbound related packets are SNATed to rewrite the source address.  As near as I can tell the source address is never touched on inbound packets that are re-directed.

[Sam Graf]

Squid only comes into play on web traffic.

Understood. I wasn't sure what kind of traffic I was reading about.

[half_life]

An update to all of this:

I may have found a bug in the way Zentyal handles SNAT.  If you check rewrite source address for a particular forward rule, it sets up a generic rewrite rule for all traffic coming out to the target address not just on the individual port.  IE

Internet--->Zentyal  (DNAT ++SNAT)----->  Internal service (correct setup for Tomcat on 8081 for instance)
Internet-->Zentyal (DNAT++SNAT) -------> Internal service  (not correct for a telemetry device with bi-directional comms)

I checked rewrite source address for the first example but not for the second one.  They both are affected by the same SNAT rule.

Shouldn't this be done by port number and not just IP address?


ETA: This is a 2.2 server that is up to date.

[half_life]

This is the rule that Zentyal adds when I select replace source address (note this is a 3.0 system that is up to date but the rule looks the same in a 2.2 system)

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
postmodules  all  --  anywhere             anywhere           
SNAT       tcp  --  anywhere             192.168.0.174        tcp dpt:http ctstate DNAT to:192.168.0.3
SNAT       udp  --  anywhere             192.168.0.174        udp dpt:http ctstate DNAT to:192.168.0.3
MASQUERADE  all  --  anywhere             anywhere

Notice that it doesn't specify the port as in 192.168.0.3:82 which it should be in this case.  This affects incoming packets to the target server so in effect all traffic appears to be coming from zentyal no matter which port you are speaking of totally nullifying the reason to use DNAT.  Am I just not understanding something or is this not a bug?

[christian]

your explanation makes sense 
did you already create a ticket so that Zentyal team investigates this?

[half_life]

I now believe the mistake was on my part.

SNAT       udp  --  anywhere             192.168.0.174        udp dpt:http ctstate DNAT to:192.168.0.3

In the above snippet  I believe  udp dpt:http ctstate DNAT  is giving the parameters that I said was missing.  I now believe the root cause was an overly broad redirect with SNAT that encompassed my gateways internal IP address.  I am testing now and will report back with what I learn.

[half_life]

I am now sure that I made a mistake when setting up port forwarding 1.5 years ago.  I am the IT Manager for a local transit company.  We track the buses in real time so as to be able to inform the public via multiple means just exactly where their bus is and when it will be arriving.  Each vehicle has a mobile data terminal as well as GPS equipment.  Zentyal is used as the gateway for our facility.

 The mobile data terminals communicate via one port (bi directional).  I setup port forwarding to connect the MDTs to the middleware on the database server.

  I later added a Tomcat install on the same server (vendor doesn't support running their webportal on anything besides Windows)  to display where the bus was and ETA on a google map.  I linked to this webportal on our main website and setup port forwarding to the tomcat install.

When I tested it all everything worked except that people inside the facility couldn't just click the bus locator link on our website and get to the vendors webportal.  Easy enough to understand,  the initial request would be sent to the Zentyal server so the client on the local network would be expecting a response from the Zentyal server, not the database server.

Easy enough to fix, or so I thought.  I adjusted the port forward on my internal nic that forwarded port 8081 on to my database server to rewrite the source address.  By default it will setup as from "anywhere" and I left it that way.

Everything worked so I was happy and over the last 18 months the details of my decisions got sort of blurry.  The MDT's were working and the webportal was working for clients on the inside and the outside from the same link.

I added a new type of display for the vehicles (a consumer grade tablet) that sent on one port and received on another.  I could not get it to work via cellular.  I could get it to work connected to the facility wireless.

I quickly grew irritated and shipped the whole lot of them off to the vendor to troubleshoot.  They identified eventually that we were not passing the real IP address into the server but instead passing in the IP of the gateway as the source address. For 18 months the firewall had been misconfigured and SNATING any traffic coming out of the Zentyal server destined for the database server.  It didn't bother the older MDT's because they did all communication on the same port.

Long story short,  when dealing with the "rewrite source address" option in portforwarding,  one needs to really stop and think hard before one accepts the "from anywhere" default. 

I created a network object that covered the IP address range allocated to my dhcp server and selected that instead of "anywhere".

DNAT is destination rewriting and happens during prerouting (affects inbound traffic)
SNAT is source rewriting and happens during postrouting (affects outbound traffic)

My error was to not consider that "outbound traffic" means from my server to both internal IPs and external IPs and from anywhere included my own gateway server.

I will spend the weekend retyping all of the firewall rules using edlin as penance.

[christian]

I will spend the weekend retyping all of the firewall rules using edlin as penance.

 
At least you figured out by yourself what the error was which is already quite a good achievement isn't it?

Now thtat I understand better where your problem was: I though you were also using reverse proxy  Can't it be used as relay to Tomcat ?

[half_life]

You remember correctly that I had setup nginx.  I also had setup Cherokee.  I manage two Zentyal gateways,  one for my home and one for work.  The home install is where I was working with reverse proxy.  Yes,  either one of the products I mentioned will handle Tomcat (Cherokee with more manageability). 

As to solving the problem, they say that necessity is the mother of invention.  What they fail to mention is that it is the source of a lot of adult verbiage in the process. 

I probably will install Cherokee at work to manage my ever growing fleet of web services.  I have developed some in house software that runs on Tomcat and added other services internally (Jasper Reports etc) to meet some of our business needs.  There is a request in to build an intranet (as if I have lots of spare time) over the coming months.  Currently I am installing this stuff to virtual machines to give me some isolation.  Besides it is a breeze to do so since my hypervisor of choice supports fully virtualized and container type templates.