Two eBox servers, can't route/ping through the first to the second problem

[RoboJ1M]

Hi,

I'm looking at using Ubuntu Server 8.04 + eBox to run our new network.
We're planning on having two routers.

One "internal" router as a content filter and firewall between internal users and the internet/dmz/vpn and one "external" router connected to the internet/dmz/vpn/other networks but does not have the high content filtering/caching load/firewall ruleset of the internal router.

I've got a very simple test setup at the moment, and I can't get it to work.
Here's the setup:

Networks:

192.168.28.0/30 : network to connect the two routers together
192.168.27.0/24 : Our current internal network, all three PCs attached to this network, although router-external is only connected for debug purposes

Hosts:

router-external:

eth1:192.168.28.1
eth4:192.168.27.12 (temp connection for access to web interface)

router-internal:

eth0:192.168.28.2
eth1:192.168.27.188 (temp address, this will become the default gateway for 192.168.27.0/24)

newdev: (My PC)

192.168.27.14

Routes:

Code: [Select]

administrator@router-internal:~$ ip route show
192.168.28.0/30 dev eth0  proto kernel  scope link  src 192.168.28.2
192.168.27.0/24 dev eth1  proto kernel  scope link  src 192.168.27.188

Code: [Select]

administrator@router-external:~$ ip route show
192.168.28.0/30 dev eth1  proto kernel  scope link  src 192.168.28.1
192.168.27.0/24 dev eth4  proto kernel  scope link  src 192.168.27.12

Code: [Select]

C:\>route print
===========================================================================
Interface List
0x1 ........................... MS TCP Loopback interface
0x2 ...00 1e 4f 92 f3 f9 ...... Intel(R) 82566DM-2 Gigabit Network Connection -
SecuRemote Miniport
0x10004 ...00 0a 3a 63 70 b0 ...... Bluetooth Device (Personal Area Network)
===========================================================================
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
        127.0.0.0        255.0.0.0        127.0.0.1       127.0.0.1       1
     192.168.27.0    255.255.255.0    192.168.27.14   192.168.27.14       10
    192.168.27.14  255.255.255.255        127.0.0.1       127.0.0.1       10
   192.168.27.255  255.255.255.255    192.168.27.14   192.168.27.14       10
     192.168.28.0  255.255.255.252   192.168.27.188   192.168.27.14       1
        224.0.0.0        240.0.0.0    192.168.27.14   192.168.27.14       10
  255.255.255.255  255.255.255.255    192.168.27.14   192.168.27.14       1
  255.255.255.255  255.255.255.255    192.168.27.14           10004       1
===========================================================================
Persistent Routes:
  None
Each router can ping each others 192.168.28.x address.
newdev can ping all the addresses on router-internal
newdev cannot ping 192.168.28.1 (router-external through router-internal)
I cannot trace the route to 192.168.28.1 from newdev.
There are no firewall logs for ICMP being dropped.

I have ebox-firewal and ebox-software installed.
All the modules are turned on.
All the logs are turned on and configured to log everything.
I've added Any Internal ICMP to both eBox's Services secions
I've added Allow Internal ICMP from Any to Any in the Internal Networks secions of router-internal's Packet Filter section
I've added Allow Internal ICMP from Any to the the Internal Networks to Ebox secion of router-external's Packet Filter secion

Questions:

1) Why can I not ping 192.168.28.1 from my PC newdev?
2) Why can I not ssh to 192.168.28.1 from my PC newdev? (i can ssh to both eBoxes on their 192.168.27.x addresses)

Thanks!

Jim.

[javi]

Have you set the interface with 192.168.28.2 address as external?

[RoboJ1M]

Hi,

No, all eBox interfaces are set to Internal, as 192.168.27.0/23 and 192.168.28.0/30 are internal networks.
eth0 on router-external will eventually be connected to the internet and labled External.
All other interfaces on router-external are internal networks.

Thanks,

Jim.

[javi]

Note that in your final configuration you should instruct your external router how to reach the 192.168.27.0 network. You should add a route to reach that network via 192.168.28.2. Otherwise, the outgoing pings will reach the external firewall but they won't know how to reach back that network.

If you just set the 192.168.28.2 interface as external, packages coming out from that interface from 192.168.27.0. network will be masquered to 192.168.28.2, and the external router will know how to reply.

I've just tested your configuration with my virtual machines and I can ping the external router after adding the proper route.

[RoboJ1M]

OK,

I've added the new route but still no joy.
However, there is already a default route to 192.168.27.0/24 via eth4:192.168.27.12
So I've tried to reconfigure eth4 to talk to my pc on a vlan: 192.168.1.0/24
But: ()

Earlier I took the IP address off of eth4 altogether and shut down the interface (I wanted to test ping without eth4 connected)
I lost my web access to the server (obviously) and the only way to get it back was manually edit /etc/network/interfaces.
But now eBox will not re-write /etc/network/interfaces!
Have I lost something by manually editing the file?

Thanks,

Jim.

[RoboJ1M]

UPDATE:

I have successfully reconfigured eth4 to use the vlan 192.168.1.0/24 to talk to my PC newdev.
I have also added the route to 192.168.27.0/24 via 192.168.28.2

Here is the gumf:

ROUTER-INTERNAL:

Code: [Select]

administrator@router-internal:~$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 10
    link/ether 00:1b:21:18:8f:de brd ff:ff:ff:ff:ff:ff
    inet 192.168.28.2/30 brd 192.168.28.3 scope global eth0
    inet6 fe80::21b:21ff:fe18:8fde/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 10
    link/ether 00:0f:1f:70:92:d7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.27.188/24 brd 192.168.27.255 scope global eth1
    inet6 fe80::20f:1fff:fe70:92d7/64 scope link
       valid_lft forever preferred_lft forever

Code: [Select]

administrator@router-internal:~$ ip route show
192.168.28.0/30 dev eth0  proto kernel  scope link  src 192.168.28.2
192.168.27.0/24 dev eth1  proto kernel  scope link  src 192.168.27.188
ROUTER-EXTERNAL:

Code: [Select]

administrator@router-external:~$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:1b:21:18:97:3e brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 10
    link/ether 00:1b:21:18:97:3d brd ff:ff:ff:ff:ff:ff
    inet 192.168.28.1/30 brd 192.168.28.3 scope global eth1
    inet6 fe80::21b:21ff:fe18:973d/64 scope link
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:1b:21:18:99:cd brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000
    link/ether 00:1b:21:18:90:cc brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 10
    link/ether 00:0f:1f:74:fd:ee brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global eth4
    inet6 fe80::20f:1fff:fe74:fdee/64 scope link
       valid_lft forever preferred_lft forever

Code: [Select]

administrator@router-external:~$ ip route show
192.168.28.0/30 dev eth1  proto kernel  scope link  src 192.168.28.1
192.168.1.0/24 dev eth4  proto kernel  scope link  src 192.168.1.1
192.168.27.0/24 via 192.168.28.2 dev eth1
NEWDEV:

Code: [Select]

C:\>ipconfig

Windows IP Configuration


Ethernet adapter Spur LAN:

        Connection-specific DNS Suffix  . :
        IP Address. . . . . . . . . . . . : 192.168.1.2
        Subnet Mask . . . . . . . . . . . : 255.255.255.0
        IP Address. . . . . . . . . . . . : 192.168.27.14
        Subnet Mask . . . . . . . . . . . : 255.255.255.0
        Default Gateway . . . . . . . . . : 192.168.27.1

Code: [Select]

C:\>route print
===========================================================================
Interface List
0x1 ........................... MS TCP Loopback interface
0x2 ...00 1e 4f 92 f3 f9 ...... Intel(R) 82566DM-2 Gigabit Network Connection -
SecuRemote Miniport
0x10004 ...00 0a 3a 63 70 b0 ...... Bluetooth Device (Personal Area Network)
===========================================================================
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0     192.168.27.1   192.168.27.14       10
        127.0.0.0        255.0.0.0        127.0.0.1       127.0.0.1       1
      192.168.1.0    255.255.255.0      192.168.1.2   192.168.27.14       10
      192.168.1.2  255.255.255.255        127.0.0.1       127.0.0.1       10
    192.168.1.255  255.255.255.255      192.168.1.2   192.168.27.14       10
     192.168.27.0    255.255.255.0    192.168.27.14   192.168.27.14       10
    192.168.27.14  255.255.255.255        127.0.0.1       127.0.0.1       10
   192.168.27.255  255.255.255.255    192.168.27.14   192.168.27.14       10
     192.168.28.0  255.255.255.252   192.168.27.188   192.168.27.14       1
        224.0.0.0        240.0.0.0    192.168.27.14   192.168.27.14       10
  255.255.255.255  255.255.255.255    192.168.27.14   192.168.27.14       1
  255.255.255.255  255.255.255.255    192.168.27.14           10004       1
Default Gateway:      192.168.27.1
===========================================================================
Persistent Routes:
  None
What's changed:

I can web and ssh to 192.168.1.1.
The route was created correctly on router-external
ping from newdev now returns:

Code: [Select]

C:\>ping 192.168.28.1

Pinging 192.168.28.1 with 32 bytes of data:

Reply from 192.168.27.188: Destination host unreachable.
Reply from 192.168.27.188: Destination host unreachable.
Reply from 192.168.27.188: Destination host unreachable.
Reply from 192.168.27.188: Destination host unreachable.

Ping statistics for 192.168.28.1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
It was returning "Request Timed Out" before.

ping newdev from router-external returns:

Code: [Select]

administrator@router-external:~$ ping 192.168.27.14
PING 192.168.27.14 (192.168.27.14) 56(84) bytes of data.

From 192.168.28.1 icmp_seq=10 Destination Host Unreachable
From 192.168.28.1 icmp_seq=11 Destination Host Unreachable
From 192.168.28.1 icmp_seq=12 Destination Host Unreachable
From 192.168.28.1 icmp_seq=14 Destination Host Unreachable
From 192.168.28.1 icmp_seq=15 Destination Host Unreachable
From 192.168.28.1 icmp_seq=16 Destination Host Unreachable
From 192.168.28.1 icmp_seq=18 Destination Host Unreachable
From 192.168.28.1 icmp_seq=19 Destination Host Unreachable
From 192.168.28.1 icmp_seq=20 Destination Host Unreachable

--- 192.168.27.14 ping statistics ---
21 packets transmitted, 0 received, +9 errors, 100% packet loss, time 20004ms
, pipe 3
Again, don't think that's what I got before. Interesting how is says
"From 192.168.28.1 icmp_seq=20 Destination Host Unreachable"

If the router is 192.168.28.2, why is 28.1 returning Destination Host Unreachable?

Thanks,

Jim.

[javi]

Can you post the output of the following command on the internal router:

Code: [Select]

cat /proc/sys/net/ipv4/ip_forward

If the router is 192.168.28.2, why is 28.1 returning Destination Host Unreachable?

Icmp works as follows: 28.1 is telling you it can reach the address you are trying to ping

[RoboJ1M]

Hi,

As requested:

Code: [Select]

administrator@router-internal:~$ cat /proc/sys/net/ipv4/ip_forward
1
Thanks,

Jim.

[javi]

Everything seems ok:

what's the output of the last command on your external router? It should be 1 too.


You were right earlier, 28.2 shoud be the router complaining that it can't reach newdev, not 28.1. That means the packet is not even trying to leave the network interface.

Could you install tcpdump on both routers and run:

Code: [Select]

tcpdump -i iface -p icmp
to see packets actually coming out/in

[RoboJ1M]

SOVLED! (I think)

Thankyou very much for your help, here is the final solution:

Missing route in router-external: 192.168.27.0/24 via 192.168.28.2
Having an interface directly connected to 192.168.27.0/24 on router-external (eth4) for debugging was messing up that route, needed to change it to a vlan.
And the final reason why it was not working:

Dodgy Crossover Cable Between The Routers

 

Got this this morning, wiggled the cable:

Code: [Select]

administrator@router-external:~$ ping 192.168.28.2
PING 192.168.28.2 (192.168.28.2) 56(84) bytes of data.
From 192.168.28.1 icmp_seq=1 Destination Host Unreachable
From 192.168.28.1 icmp_seq=2 Destination Host Unreachable
From 192.168.28.1 icmp_seq=3 Destination Host Unreachable
From 192.168.28.1 icmp_seq=5 Destination Host Unreachable
From 192.168.28.1 icmp_seq=6 Destination Host Unreachable
From 192.168.28.1 icmp_seq=7 Destination Host Unreachable
From 192.168.28.1 icmp_seq=9 Destination Host Unreachable
From 192.168.28.1 icmp_seq=10 Destination Host Unreachable
From 192.168.28.1 icmp_seq=11 Destination Host Unreachable
64 bytes from 192.168.28.2: icmp_seq=12 ttl=64 time=11.0 ms
64 bytes from 192.168.28.2: icmp_seq=13 ttl=64 time=0.734 ms
64 bytes from 192.168.28.2: icmp_seq=14 ttl=64 time=0.658 ms
64 bytes from 192.168.28.2: icmp_seq=15 ttl=64 time=0.585 ms
64 bytes from 192.168.28.2: icmp_seq=16 ttl=64 time=0.763 ms
64 bytes from 192.168.28.2: icmp_seq=17 ttl=64 time=0.686 ms
64 bytes from 192.168.28.2: icmp_seq=18 ttl=64 time=0.612 ms
64 bytes from 192.168.28.2: icmp_seq=19 ttl=64 time=0.539 ms
64 bytes from 192.168.28.2: icmp_seq=20 ttl=64 time=0.716 ms
64 bytes from 192.168.28.2: icmp_seq=21 ttl=64 time=0.641 ms
64 bytes from 192.168.28.2: icmp_seq=22 ttl=64 time=0.566 ms
64 bytes from 192.168.28.2: icmp_seq=23 ttl=64 time=0.493 ms
64 bytes from 192.168.28.2: icmp_seq=24 ttl=64 time=0.670 ms
64 bytes from 192.168.28.2: icmp_seq=25 ttl=64 time=0.596 ms
64 bytes from 192.168.28.2: icmp_seq=26 ttl=64 time=0.522 ms
64 bytes from 192.168.28.2: icmp_seq=27 ttl=64 time=0.447 ms
64 bytes from 192.168.28.2: icmp_seq=28 ttl=64 time=0.623 ms
64 bytes from 192.168.28.2: icmp_seq=29 ttl=64 time=0.550 ms
I'm going to go and set fire to it now.

I now have 72 hours to finish the new network.

eBox don't fail me now, I really don't want to have to go back to Bering-uClibc! (It's just too unmanageable for the size of our network now)

Thanks!! 

Jim.

[RoboJ1M]

Hi,

Not quite solved

I still have the problem that router-external has stopped allowing eBox to re-write the /etc/network/interfaces file
I had to manually edit the file myself and now eBox has lost the ability to edit it itself.

Thanks,

Jim.

[RoboJ1M]

Hi,

Update:

From /val/log/ebox/ebox.log:

Code: [Select]

2008/05/30 13:10:19 INFO> Module.pm:154 EBox::Module::save - Restarting service for module: network
2008/05/30 13:10:20 INFO> Network.pm:1700 EBox::Network::generateInterfaces - Skipping modification of /etc/network/interfaces
I've tried:

Removing /etc/network/interfaces --> Error
Changing the group to the same as router-internal (227 to 254) --> No difference
I've changed the group back.

The permissions are the same on both files.

Regards,

Jim

[javi]

That's how eBox is supposed to work. If you modify a file manually, eBox tries to preserve your changes until you allow it to overwrite it.

Enable/disable the network module and save changes. Now eBox will warn you about the file modification and ask you permission to go ahead with the modification, accept it and you are done.