Cfrtjryk

I am tasked to set up two test servers running CentOS 7.1 (Minimal install; the NetworkManager is disabled on both), and have run into a set of issues that I would like to get help here. The following first describes the background, and then lists the questions towards the end.

Background info

Each one has four network interfaces. Server A resides on subnet A 192.168.15.0/24 and is connected to switch A. Server B resides on subnet B 192.168.16.0/24 and is connected to switch B. The two L3 switches A and B are connected to each other as well. Switch A has VLAN0015 that has a 192.168.15.1 as gw for subnet A. Likewise, switch B has a VLAN0016 that has 192.168.16.1 as gw for subnet B. Each switch has a static route for routing traffic between the two VLANs.

The server network setup requirements are the following:

• All four interfaces on each server should be able to send/receive traffic independently (e.g. ping -I 192.168.15.100 -c 2 192.168.16.103 (note the two IPs have different last byte!) should see ICMP traffic between the pair only), i.e. any interface in a subnet should be able to communicate with any interface on the other subnet.


• Interface bonding is intentionally not used


• All four interfaces should use the gw of their respective subnet (for subnet A: 192.168.15.1; for subnet B: 192.168.16.1) to communicate with interfaces of the other server on the other subnet.

What I have done:

• Added to /etc/iproutes/rt_tables the following:
  
4 ens1f1table
3 ens1f0table
2 ens20f1table
1 ens20f0table

  


• Introduced /etc/sysconfig/network-scripts/ route-* and rule-* for each of the four interfaces on each server (an example is given below)

For example, for server A's interface ens20f0, I have the following in /etc/sysconfig/network-scripts/route-ens20f0:

192.168.15.0/24 dev ens20f0 src 192.168.15.100 table ens20f0table
default via 192.168.15.1 dev ens20f0 table ens20f0table

and in its /etc/sysconfig/network-scripts/rule-ens20f0:

from 192.168.15.100/32 table ens20f0table
to 192.168.15.100/32 table ens20f0table


The setup "sort of works" but from time to time, from one server I couldn't ping any interface of the other server. After some tracerouting, I realized that some interfaces didn't have the right route for traffic. As a get around, I applied the following

/sbin/route add -net 192.168.16.0/24 gw 192.168.15.1 dev ...

to each of the ens20f0|ens20f1|ens1f0|ens1f1 to force the kernel routing table of e.g. server A to look like below:

Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.11.3 0.0.0.0 UG 0 0 0 ens10f0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ens20f0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ens10f0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ens20f1
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ens1f0
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ens1f1
192.168.11.0 0.0.0.0 255.255.255.0 U 0 0 0 ens10f0
192.168.15.0 0.0.0.0 255.255.255.0 U 0 0 0 ens20f0
192.168.15.0 0.0.0.0 255.255.255.0 U 0 0 0 ens20f1
192.168.15.0 0.0.0.0 255.255.255.0 U 0 0 0 ens1f0
192.168.15.0 0.0.0.0 255.255.255.0 U 0 0 0 ens1f1
192.168.16.0 192.168.15.1 255.255.255.0 UG 0 0 0 ens1f1
192.168.16.0 192.168.15.1 255.255.255.0 UG 0 0 0 ens1f0
192.168.16.0 192.168.15.1 255.255.255.0 UG 0 0 0 ens20f1
192.168.16.0 192.168.15.1 255.255.255.0 UG 0 0 0 ens20f0


Questions:

Why I tried used the deprecated /sbin/route instead of ip route add? It's because ip route add wouldn't add the desired route entry.

Obviously, the "get-around" setup is not persistent. So,

1. How can I make my current "get-around" at least persistent across system reboots? There is a /etc/sysconfig/network-scripts/ifup-post, but is it the right place? RHEL 7 documentation doesn't even mention this script, not for non-subscribers anyway.


2. One server A, I have tried to put in 192.168.16.0/24 via 192.168.15.1 dev ens20f0 into the route-ens20f0. After a /sbin/ifdown ens20f0 and then /sbin/ifup ens20f0, the desired route didn't show up in the kernel routing table. What did I do wrong? I reviewed man ip-route(8) many times but couldn't tell.


3. Why the /sbin/route command was able to add a desire route to the kernel routing table, but the newer ip route add|replace couldn't, even on CLI?


4. The man ip-route mentions an append other than ip route { add | del | change | append | replace } ROUTE, but no descriptions whatsoever for the append directive, what is the use of it?


5. Is the approach that I described above the correct way of using "policy-based routing" to meet the requirements that are given to me? It seems to "sort of work", e.g. I could ping from some interfaces of a server using ping -I iface to the interfaces of the other server. But the ability to do so seems not to be reliable, as my tracerouting checks have revealed to me.

I am at a loss here as to what else to try. Repeated reading of RHEL 7 Networking Guide 2.4.1. Configuring a Network Interface Using ifcfg Files didn't help. I would be grateful to any hints as to what I have missed.

--Zack

so, you basically are trying to

1. have four addresses of the same ip-net on the same vlan


2. route traffic over interfaces with similar ip-adresses on the remote net to the remote interfaces with just these similar addresses?

I've never heard of a solution like yours, but i'm out of the network business for a long time now. So just to give you an idea: Why don't you try to subnet the 15.X / 16.x into 4 subnets on the two machines, and then just add routes to the corresponding net and the def-gw to the subnet interfaces?

With a mask of 26 bits (255.255.255.192) this would give 15.0(net) - 15.63(bcast) / 15.64-15.127 / 15.128-15.191 / 15.192-15.255 , similar on the 16.X.

Hope this is of some help...

A host should not have a gateway configured for an IP network to which it is directly connected.

You do not need multiple routing tables for this - that is massively overcomplicating things, and you have not said anything about putting packet marking into your iptables ruleset, so the routing tables are probably being ignored anyway

Also, I strongly recommend you draw up a picture/map of your network, so it shows the IPs and netmasks. Here's a decent example:

So with a map like this, you can follow the path of a packet and see what happens. Your TCP packets should take the same path there and back, whereas ICMP packets are more forgiving.

Now specific to your config, you have the 192.168.15.x/24 network accessible on all four interfaces ens20f0, ens20f1, ens1f0 and ens1f1.

Your kernel will use the last route that was added. Thing is, the switch will have associated your IP address to one of the earlier added routes, and therefore a different MAC address in the switch's ARP table.

So when it works, it works by accident or coincidence.

I suggest you draw up a plan like the example one, but with your details. And I suspect that you'll have a lightbulb moment while drawing it. Which is the entire point.

If no lightbulbs, then post your image and we'll all discuss it further.