Configuring CentOS 7 with two IP addresses on Same LAN

[wits-zach]

Hello all,

I have a CentOS 7 server with an Emulex OCE14102-NT card installed into one of its PCI slots. The card has two 10Gbps NICs with unique MAC addresses for each NIC. I am trying to configure each NIC with an address on a 172.16.20.0/16 network. I have disabled NetworkManager and statically assigned each NIC an address. The issue I am running into is that only one of the NICs is able to communicate out. How do I get both NICs to be reachable on the LAN? I am assuming that it is a gateway/routing conflict, but I am not sure what I am doing wrong. Config details are below:

• NIC enp4s0f0 is able to ping its gateway at 172.16.20.1 and another host at 172.16.20.226. However, enp4s0f1 cannot. In fact, enp4s0f0 cannot ping enp4s0f1 on the same host; only the loopback address can ping it.
• The ifcfg-<name> scripts are configured identically as far as I can tell
• The management network is on 192.168.1.0/24.

ifconfig:​

enp4s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500​

inet 172.16.20.225 netmask 255.255.0.0 broadcast 172.16.255.255​

inet6 fe80::290:faff:fe9d:b1be prefixlen 64 scopeid 0x20<link>​

ether 00:90:fa:9d:be:b1 txqueuelen 1000 (Ethernet)​

RX packets 710 bytes 96452 (94.1 KiB)​

RX errors 0 dropped 0 overruns 0 frame 0​

TX packets 14 bytes 908 (908.0 B)​

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0​

​

enp4s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500​

inet 172.16.20.245 netmask 255.255.0.0 broadcast 172.16.255.255​

inet6 fe80::290:faff:fe9d:b1c6 prefixlen 64 scopeid 0x20<link>​

ether 00:90:fa:9d:c6:b1 txqueuelen 1000 (Ethernet)​

RX packets 701 bytes 94520 (92.3 KiB)​

RX errors 0 dropped 0 overruns 0 frame 0​

TX packets 14 bytes 908 (908.0 B)​

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0​

pings:​

[root@host]# ping 172.16.20.226 -I enp4s0f0​

PING 172.16.20.226 (172.16.20.226) from 172.16.20.225 enp4s0f0: 56(84) bytes of data.​

64 bytes from 172.16.20.226: icmp_seq=1 ttl=64 time=0.185 ms​

64 bytes from 172.16.20.226: icmp_seq=2 ttl=64 time=0.132 ms​

64 bytes from 172.16.20.226: icmp_seq=3 ttl=64 time=0.163 ms​

64 bytes from 172.16.20.226: icmp_seq=4 ttl=64 time=0.162 ms​

64 bytes from 172.16.20.226: icmp_seq=5 ttl=64 time=0.167 ms​

64 bytes from 172.16.20.226: icmp_seq=6 ttl=64 time=0.166 ms​

^C​

--- 172.16.20.226 ping statistics ---​

6 packets transmitted, 6 received, 0% packet loss, time 4999ms​

rtt min/avg/max/mdev = 0.132/0.162/0.185/0.020 ms​

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

​

[root@host]# ping 172.16.20.226 -I enp4s0f1​

PING 172.16.20.226 (172.16.20.226) from 172.16.20.245 enp4s0f1: 56(84) bytes of data.​

^C​

--- 172.16.20.226 ping statistics ---​

5 packets transmitted, 0 received, 100% packet loss, time 3999ms​

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
​

[root@host]# ping 172.16.20.245 -I enp4s0f0​

PING 172.16.20.245 (172.16.20.245) from 172.16.20.225 enp4s0f0: 56(84) bytes of data.​

^C​

--- 172.16.20.245 ping statistics ---​

6 packets transmitted, 0 received, 100% packet loss, time 4999ms​

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------​

[root@ovirt1 witsadmin]# ping 172.16.20.245 -I lo​

ping: Warning: source address might be selected on device other than lo.​

PING 172.16.20.245 (172.16.20.245) from 172.16.20.225 lo: 56(84) bytes of data.​

64 bytes from 172.16.20.245: icmp_seq=1 ttl=64 time=0.040 ms​

64 bytes from 172.16.20.245: icmp_seq=2 ttl=64 time=0.021 ms​

64 bytes from 172.16.20.245: icmp_seq=3 ttl=64 time=0.021 ms​

64 bytes from 172.16.20.245: icmp_seq=4 ttl=64 time=0.021 ms​

64 bytes from 172.16.20.245: icmp_seq=5 ttl=64 time=0.021 ms​

64 bytes from 172.16.20.245: icmp_seq=6 ttl=64 time=0.022 ms​

^C​

--- 172.16.20.245 ping statistics ---​

6 packets transmitted, 6 received, 0% packet loss, time 4999ms​

rtt min/avg/max/mdev = 0.021/0.024/0.040/0.008 ms​

Routing Table:

[root@host]# route -n​

Kernel IP routing table​

Destination Gateway Genmask Flags Metric Ref Use Iface​

0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 mgmt​

169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 enp4s0f0​

169.254.0.0 0.0.0.0 255.255.0.0 U 1005 0 0 enp4s0f1​

169.254.0.0 0.0.0.0 255.255.0.0 U 1021 0 0 mgmt​

172.16.0.0 0.0.0.0 255.255.0.0 U 0 0 0 enp4s0f0​

172.16.0.0 0.0.0.0 255.255.0.0 U 0 0 0 enp4s0f1​

192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 mgmt​

Network-Scripts:

[root@host]# cat ifcfg-enp4s0f0​

TYPE=Ethernet​

PROXY_METHOD=none​

BROWSER_ONLY=no​

BOOTPROTO=none​

DEFROUTE=no​

IPV4_FAILURE_FATAL=no​

IPV6INIT=no​

IPV6_AUTOCONF=no​

IPV6_DEFROUTE=no​

IPV6_FAILURE_FATAL=no​

IPV6_ADDR_GEN_MODE=stable-privacy​

NAME=enp4s0f0​

UUID=6cbb08e3-XXXX-YYYY-ZZZZ-XXXXXXXXXXX​

DEVICE=enp4s0f0​

ONBOOT=yes​

IPADDR=172.16.20.225​

NETMASK=255.255.0.0​

GATEWAY=172.16.20.1​

NM_CONTROLLED=no​

ZONE=public​

[root@host]# cat ifcfg-enp4s0f1​

TYPE=Ethernet​

PROXY_METHOD=none​

BROWSER_ONLY=no​

BOOTPROTO=none​

DEFROUTE=no​

IPV4_FAILURE_FATAL=no​

IPV6INIT=no​

IPV6_AUTOCONF=no​

IPV6_DEFROUTE=no​

IPV6_FAILURE_FATAL=no​

IPV6_ADDR_GEN_MODE=stable-privacy​

NAME=enp4s0f1​

UUID=6cbb08e3-XXXX-YYYY-ZZZZ-XXXXXXXXXXX​

DEVICE=enp4s0f1​

ONBOOT=yes​

IPADDR=172.16.20.245​

NETMASK=255.255.0.0​

GATEWAY=172.16.20.1​

NM_CONTROLLED=no​

ZONE=public​

I am assuming I have done something wrong with enp4s0f1's gateway/routing (perhaps one NIC needs to be the "gateway" for both?) etc., but because enp4s0f0 is working correctly I am at a little bit a of a loss. Can anyone tell me where I can going wrong?

 

[TechnoJunky]

First, and this is just me being anal, you don't have 1 card with 2 NICs. You have 1 NIC with 2 ports. NIC is an accronym for Network Interface Card. So you don't have 1 card with 2 cards on it . Also, not that you did it, but when people say NIC card, it gets me too. Network Interface Card card.

Anyway, it sounds like you've done all the software troubleshooting, but have you done the hardware troubleshooting? Do you KNOW it's a good card? Have you used this card in any other computer? Do you know that it CAN have both ports working at the same time? Try unplugging the good port and see if it will fail over to the bad port. Try switching the cables (try plugging port A's cable into Port B and vise versa). I work at a large company and know that we sometimes get brand new NICs with one of the ports bad and sometimes cables are the same way, brand new and just don't work. Also it could be the switch/hub port as well. Try switching the cables at the switch and see if the network traffic flows over the other port on the card now.

The NIC should have lights on it letting you know if it has a connection for both ports. Are the lights on for both ports? There should be 2 lights each, 1 indicating the connection and the other indicating the acknowledgement of traffic (blinking). Both ports should be seeing this. Also have you checked the switch to confirm it sees the connection on both of it's ports?

 

[wits-zach]

First, and this is just me being anal, you don't have 1 card with 2 NICs. You have 1 NIC with 2 ports. NIC is an accronym for Network Interface Card. So you don't have 1 card with 2 cards on it . Also, not that you did it, but when people say NIC card, it gets me too. Network Interface Card card.

Anyway, it sounds like you've done all the software troubleshooting, but have you done the hardware troubleshooting? Do you KNOW it's a good card? Have you used this card in any other computer? Do you know that it CAN have both ports working at the same time? Try unplugging the good port and see if it will fail over to the bad port. Try switching the cables (try plugging port A's cable into Port B and vise versa). I work at a large company and know that we sometimes get brand new NICs with one of the ports bad and sometimes cables are the same way, brand new and just don't work. Also it could be the switch/hub port as well. Try switching the cables at the switch and see if the network traffic flows over the other port on the card now.

The NIC should have lights on it letting you know if it has a connection for both ports. Are the lights on for both ports? There should be 2 lights each, 1 indicating the connection and the other indicating the acknowledgement of traffic (blinking). Both ports should be seeing this. Also have you checked the switch to confirm it sees the connection on both of it's ports?

1) I admittedly do not know if it is a good card, but I have two of the exact same card in two different systems and they are both behaving this way with the same configuration, same server, same OS. Seems unlikely to me that that I would get two bad card/cable combinations.

2) All lights are on for both ports on the NIC and on the 10Gbps switch, indicating connectivity.

3) The switch's admin interface is showing 10Gbps Full-Duplex for both ports, indicating it is reading that it sees the ports are live.

4) I also meant to include this, but I get !H errors when I attempt to traceroute to hosts that are up.

 

[TechnoJunky]

Well, it sounds like it's not hardware, but I'd still try swapping the cables. Just to rule it out. If it's not that, then it has to be software settings. Unfortunatly I can't help much here, but I"m sure someone eles on here can. I do know though that you can configure your NICs to be teamed with fault tolerance, or not teamed. See if you can find those settings. You don't want them teamed, you want each to run independently.

 

[dos2unix]

what is the output of "ip addr" and "netstat -nr"

Just curious, since your gateway is on the same subnet as your IP addresses
why not use a /24 netmask? (255.255.255.0)