Routing packets to a container that has a destination that is an interface's IP, stopped working in recent...
I have a flannel-based k8s system with a container at 10.233.64.18.
I also have an interface ens224:1 with the IP address 10.5.2.11
With older kernels (3.x) I'm able to force packets destined to 10.5.2.11 to be routed to the container using a fwmark and an ip rule, and matching ip route entries. This is useful for dealing with ingress locally in k8s instead of the usual "ingress is someone else's problem" that is typical with k8s. (I want full control: I'm running a custom router on a raw socket in that container).
This has stopped working in 4.x kernels I've tried. The packets (e.g. ping) just dead end. I've verified I have rp_filter set to 0 and martian packet logging enabled (and no such log messages).
If I ping from a remote machine, the packets to 10.5.2.11 come in to ens224 but go no where. If I ping 10.5.2.11 from the local machine packets get routed correctly.
The following is the config that works in 3.x kernels:
IP Tables:
-A PREROUTING -d 10.5.2.11/32 -j MARK --set-xmark 0x1/0x1
-A OUTPUT -d 10.5.2.11/32 -j MARK --set-xmark 0x1/0x1
IP Rules:
49: from iif l0 lookup 245
50: from all fwmark 0x1/0x1 lookup 245
100 from all lookup 246
ip route show table 245
10.5.2.11 via 10.233.64.18 dev cni0
Table 246 shows the local route that would be normally in table local, which I've deleted from table local. I moved it to table 246 so that packets are not routed to the Linux socket listener tables. But at the same time I want the interface to respond to ARPs.
ip route show table 246
local 10.5.2.11 dev ens224 proto kernel scope host src 10.5.2.11
ip route delete 10.5.2.11 table local
And for those not familiar with CNI, it's just a bridge. In the non-working condition, I see no ICMP packets on cni0 on 4.x, but I do see them get routed to cni0 in 3.x
bridge vlan
port cni0 vlan 1 ids PVID Egress Untaged
[...]
So, why does this arrangement work on Linux version 3.x but not 4.x? Or perhaps there's something else causing the iCMP (or TCP, or UDP..) packets to get routed nowhere?
iptables routing
edited Dec 18 at 3:41
Michael Prokopec
1,022116
asked Dec 17 at 20:47
Paul S
101
add a comment |
I have a flannel-based k8s system with a container at 10.233.64.18.
I also have an interface ens224:1 with the IP address 10.5.2.11
With older kernels (3.x) I'm able to force packets destined to 10.5.2.11 to be routed to the container using a fwmark and an ip rule, and matching ip route entries. This is useful for dealing with ingress locally in k8s instead of the usual "ingress is someone else's problem" that is typical with k8s. (I want full control: I'm running a custom router on a raw socket in that container).
This has stopped working in 4.x kernels I've tried. The packets (e.g. ping) just dead end. I've verified I have rp_filter set to 0 and martian packet logging enabled (and no such log messages).
If I ping from a remote machine, the packets to 10.5.2.11 come in to ens224 but go no where. If I ping 10.5.2.11 from the local machine packets get routed correctly.
The following is the config that works in 3.x kernels:
IP Tables:
-A PREROUTING -d 10.5.2.11/32 -j MARK --set-xmark 0x1/0x1
-A OUTPUT -d 10.5.2.11/32 -j MARK --set-xmark 0x1/0x1
IP Rules:
49: from iif l0 lookup 245
50: from all fwmark 0x1/0x1 lookup 245
100 from all lookup 246
ip route show table 245
10.5.2.11 via 10.233.64.18 dev cni0
Table 246 shows the local route that would be normally in table local, which I've deleted from table local. I moved it to table 246 so that packets are not routed to the Linux socket listener tables. But at the same time I want the interface to respond to ARPs.
ip route show table 246
local 10.5.2.11 dev ens224 proto kernel scope host src 10.5.2.11
ip route delete 10.5.2.11 table local
And for those not familiar with CNI, it's just a bridge. In the non-working condition, I see no ICMP packets on cni0 on 4.x, but I do see them get routed to cni0 in 3.x
bridge vlan
port cni0 vlan 1 ids PVID Egress Untaged
[...]
So, why does this arrangement work on Linux version 3.x but not 4.x? Or perhaps there's something else causing the iCMP (or TCP, or UDP..) packets to get routed nowhere?
iptables routing
edited Dec 18 at 3:41
Michael Prokopec
1,022116
asked Dec 17 at 20:47
Paul S
101
I'm not familiar with flannel, and I don't understand your goal completely, but (1)ens224:1is not an "interface", it's just the old way to add multiple IP addresses to the interfaceens224, (2) thelocaltable etc. is a bit involved (I'd have to look this up), and messing with it will likely break things, as this probably changed from 3.x to 4.x, (3) if all you want is an interface in a container which acts like it was a part ofens224with an IP address of10.5.2.11/32, just use amacvlaninterface in your container withens224as master. No routing needed.
– dirkt
Dec 18 at 7:27
add a comment |
I have a flannel-based k8s system with a container at 10.233.64.18.
I also have an interface ens224:1 with the IP address 10.5.2.11
With older kernels (3.x) I'm able to force packets destined to 10.5.2.11 to be routed to the container using a fwmark and an ip rule, and matching ip route entries. This is useful for dealing with ingress locally in k8s instead of the usual "ingress is someone else's problem" that is typical with k8s. (I want full control: I'm running a custom router on a raw socket in that container).
This has stopped working in 4.x kernels I've tried. The packets (e.g. ping) just dead end. I've verified I have rp_filter set to 0 and martian packet logging enabled (and no such log messages).
If I ping from a remote machine, the packets to 10.5.2.11 come in to ens224 but go no where. If I ping 10.5.2.11 from the local machine packets get routed correctly.
The following is the config that works in 3.x kernels:
IP Tables:
-A PREROUTING -d 10.5.2.11/32 -j MARK --set-xmark 0x1/0x1
-A OUTPUT -d 10.5.2.11/32 -j MARK --set-xmark 0x1/0x1
IP Rules:
49: from iif l0 lookup 245
50: from all fwmark 0x1/0x1 lookup 245
100 from all lookup 246
ip route show table 245
10.5.2.11 via 10.233.64.18 dev cni0
Table 246 shows the local route that would be normally in table local, which I've deleted from table local. I moved it to table 246 so that packets are not routed to the Linux socket listener tables. But at the same time I want the interface to respond to ARPs.
ip route show table 246
local 10.5.2.11 dev ens224 proto kernel scope host src 10.5.2.11
ip route delete 10.5.2.11 table local
And for those not familiar with CNI, it's just a bridge. In the non-working condition, I see no ICMP packets on cni0 on 4.x, but I do see them get routed to cni0 in 3.x
bridge vlan
port cni0 vlan 1 ids PVID Egress Untaged
[...]
So, why does this arrangement work on Linux version 3.x but not 4.x? Or perhaps there's something else causing the iCMP (or TCP, or UDP..) packets to get routed nowhere?
iptables routing
edited Dec 18 at 3:41
Michael Prokopec
1,022116
asked Dec 17 at 20:47
Paul S
101
I have a flannel-based k8s system with a container at 10.233.64.18.
I also have an interface ens224:1 with the IP address 10.5.2.11
With older kernels (3.x) I'm able to force packets destined to 10.5.2.11 to be routed to the container using a fwmark and an ip rule, and matching ip route entries. This is useful for dealing with ingress locally in k8s instead of the usual "ingress is someone else's problem" that is typical with k8s. (I want full control: I'm running a custom router on a raw socket in that container).
This has stopped working in 4.x kernels I've tried. The packets (e.g. ping) just dead end. I've verified I have rp_filter set to 0 and martian packet logging enabled (and no such log messages).
If I ping from a remote machine, the packets to 10.5.2.11 come in to ens224 but go no where. If I ping 10.5.2.11 from the local machine packets get routed correctly.
The following is the config that works in 3.x kernels:
IP Tables:
-A PREROUTING -d 10.5.2.11/32 -j MARK --set-xmark 0x1/0x1
-A OUTPUT -d 10.5.2.11/32 -j MARK --set-xmark 0x1/0x1
IP Rules:
49: from iif l0 lookup 245
50: from all fwmark 0x1/0x1 lookup 245
100 from all lookup 246
ip route show table 245
10.5.2.11 via 10.233.64.18 dev cni0
Table 246 shows the local route that would be normally in table local, which I've deleted from table local. I moved it to table 246 so that packets are not routed to the Linux socket listener tables. But at the same time I want the interface to respond to ARPs.
ip route show table 246
local 10.5.2.11 dev ens224 proto kernel scope host src 10.5.2.11
ip route delete 10.5.2.11 table local
And for those not familiar with CNI, it's just a bridge. In the non-working condition, I see no ICMP packets on cni0 on 4.x, but I do see them get routed to cni0 in 3.x
bridge vlan
port cni0 vlan 1 ids PVID Egress Untaged
[...]
So, why does this arrangement work on Linux version 3.x but not 4.x? Or perhaps there's something else causing the iCMP (or TCP, or UDP..) packets to get routed nowhere?
iptables routing
iptables routing
edited Dec 18 at 3:41
Michael Prokopec
1,022116
asked Dec 17 at 20:47
Paul S
101
edited Dec 18 at 3:41
Michael Prokopec
1,022116
asked Dec 17 at 20:47
Paul S
101
edited Dec 18 at 3:41
Michael Prokopec
1,022116
edited Dec 18 at 3:41
Michael Prokopec
1,022116
edited Dec 18 at 3:41
Michael Prokopec
1,022116
1,022116
asked Dec 17 at 20:47
Paul S
101
asked Dec 17 at 20:47
Paul S
101
asked Dec 17 at 20:47
Paul S
101
101
I'm not familiar with flannel, and I don't understand your goal completely, but (1)ens224:1is not an "interface", it's just the old way to add multiple IP addresses to the interfaceens224, (2) thelocaltable etc. is a bit involved (I'd have to look this up), and messing with it will likely break things, as this probably changed from 3.x to 4.x, (3) if all you want is an interface in a container which acts like it was a part ofens224with an IP address of10.5.2.11/32, just use amacvlaninterface in your container withens224as master. No routing needed.
– dirkt
Dec 18 at 7:27
add a comment |
I'm not familiar with flannel, and I don't understand your goal completely, but (1)ens224:1is not an "interface", it's just the old way to add multiple IP addresses to the interfaceens224, (2) thelocaltable etc. is a bit involved (I'd have to look this up), and messing with it will likely break things, as this probably changed from 3.x to 4.x, (3) if all you want is an interface in a container which acts like it was a part ofens224with an IP address of10.5.2.11/32, just use amacvlaninterface in your container withens224as master. No routing needed.
– dirkt
Dec 18 at 7:27
I'm not familiar with flannel, and I don't understand your goal completely, but (1)
ens224:1 is not an "interface", it's just the old way to add multiple IP addresses to the interface ens224, (2) the local table etc. is a bit involved (I'd have to look this up), and messing with it will likely break things, as this probably changed from 3.x to 4.x, (3) if all you want is an interface in a container which acts like it was a part of ens224 with an IP address of 10.5.2.11/32, just use a macvlan interface in your container with ens224 as master. No routing needed.– dirkt
Dec 18 at 7:27
I'm not familiar with flannel, and I don't understand your goal completely, but (1)
ens224:1 is not an "interface", it's just the old way to add multiple IP addresses to the interface ens224, (2) the local table etc. is a bit involved (I'd have to look this up), and messing with it will likely break things, as this probably changed from 3.x to 4.x, (3) if all you want is an interface in a container which acts like it was a part of ens224 with an IP address of 10.5.2.11/32, just use a macvlan interface in your container with ens224 as master. No routing needed.– dirkt
Dec 18 at 7:27
add a comment |
active
oldest
votes
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f489563%2frouting-packets-to-a-container-that-has-a-destination-that-is-an-interfaces-ip%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f489563%2frouting-packets-to-a-container-that-has-a-destination-that-is-an-interfaces-ip%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I'm not familiar with flannel, and I don't understand your goal completely, but (1)
ens224:1is not an "interface", it's just the old way to add multiple IP addresses to the interfaceens224, (2) thelocaltable etc. is a bit involved (I'd have to look this up), and messing with it will likely break things, as this probably changed from 3.x to 4.x, (3) if all you want is an interface in a container which acts like it was a part ofens224with an IP address of10.5.2.11/32, just use amacvlaninterface in your container withens224as master. No routing needed.– dirkt
Dec 18 at 7:27