profile
viewpoint

Ask questionsPods can't reach networks outside of node

I'have a runing rke cluster v1.0 in a one note debian buster installation. but i can't reach networks outside of my node.

RKE version: 1.0.0

### version of rke

user@node1:~$ rke -v
rke version v1.0.0

Docker version: (docker version,docker info preferred)

user@node1:~$ docker info
Client:
 Debug Mode: false

Server:
 Containers: 65
  Running: 22
  Paused: 0
  Stopped: 43
 Images: 35
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.19.0-6-amd64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.263GiB
 Name: node1
 ID: 3VJ7:6ZSO:4MDH:43PM:S5YM:THLX:XCZS:X2YH:7U6N:ISNJ:F5K7:ILWW
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support


Operating system and kernel: (cat /etc/os-release, uname -r preferred)

### Kernel
user@node1:~$ uname -r
4.19.0-6-amd64

Type/provider of hosts:

Bare-metal

cluster.yml file:


# If you intened to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes:
- address: node1.fritz.box
  port: "22"
  internal_address: node1.fritz.box
  role:
  - controlplane
  - worker
  - etcd
  hostname_override: ""
  user: rke
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: ~/.ssh/id_rsa
  ssh_cert: ""
  ssh_cert_path: ""
  labels: {}
  taints: []
services:
  etcd:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    external_urls: []
    ca_cert: ""
    cert: ""
    key: ""
    path: ""
    uid: 0
    gid: 0
    snapshot: null
    retention: ""
    creation: ""
    backup_config: null
  kube-api:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    service_cluster_ip_range: 10.43.0.0/16
    service_node_port_range: ""
    pod_security_policy: false
    always_pull_images: false
    secrets_encryption_config: null
    audit_log: null
    admission_configuration: null
    event_rate_limit: null
  kube-controller:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_cidr: 10.42.0.0/16
    service_cluster_ip_range: 10.43.0.0/16
  scheduler:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
  kubelet:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_domain: cluster.local
    infra_container_image: ""
    cluster_dns_server: 10.43.0.10
    fail_swap_on: false
    generate_serving_certificate: false
  kubeproxy:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
network:
  plugin: canal
  options: {}
  node_selector: {}
authentication:
  strategy: x509
  sans: []
  webhook: null
addons: ""
addons_include: []
system_images:
  etcd: rancher/coreos-etcd:v3.3.15-rancher1
  alpine: rancher/rke-tools:v0.1.51
  nginx_proxy: rancher/rke-tools:v0.1.51
  cert_downloader: rancher/rke-tools:v0.1.51
  kubernetes_services_sidecar: rancher/rke-tools:v0.1.51
  kubedns: rancher/k8s-dns-kube-dns:1.15.0
  dnsmasq: rancher/k8s-dns-dnsmasq-nanny:1.15.0
  kubedns_sidecar: rancher/k8s-dns-sidecar:1.15.0
  kubedns_autoscaler: rancher/cluster-proportional-autoscaler:1.7.1
  coredns: rancher/coredns-coredns:1.6.2
  coredns_autoscaler: rancher/cluster-proportional-autoscaler:1.7.1
  kubernetes: rancher/hyperkube:v1.16.3-rancher1
  flannel: rancher/coreos-flannel:v0.11.0-rancher1
  flannel_cni: rancher/flannel-cni:v0.3.0-rancher5
  calico_node: rancher/calico-node:v3.8.1
  calico_cni: rancher/calico-cni:v3.8.1
  calico_controllers: rancher/calico-kube-controllers:v3.8.1
  calico_ctl: ""
  calico_flexvol: rancher/calico-pod2daemon-flexvol:v3.8.1
  canal_node: rancher/calico-node:v3.8.1
  canal_cni: rancher/calico-cni:v3.8.1
  canal_flannel: rancher/coreos-flannel:v0.11.0
  canal_flexvol: rancher/calico-pod2daemon-flexvol:v3.8.1
  weave_node: weaveworks/weave-kube:2.5.2
  weave_cni: weaveworks/weave-npc:2.5.2
  pod_infra_container: rancher/pause:3.1
  ingress: rancher/nginx-ingress-controller:nginx-0.25.1-rancher1
  ingress_backend: rancher/nginx-ingress-controller-defaultbackend:1.5-rancher1
  metrics_server: rancher/metrics-server:v0.3.4
  windows_pod_infra_container: rancher/kubelet-pause:v0.1.3
ssh_key_path: ~/.ssh/id_rsa
ssh_cert_path: ""
ssh_agent_auth: false
authorization:
  mode: none
  options: {}
ignore_docker_version: false
kubernetes_version: ""
private_registries: []
ingress:
  provider: ""
  options: {}
  node_selector: {}
  extra_args: {}
  dns_policy: ""
  extra_envs: []
  extra_volumes: []
  extra_volume_mounts: []
cluster_name: ""
cloud_provider:
  name: ""
prefix_path: ""
addon_job_timeout: 0
bastion_host:
  address: ""
  port: ""
  user: ""
  ssh_key: ""
  ssh_key_path: ""
  ssh_cert: ""
  ssh_cert_path: ""
monitoring:
  provider: ""
  options: {}
  node_selector: {}
restore:
  restore: false
  snapshot_name: ""
dns: null

shell-demo.yml file:


apiVersion: v1
kind: Pod
metadata:
  name: shell-demo
spec:
  volumes:
  - name: shared-data
    emptyDir: {}
  containers:
  - name: multitool
    image: praqma/network-multitool
    volumeMounts:
    - name: shared-data
      mountPath: /usr/share/nginx/html
  dnsPolicy: Default

Steps to Reproduce:

rancher up
kubectl apply -f shell-demo.yml

Results:

not rachabel networkes outside the cluster.

user@node1:~$ kubectl exec -it shell-demo -- /bin/ping -c 2 192.168.178.1
PING 192.168.178.1 (192.168.178.1) 56(84) bytes of data.

--- 192.168.178.1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 9ms

command terminated with exit code 1

So without route Internet and DNS are also not reachable:


user@node1:~$ kubectl exec -it shell-demo -- /bin/ping -c 2 www.heise.de
ping: www.heise.de: Try again
command terminated with exit code 2

my notes of searching the error

Nodes could access other networks.

### version of rke

user@node1:~$ rke -v
rke version v1.0.0

### version of docker

user@node1:~/Projects/wudo7-rke$ docker -v
Docker version 19.03.5, build 633a0ea838

### ip settings

user@node1:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 54:80:28:4f:da:0c brd ff:ff:ff:ff:ff:ff
    inet 192.168.178.100/24 brd 192.168.178.255 scope global dynamic noprefixroute enp2s0f0
       valid_lft 863933sec preferred_lft 863933sec
    inet6 2003:a:42f:8900:f973:d5cf:11f9:652e/64 scope global temporary dynamic 
       valid_lft 7127sec preferred_lft 1241sec
    inet6 2003:a:42f:8900:5680:28ff:fe4f:da0c/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 7127sec preferred_lft 1241sec
    inet6 fe80::5680:28ff:fe4f:da0c/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
...

### version of debian

user@node1:~$ cat /etc/debian_version 
10.2cd

###

user@node1:~$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

### Kernel
user@node1:~$ uname -r
4.19.0-6-amd64


### router is reachble

user@node1:~$ ping -c 1 192.168.178.1
PING 192.168.178.1 (192.168.178.1) 56(84) bytes of data.
64 bytes from 192.168.178.1: icmp_seq=1 ttl=64 time=0.388 ms

--- 192.168.178.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.388/0.388/0.388/0.000 ms

### real world is reachble

user@node1:~$ ping -4 -c 1 www.google.de
PING www.google.de (172.217.16.131) 56(84) bytes of data.
64 bytes from zrh04s06-in-f131.1e100.net (172.217.16.131): icmp_seq=1 ttl=56 time=18.10 ms

--- www.google.de ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 18.978/18.978/18.978/0.000 ms

### routes are set like this

user@node1:~$ ip route
default via 192.168.178.1 dev enp2s0f0 proto dhcp metric 100 
10.42.0.0/24 dev cni0 proto kernel scope link src 10.42.0.1 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.178.0/24 dev enp2s0f0 proto kernel scope link src 192.168.178.100 metric 100

### firewall settings

user@node1:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            
# Warning: iptables-legacy tables present, use iptables-legacy to see them


### and the iptables-legacy roules 

user@node1:~$ sudo iptables-legacy -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
ACCEPT     all  --  10.42.0.0/16         anywhere            
ACCEPT     all  --  anywhere             10.42.0.0/16        

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            

Chain KUBE-EXTERNAL-SERVICES (1 references)
target     prot opt source               destination         

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-FORWARD (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             ctstate INVALID
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT     all  --  10.42.0.0/16         anywhere             /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             10.42.0.0/16         /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-SERVICES (3 references)
target     prot opt source               destination         

### from a started docker container, not in cluster, internt is also reachble.

user@node1:~$ docker run  -a stdout -i -t praqma/network-multitool /bin/ping -c 1 www.heise.de
The directory /usr/share/nginx/html is not mounted.
Over-writing the default index.html file with some useful information.
PING www.heise.de (193.99.144.85) 56(84) bytes of data.
64 bytes from www.heise.de (193.99.144.85): icmp_seq=1 ttl=246 time=19.2 ms

--- www.heise.de ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 19.219/19.219/19.219/0.000 ms


But inside k8s, all pods i can't reach the surrounding network nodes, PCs or router.

kubectl exec -it shell-demo -- /bin/bash
bash-5.0# 

### then inside the Pod

bash-5.0# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 0a:58:0a:2a:00:0f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.42.0.15/24 scope global eth0
       valid_lft forever preferred_lft forever

### routing inside my pod

bash-5.0# ip route
default via 10.42.0.1 dev eth0 
10.42.0.0/24 dev eth0 proto kernel scope link src 10.42.0.15 
10.42.0.0/16 via 10.42.0.1 dev eth0

### ping cluster intern:

bash-5.0# ping -c2  10.42.0.1
PING 10.42.0.1 (10.42.0.1) 56(84) bytes of data.
64 bytes from 10.42.0.1: icmp_seq=1 ttl=64 time=0.134 ms
64 bytes from 10.42.0.1: icmp_seq=2 ttl=64 time=0.117 ms

--- 10.42.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 20ms
rtt min/avg/max/mdev = 0.117/0.125/0.134/0.014 ms

### inside the pod could reach the hosting node.

bash-5.0# ping -c 2 192.168.178.100
PING 192.168.178.100 (192.168.178.100) 56(84) bytes of data.
64 bytes from 192.168.178.100: icmp_seq=1 ttl=64 time=0.102 ms
64 bytes from 192.168.178.100: icmp_seq=2 ttl=64 time=0.098 ms

--- 192.168.178.100 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 32ms
rtt min/avg/max/mdev = 0.098/0.100/0.102/0.002 ms

### but I can't reach the PCs, router or dns.

ping -c 2 192.168.178.1
PING 192.168.178.1 (192.168.178.1) 56(84) bytes of data.
^C
--- 192.168.178.1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 9ms


rancher/rke

Answer questions GrosseBen

is Buster config of rke part of the Problem?

Maybe that's an Debian Buster specific rke problem. I have the same behavior on k3s.

useful!

Related questions

"Failed to reconcile etcd plane" when updating RKE binary hot 3
Failed to get /health for host - remote error: tls: bad certificate hot 2
Failed to rotate expired certificates on an RKE cluster: unable to reach api server to fetch CA hot 2
Error response from daemon: chown /etc/resolv.conf: operation not permitted hot 1
rke 0.1.17 Can't initiate NewClient: protocol not available hot 1
Calico node failed to start after upgrading the cluster hot 1
Job rke-network-plugin-deploy-job never completes (virtualbox) hot 1
rke up --local fails to deploy successfully hot 1
Job rke-network-plugin-deploy-job never completes (virtualbox) hot 1
go panic on intial rke up hot 1
Unable to update cluster "crypto/rsa: verification error" hot 1
Calico node failed to start after upgrading the cluster hot 1
pods in "CrashLoopBackOff" status after restoring from backup hot 1
[SOLVED] Failed to apply the ServiceAccount needed for job execution: Post https://10.102.X.X:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings: Forbidden hot 1
Failed to get /health for host - remote error: tls: bad certificate hot 1
Github User Rank List