@amye Codes are on their way. We'll have some DNS records to manage in order for the site routing to keep working.

Domain: transfer domain to CNCF/LF (ITx)

@amye I own the domain and am ready to transfer. Instructions?

Trademarks: transfer any trademark and logo mark assets over to the LF

Red Hat lawyers are preparing this now

Opened https://bugzilla.redhat.com/show_bug.cgi?id=1874901 but I forget how to link it up :)

Ignore 'hotfix' entries in channels. Remove the 'shadow' class when clicking off of a clicked class.

LGTM!

A few years late here but would love to see this as a Google Font. Great work on the project.

Looks good to me.

Description

When running the kubelet + cri-o in "standalone" mode, aka no control plane, I see what looks like a race condition on Pod startup, with it frequently complaining that the veth mac address doesn't match the expected value.

remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to get network status for pod sandbox k8s_nginx-combined-127.0.0.1_default_19b0558958fc9f867987740a6eb88042_2(f898a0f1feaa8e63f45eaec61a38d4f8f9cf2298cde3879c3726d2b8b0696cfe): Interface vetha38a080a Mac doesn't match: ee:7b:4d:57:3a:d9 not found

The kubelet will back off several times as it reattempts to start the Pod, and typically will eventually succeed. Other times, I will remove the Pod manifest from /etc/kubernetes/manifests and recreate it.

Steps to reproduce the issue: This is verbose, but exactly how I set up the kubelet:

1. On Fedora Core 31

CRI-O is available as a DNF module stream. On FC31 be sure to install the latest version to circumvent this bug: https://bugzilla.redhat.com/show_bug.cgi?id=1754170

2. dnf -y module enable cri-o:1.16 dnf -y install crio
3. Delete any existing CNI rm /etc/cni/net.d/*.conf
4. Add /etc/cni/net.d/100-crio-bridge.conflist

{
  "cniVersion": "0.4.0",
  "name": "bridge-firewalld",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni0",
      "isDefaultGateway": true,
      "isGateway": true,
      "ipMasq": true,
      "ipam": {
        "type": "host-local",
        "subnet": "10.88.0.0/16",
        "routes": [
          {
            "dst": "0.0.0.0/0"
          }
        ]
      }
    },
    { 
      "type": "portmap",
      "capabilities": {
	"portMappings": true
     }
    },
    {
      "type": "firewall"
    }
  ]
}

5. dnf -y install kubernetes-node
6. Disable docker which is brought in automatically systemctl mask docker cp /usr/lib/systemd/system/kubelet.service /etc/systemd/system/kubelet.service

Replace /etc/systemd/system/kubelet.service with crio:

 [Unit]
 Description=Kubernetes Kubelet Server
 Documentation=https://github.com/GoogleCloudPlatform/kubernetes
-After=docker.service
-Requires=docker.service
+After=crio.service
+Requires=crio.service
 
 [Service]
 WorkingDirectory=/var/lib/kubelet

7. systemctl daemon-reload
8. Remove the now obsolete flag --allow-privileged from /etc/kubernetes/config that would prevent the service from starting:

 KUBE_LOG_LEVEL="--v=0"
 
 # Should this cluster be allowed to run privileged docker containers
-KUBE_ALLOW_PRIV="--allow-privileged=true"
+KUBE_ALLOW_PRIV=""
 
 # How the controller-manager, scheduler, and proxy find the apiserver
 KUBE_MASTER="--master=http://127.0.0.1:8080"

9. Apply the below modifications to /etc/kubernetes/kubelet:

 KUBELET_HOSTNAME="--hostname-override=127.0.0.1"
 
 # Add your own!
-KUBELET_ARGS="--cgroup-driver=systemd --fail-swap-on=false"
+KUBELET_ARGS="--cgroup-driver=systemd --fail-swap-on=false --pod-manifest-path=/etc/kubernetes/manifests --container-runtime=remote --container-runtime-endpoint=unix:///var/run/crio/crio.sock --runtime-request-timeout=10m"

10. mkdir /etc/kubernetes/manifests
11. systemctl enable kubelet --now
12. Drop sample Pod in /etc/kubernetes/manifests

apiVersion: v1
kind: Pod
metadata:
  name: echoserver
spec:
  containers:
    - name: nginx-combined
      image: gcr.io/google-containers/echoserver:1.10
      ports:
        - name: web
          containerPort: 8080
          hostPort: 9091
          protocol: TCP
      resources:
        limits:
          cpu: "100m"
          memory: "50Mi"

Describe the results you received:

Mar 19 16:31:17 localhost.localdomain kubelet[1166]: E0319 16:31:17.672240    1166 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to get network status for pod sandbox k8s_nginx-combined-127.0.0.1_default_19b0558958fc9f867987740a6eb88042_2(f898a0f1feaa8e63f45eaec61a38d4f8f9cf2298cde3879c3726d2b8b0696cfe): Interface vetha38a080a Mac doesn't match: ee:7b:4d:57:3a:d9 not found
Mar 19 16:31:17 localhost.localdomain kubelet[1166]: E0319 16:31:17.672295    1166 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "nginx-combined-127.0.0.1_default(19b0558958fc9f867987740a6eb88042)" failed: rpc error: code = Unknown desc = failed to get network status for pod sandbox k8s_nginx-combined-127.0.0.1_default_19b0558958fc9f867987740a6eb88042_2(f898a0f1feaa8e63f45eaec61a38d4f8f9cf2298cde3879c3726d2b8b0696cfe): Interface vetha38a080a Mac doesn't match: ee:7b:4d:57:3a:d9 not found
Mar 19 16:31:17 localhost.localdomain kubelet[1166]: E0319 16:31:17.672322    1166 kuberuntime_manager.go:692] createPodSandbox for pod "nginx-combined-127.0.0.1_default(19b0558958fc9f867987740a6eb88042)" failed: rpc error: code = Unknown desc = failed to get network status for pod sandbox k8s_nginx-combined-127.0.0.1_default_19b0558958fc9f867987740a6eb88042_2(f898a0f1feaa8e63f45eaec61a38d4f8f9cf2298cde3879c3726d2b8b0696cfe): Interface vetha38a080a Mac doesn't match: ee:7b:4d:57:3a:d9 not found
Mar 19 16:31:17 localhost.localdomain kubelet[1166]: E0319 16:31:17.672392    1166 pod_workers.go:190] Error syncing pod 19b0558958fc9f867987740a6eb88042 ("nginx-combined-127.0.0.1_default(19b0558958fc9f867987740a6eb88042)"), skipping: failed to "CreatePodSandbox" for "nginx-combined-127.0.0.1_default(19b0558958fc9f867987740a6eb88042)" with CreatePodSandboxError: "CreatePodSandbox for pod \"nginx-combined-127.0.0.1_default(19b0558958fc9f867987740a6eb88042)\" failed: rpc error: code = Unknown desc = failed to get network status for pod sandbox k8s_nginx-combined-127.0.0.1_default_19b0558958fc9f867987740a6eb88042_2(f898a0f1feaa8e63f45eaec61a38d4f8f9cf2298cde3879c3726d2b8b0696cfe): Interface vetha38a080a Mac doesn't match: ee:7b:4d:57:3a:d9 not found"

Describe the results you expected: Pod starts the first time the kubelet attempts it.

Additional information you deem important (e.g. issue happens only occasionally): This seems to be a race condition with the veth being created because it will eventually work most of the time. I don't know how to instrument it further.

Output of crio --version:

crio -v
crio version 1.16.1
commit: "53843bd41d4bb3385c3e035bbf082e629b46f3bd-dirty"

Additional environment details (AWS, VirtualBox, physical, etc.): Fedora Core 31 on x86 workstation, no virtualization

$ /usr/bin/kubelet --version
Kubernetes v1.15.8-beta.0

LGTM, his sounds great to me. Very pragmatic with low risk.