profile
viewpoint

Ask questions"Failed to reconcile etcd plane" when updating RKE binary

RKE version: v0.1.17 and v0.2.1+

Docker version: (docker version,docker info preferred) 18.9.4

Operating system and kernel: (cat /etc/os-release, uname -r preferred) Kernel Version: 4.4.0-142-generic Ubuntu 16.04.5 LTS

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO) VM

cluster.yml file:

I recreated with similar:

nodes: 
- address: 127.0.0.1 
  user: root 
  port: 22
  role: 
  - controlplane 
  - etcd
  - worker

kubernetes_version: v1.12.6-rancher1-1

Steps to Reproduce:

  1. create cluster with rke 0.1.17 and kubernetes version 1.12.6
  2. upgrade rke binary to 0.2.1
  3. run rke up to upgrade kubernetes to 1.12.7

Results:

The customer got this output (via ansible, but shouldn't be relavent)

"time=\"2019-04-25T11:27:50+02:00\" level=info msg=\"[reconcile] host [localhost] is active master on the cluster\"", 
"time=\"2019-04-25T11:27:50+02:00\" level=info msg=\"[certificates] Successfully deployed kubernetes certificates to Cluster nodes\"", 
"time=\"2019-04-25T11:27:50+02:00\" level=info msg=\"[reconcile] Reconciling cluster state\"", 
"time=\"2019-04-25T11:27:50+02:00\" level=info msg=\"[reconcile] Check etcd hosts to be deleted\"", 
"time=\"2019-04-25T11:27:50+02:00\" level=warning msg=\"Etcd plane nodes are replaced. Stopping provisioning. Please restore your cluster from backup.\"", 
"time=\"2019-04-25T11:27:50+02:00\" level=fatal msg=\"Failed to reconcile etcd plane: Etcd plane nodes are replaced. Stopping provisioning. Please restore your cluster from backup.\""
rancher/rke

Answer questions mtparet

Tested with -rc2 but I still get the same issue. (Details in the support ticket)

useful!

Related questions

Failed to get /health for host - remote error: tls: bad certificate hot 2
Failed to rotate expired certificates on an RKE cluster: unable to reach api server to fetch CA hot 2
Error response from daemon: chown /etc/resolv.conf: operation not permitted hot 1
Pods can't reach networks outside of node hot 1
rke 0.1.17 Can't initiate NewClient: protocol not available hot 1
Calico node failed to start after upgrading the cluster hot 1
Job rke-network-plugin-deploy-job never completes (virtualbox) hot 1
rke up --local fails to deploy successfully hot 1
Job rke-network-plugin-deploy-job never completes (virtualbox) hot 1
go panic on intial rke up hot 1
Unable to update cluster "crypto/rsa: verification error" hot 1
Calico node failed to start after upgrading the cluster hot 1
pods in "CrashLoopBackOff" status after restoring from backup hot 1
[SOLVED] Failed to apply the ServiceAccount needed for job execution: Post https://10.102.X.X:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings: Forbidden hot 1
Failed to get /health for host - remote error: tls: bad certificate hot 1
Github User Rank List