profile
viewpoint

Ask questionsUnable to update cluster "crypto/rsa: verification error"

I have a RKE cluster that i am trying to update the version of kubernetes that it is running. Currently its at 1.11.1 im trying to update it to 1.11.5. I had previously not defined the version of kubernetes that it was using and now after making the change to specifically use the 1.11.5 version I am getting the error mentioned below. I am using all the defaults here and am having the cluster generate the certs it self so not sure why it would be causing this issue on its own. I first tried the upgrade with an older version of RKE with the same result as running with this newer version(v0.1.14). I do believe the nodes were isolated from one another for some time due a network change however they should now be able to pass traffic between themselves.

RKE version: v0.1.14 Docker version: (docker version,docker info preferred)

docker version
Client:
 Version:      1.13.1
 API version:  1.26
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 06:38:28 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.13.1
 API version:  1.26 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   092cba3
 Built:        Wed Feb  8 06:38:28 2017
 OS/Arch:      linux/amd64
 Experimental: false

Operating system and kernel: (cat /etc/os-release, uname -r preferred)

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO) vSphere cluster.yml file:

bastion_host:
   address: 172.10.10.10 
   user: rancher
   port: 22
   ssh_key_path: ~/.ssh/id_rsa

nodes:
  - address: 172.10.10.11
    user: rancher
    role: [controlplane,etcd,worker]
    ssh_key_path: ~/.ssh/id_rsa
  - address: 172.10.10.12
    user: rancher
    role: [controlplane,etcd,worker]
    ssh_key_path: ~/.ssh/id_rsa
  - address: 172.10.10.13
    user: rancher
    role: [controlplane,etcd,worker]
    ssh_key_path: ~/.ssh/id_rsa

system_images:
    kubernetes: rancher/hyperkube:v1.11.5-rancher1

services:
  etcd:
    snapshot: true # enables recurring etcd snapshots
    creation: 6h0s # time increment between snapshots
    retention: 24h # time increment before snapshot purge

Steps to Reproduce: Run rke up --config rancher-cluster.yml

Results:

time="2019-01-08T21:02:39Z" level=fatal msg="[controlPlane] Failed to bring up Control Plane: Failed to verify healthcheck: Service [kube-apiserver] is not healthy on host [172.10.10.11]. Response code: [401], response body: {\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"Unauthorized\",\"reason\":\"Unauthorized\",\"code\":401}\n, log: E0108 21:02:34.055359       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"kube-ca\")]"

docker logs --tail=all etcd

2019-01-08 19:14:08.097697 I | etcdmain: etcd Version: 3.2.18
2019-01-08 19:14:08.097786 I | etcdmain: Git SHA: eddf599c6
2019-01-08 19:14:08.097793 I | etcdmain: Go Version: go1.8.7
2019-01-08 19:14:08.097798 I | etcdmain: Go OS/Arch: linux/amd64
2019-01-08 19:14:08.097807 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
2019-01-08 19:14:08.097870 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2019-01-08 19:14:08.097924 I | embed: peerTLS: cert = /etc/kubernetes/ssl/kube-etcd-172-10-10-11.pem, key = /etc/kubernetes/ssl/kube-etcd-172-10-10-11-key.pem, ca = , trusted-ca = /etc/kubernetes/ssl/kube-ca.pem, client-cert-auth = true
2019-01-08 19:14:08.124953 I | embed: listening for peers on https://0.0.0.0:2380
2019-01-08 19:14:08.125127 I | embed: listening for client requests on 0.0.0.0:2379
2019-01-08 19:14:08.873460 I | etcdserver: recovered store from snapshot at index 32800328
2019-01-08 19:14:08.878738 I | mvcc: restore compact to 23672433
2019-01-08 19:14:08.912714 I | etcdserver: name = etcd-172.10.10.11
2019-01-08 19:14:08.912734 I | etcdserver: data dir = /var/lib/rancher/etcd/
2019-01-08 19:14:08.912743 I | etcdserver: member dir = /var/lib/rancher/etcd/member
2019-01-08 19:14:08.912749 I | etcdserver: heartbeat = 500ms
2019-01-08 19:14:08.912754 I | etcdserver: election = 5000ms
2019-01-08 19:14:08.912761 I | etcdserver: snapshot count = 100000
2019-01-08 19:14:08.912776 I | etcdserver: advertise client URLs = https://172.10.10.11:2379,https://172.10.10.11:4001
2019-01-08 19:14:09.326979 I | etcdserver: restarting member c03ba5f4ddaf4890 in cluster c4a56d558e3dd6eb at commit index 32814534
2019-01-08 19:14:09.332003 I | raft: c03ba5f4ddaf4890 became follower at term 116
2019-01-08 19:14:09.332049 I | raft: newRaft c03ba5f4ddaf4890 [peers: [68ac97d90f648258,c03ba5f4ddaf4890,d7b67dc6eae5e603], term: 116, commit: 32814534, applied: 32800328, lastindex: 32814535, lastterm: 116]
2019-01-08 19:14:09.332380 I | etcdserver/api: enabled capabilities for version 3.2
2019-01-08 19:14:09.332410 I | etcdserver/membership: added member c03ba5f4ddaf4890 [https://172.10.10.11:2380] to cluster c4a56d558e3dd6eb from store
2019-01-08 19:14:09.332427 I | etcdserver/membership: added member d7b67dc6eae5e603 [https://172.10.10.13:2380] to cluster c4a56d558e3dd6eb from store
2019-01-08 19:14:09.332441 I | etcdserver/membership: added member 68ac97d90f648258 [https://172.10.10.12:2380] to cluster c4a56d558e3dd6eb from store
2019-01-08 19:14:09.332455 I | etcdserver/membership: set the cluster version to 3.2 from store
2019-01-08 19:14:09.340529 I | mvcc: restore compact to 23672433
2019-01-08 19:14:09.350973 W | auth: simple token is not cryptographically signed
2019-01-08 19:14:09.357651 I | rafthttp: starting peer 68ac97d90f648258...
2019-01-08 19:14:09.357697 I | rafthttp: started HTTP pipelining with peer 68ac97d90f648258
2019-01-08 19:14:09.358450 I | rafthttp: started peer 68ac97d90f648258
2019-01-08 19:14:09.358849 I | rafthttp: added peer 68ac97d90f648258
2019-01-08 19:14:09.358872 I | rafthttp: starting peer d7b67dc6eae5e603...
2019-01-08 19:14:09.358887 I | rafthttp: started HTTP pipelining with peer d7b67dc6eae5e603
2019-01-08 19:14:09.359282 I | rafthttp: started streaming with peer 68ac97d90f648258 (stream Message reader)
2019-01-08 19:14:09.365082 I | rafthttp: started streaming with peer 68ac97d90f648258 (writer)
2019-01-08 19:14:09.365143 I | rafthttp: started streaming with peer 68ac97d90f648258 (stream MsgApp v2 reader)
2019-01-08 19:14:09.366188 I | rafthttp: started streaming with peer d7b67dc6eae5e603 (writer)
2019-01-08 19:14:09.366202 I | rafthttp: started streaming with peer 68ac97d90f648258 (writer)
2019-01-08 19:14:09.366210 I | rafthttp: started streaming with peer d7b67dc6eae5e603 (writer)
2019-01-08 19:14:09.366400 I | rafthttp: started peer d7b67dc6eae5e603
2019-01-08 19:14:09.366419 I | rafthttp: added peer d7b67dc6eae5e603
2019-01-08 19:14:09.366449 I | etcdserver: starting server... [version: 3.2.18, cluster version: 3.2]
2019-01-08 19:14:09.366536 I | rafthttp: started streaming with peer d7b67dc6eae5e603 (stream MsgApp v2 reader)
2019-01-08 19:14:09.366717 I | rafthttp: started streaming with peer d7b67dc6eae5e603 (stream Message reader)
2019-01-08 19:14:09.367317 I | embed: ClientTLS: cert = /etc/kubernetes/ssl/kube-etcd-172-10-10-11.pem, key = /etc/kubernetes/ssl/kube-etcd-172-10-10-11-key.pem, ca = , trusted-ca = /etc/kubernetes/ssl/kube-ca.pem, client-cert-auth = true
2019-01-08 19:14:09.369279 I | rafthttp: peer 68ac97d90f648258 became active
2019-01-08 19:14:09.369293 I | rafthttp: established a TCP streaming connection with peer 68ac97d90f648258 (stream MsgApp v2 writer)
2019-01-08 19:14:09.369430 I | rafthttp: established a TCP streaming connection with peer 68ac97d90f648258 (stream Message writer)
2019-01-08 19:14:09.370027 I | rafthttp: peer d7b67dc6eae5e603 became active
2019-01-08 19:14:09.370040 I | rafthttp: established a TCP streaming connection with peer d7b67dc6eae5e603 (stream MsgApp v2 writer)
2019-01-08 19:14:09.370174 I | rafthttp: established a TCP streaming connection with peer d7b67dc6eae5e603 (stream Message writer)
2019-01-08 19:14:09.390466 I | raft: raft.node: c03ba5f4ddaf4890 elected leader d7b67dc6eae5e603 at term 116
2019-01-08 19:14:09.394397 I | rafthttp: established a TCP streaming connection with peer 68ac97d90f648258 (stream Message reader)
2019-01-08 19:14:09.394521 I | rafthttp: established a TCP streaming connection with peer 68ac97d90f648258 (stream MsgApp v2 reader)
2019-01-08 19:14:09.394680 I | rafthttp: established a TCP streaming connection with peer d7b67dc6eae5e603 (stream MsgApp v2 reader)
2019-01-08 19:14:09.398591 I | rafthttp: established a TCP streaming connection with peer d7b67dc6eae5e603 (stream Message reader)
2019-01-08 19:14:09.417838 I | etcdserver: published {Name:etcd-172.10.10.11 ClientURLs:[https://172.10.10.11:2379 https://172.10.10.11:4001]} to cluster c4a56d558e3dd6eb
2019-01-08 19:14:09.417913 I | embed: ready to serve client requests
2019-01-08 19:14:09.418223 I | embed: serving client requests on [::]:2379
2019-01-08 19:14:09.418556 I | etcdserver: c03ba5f4ddaf4890 initialzed peer connection; fast-forwarding 8 ticks (election ticks 10) with 2 active peer(s)
2019-01-08 19:15:28.665684 I | etcdmain: rejected connection from "172.10.10.12:21076" (error "EOF", ServerName "")
2019-01-08 19:15:28.669695 I | etcdmain: rejected connection from "172.10.10.13:32179" (error "EOF", ServerName "")
2019-01-08 19:15:28.773686 I | etcdmain: rejected connection from "172.10.10.11:29672" (error "EOF", ServerName "")
2019-01-08 19:17:03.200232 I | mvcc: store.index: compact 23672973
2019-01-08 19:17:03.203427 I | mvcc: finished scheduled compaction at 23672973 (took 2.24685ms)
2019-01-08 19:17:41.834761 I | etcdmain: rejected connection from "172.10.10.12:13310" (error "EOF", ServerName "")
2019-01-08 19:17:41.872416 I | etcdmain: rejected connection from "172.10.10.13:2356" (error "EOF", ServerName "")
2019-01-08 19:17:41.922684 I | etcdmain: rejected connection from "172.10.10.11:33277" (error "EOF", ServerName "")
2019-01-08 19:18:10.423424 W | rafthttp: lost the TCP streaming connection with peer 68ac97d90f648258 (stream MsgApp v2 reader)
2019-01-08 19:18:10.423464 E | rafthttp: failed to read 68ac97d90f648258 on stream MsgApp v2 (unexpected EOF)
2019-01-08 19:18:10.423473 I | rafthttp: peer 68ac97d90f648258 became inactive
2019-01-08 19:18:10.428277 W | rafthttp: lost the TCP streaming connection with peer 68ac97d90f648258 (stream Message reader)
2019-01-08 19:18:12.027179 I | rafthttp: peer 68ac97d90f648258 became active
2019-01-08 19:18:12.027205 I | rafthttp: established a TCP streaming connection with peer 68ac97d90f648258 (stream MsgApp v2 reader)
2019-01-08 19:18:12.027638 I | rafthttp: established a TCP streaming connection with peer 68ac97d90f648258 (stream Message reader)
2019-01-08 19:18:12.028760 W | rafthttp: closed an existing TCP streaming connection with peer 68ac97d90f648258 (stream Message writer)
2019-01-08 19:18:12.028779 I | rafthttp: established a TCP streaming connection with peer 68ac97d90f648258 (stream Message writer)
2019-01-08 19:18:12.041325 W | rafthttp: closed an existing TCP streaming connection with peer 68ac97d90f648258 (stream MsgApp v2 writer)
2019-01-08 19:18:12.041341 I | rafthttp: established a TCP streaming connection with peer 68ac97d90f648258 (stream MsgApp v2 writer)
2019-01-08 19:22:03.209796 I | mvcc: store.index: compact 23673513
2019-01-08 19:22:03.213061 I | mvcc: finished scheduled compaction at 23673513 (took 2.360729ms)
2019-01-08 19:24:05.199646 I | etcdmain: rejected connection from "172.10.10.12:8882" (error "EOF", ServerName "")
2019-01-08 19:24:05.200142 I | etcdmain: rejected connection from "172.10.10.13:15092" (error "EOF", ServerName "")
2019-01-08 19:24:05.393076 I | etcdmain: rejected connection from "172.10.10.11:19861" (error "EOF", ServerName "")
2019-01-08 19:25:42.933561 I | etcdmain: rejected connection from "172.10.10.12:30158" (error "EOF", ServerName "")
2019-01-08 19:25:42.960912 I | etcdmain: rejected connection from "172.10.10.13:24532" (error "EOF", ServerName "")
2019-01-08 19:25:42.993838 I | etcdmain: rejected connection from "172.10.10.11:6444" (error "EOF", ServerName "")
2019-01-08 19:26:16.944693 W | rafthttp: lost the TCP streaming connection with peer d7b67dc6eae5e603 (stream Message reader)
2019-01-08 19:26:16.944755 E | rafthttp: failed to read d7b67dc6eae5e603 on stream Message (unexpected EOF)
2019-01-08 19:26:16.944777 I | rafthttp: peer d7b67dc6eae5e603 became inactive
2019-01-08 19:26:16.945218 W | rafthttp: lost the TCP streaming connection with peer d7b67dc6eae5e603 (stream MsgApp v2 reader)
2019-01-08 19:26:17.755087 W | rafthttp: lost the TCP streaming connection with peer d7b67dc6eae5e603 (stream Message writer)
2019-01-08 19:26:18.024599 I | rafthttp: peer d7b67dc6eae5e603 became active
2019-01-08 19:26:18.024627 I | rafthttp: established a TCP streaming connection with peer d7b67dc6eae5e603 (stream Message reader)
2019-01-08 19:26:18.027689 I | rafthttp: established a TCP streaming connection with peer d7b67dc6eae5e603 (stream MsgApp v2 reader)
2019-01-08 19:26:18.040454 I | rafthttp: established a TCP streaming connection with peer d7b67dc6eae5e603 (stream Message writer)
2019-01-08 19:26:18.051424 W | rafthttp: closed an existing TCP streaming connection with peer d7b67dc6eae5e603 (stream MsgApp v2 writer)
2019-01-08 19:26:18.051456 I | rafthttp: established a TCP streaming connection with peer d7b67dc6eae5e603 (stream MsgApp v2 writer)
2019-01-08 19:26:20.450653 I | raft: c03ba5f4ddaf4890 [logterm: 116, index: 32816440, vote: d7b67dc6eae5e603] ignored MsgVote from d7b67dc6eae5e603 [logterm: 116, index: 32816440] at term 116: lease is not expired (remaining ticks: 3)
2019-01-08 19:26:22.956384 I | raft: c03ba5f4ddaf4890 [term: 116] received a MsgVote message with higher term from 68ac97d90f648258 [term: 117]
2019-01-08 19:26:22.956447 I | raft: c03ba5f4ddaf4890 became follower at term 117
2019-01-08 19:26:22.956465 I | raft: c03ba5f4ddaf4890 [logterm: 116, index: 32816440, vote: 0] cast MsgVote for 68ac97d90f648258 [logterm: 116, index: 32816440] at term 117
2019-01-08 19:26:22.956477 I | raft: raft.node: c03ba5f4ddaf4890 lost leader d7b67dc6eae5e603 at term 117
2019-01-08 19:26:22.957695 I | raft: raft.node: c03ba5f4ddaf4890 elected leader 68ac97d90f648258 at term 117
2019-01-08 19:26:32.015477 W | etcdserver: timed out waiting for read index response
2019-01-08 19:27:03.222011 I | mvcc: store.index: compact 23674052
2019-01-08 19:27:03.224909 I | mvcc: finished scheduled compaction at 23674052 (took 1.957373ms)
2019-01-08 19:32:03.232492 I | mvcc: store.index: compact 23674575
2019-01-08 19:32:03.235896 I | mvcc: finished scheduled compaction at 23674575 (took 2.415744ms)
2019-01-08 19:37:03.240747 I | mvcc: store.index: compact 23675113
2019-01-08 19:37:03.243892 I | mvcc: finished scheduled compaction at 23675113 (took 2.23588ms)
2019-01-08 19:42:03.247941 I | mvcc: store.index: compact 23675651
2019-01-08 19:42:03.251171 I | mvcc: finished scheduled compaction at 23675651 (took 2.309244ms)
2019-01-08 19:47:03.259958 I | mvcc: store.index: compact 23676189
2019-01-08 19:47:03.263690 I | mvcc: finished scheduled compaction at 23676189 (took 2.702328ms)
2019-01-08 19:52:03.269789 I | mvcc: store.index: compact 23676729
2019-01-08 19:52:03.305943 I | mvcc: finished scheduled compaction at 23676729 (took 2.455472ms)
2019-01-08 19:57:03.276961 I | mvcc: store.index: compact 23677268
2019-01-08 19:57:03.280136 I | mvcc: finished scheduled compaction at 23677268 (took 2.267449ms)
2019-01-08 20:02:03.286317 I | mvcc: store.index: compact 23677809
2019-01-08 20:02:03.290409 I | mvcc: finished scheduled compaction at 23677809 (took 2.899305ms)
2019-01-08 20:07:03.293141 I | mvcc: store.index: compact 23678348
2019-01-08 20:07:03.296426 I | mvcc: finished scheduled compaction at 23678348 (took 2.40394ms)
2019-01-08 20:12:03.301742 I | mvcc: store.index: compact 23678887
2019-01-08 20:12:03.305009 I | mvcc: finished scheduled compaction at 23678887 (took 2.283419ms)
2019-01-08 20:16:17.548715 I | wal: segmented wal file /var/lib/rancher/etcd/member/wal/0000000000000241-0000000001f4dbaf.wal is created
2019-01-08 20:16:39.420702 I | pkg/fileutil: purged file /var/lib/rancher/etcd/member/wal/000000000000023c-0000000001f02961.wal successfully
2019-01-08 20:17:03.310366 I | mvcc: store.index: compact 23679428
2019-01-08 20:17:03.313553 I | mvcc: finished scheduled compaction at 23679428 (took 2.340848ms)
2019-01-08 20:22:03.319849 I | mvcc: store.index: compact 23679967
2019-01-08 20:22:03.323758 I | mvcc: finished scheduled compaction at 23679967 (took 2.946093ms)
2019-01-08 20:23:56.128697 I | etcdmain: rejected connection from "172.10.10.11:1534" (error "EOF", ServerName "")
2019-01-08 20:23:56.138513 I | etcdmain: rejected connection from "172.10.10.12:22484" (error "EOF", ServerName "")
2019-01-08 20:23:56.158076 I | etcdmain: rejected connection from "172.10.10.13:1300" (error "EOF", ServerName "")
2019-01-08 20:25:09.414113 I | etcdmain: rejected connection from "172.10.10.12:11248" (error "EOF", ServerName "")
2019-01-08 20:25:09.417489 I | etcdmain: rejected connection from "172.10.10.13:18558" (error "EOF", ServerName "")
2019-01-08 20:25:09.436928 I | etcdmain: rejected connection from "172.10.10.11:5773" (error "EOF", ServerName "")
2019-01-08 20:27:03.327368 I | mvcc: store.index: compact 23680508
2019-01-08 20:27:03.331061 I | mvcc: finished scheduled compaction at 23680508 (took 2.670325ms)
2019-01-08 20:28:56.169393 I | etcdmain: rejected connection from "172.10.10.13:9304" (error "EOF", ServerName "")
2019-01-08 20:28:56.183384 I | etcdmain: rejected connection from "172.10.10.11:28361" (error "EOF", ServerName "")
2019-01-08 20:28:56.194397 I | etcdmain: rejected connection from "172.10.10.12:24295" (error "EOF", ServerName "")
2019-01-08 20:32:33.626736 I | etcdmain: rejected connection from "172.10.10.13:32910" (error "EOF", ServerName "")
2019-01-08 20:32:33.636988 I | etcdmain: rejected connection from "172.10.10.11:12445" (error "EOF", ServerName "")
2019-01-08 20:32:33.644062 I | etcdmain: rejected connection from "172.10.10.12:31820" (error "EOF", ServerName "")
2019-01-08 20:36:27.725461 I | etcdmain: rejected connection from "172.10.10.12:8613" (error "EOF", ServerName "")
2019-01-08 20:36:27.747593 I | etcdmain: rejected connection from "172.10.10.13:23179" (error "EOF", ServerName "")
2019-01-08 20:36:27.755228 I | etcdmain: rejected connection from "172.10.10.11:3293" (error "EOF", ServerName "")
2019-01-08 20:39:31.125783 I | mvcc: store.index: compact 23681846
2019-01-08 20:39:31.540563 I | mvcc: finished scheduled compaction at 23681846 (took 369.541568ms)
2019-01-08 20:39:31.543826 W | etcdserver: apply entries took too long [228.477147ms for 1 entries]
2019-01-08 20:39:31.543847 W | etcdserver: avoid queries with large range/delete range!
2019-01-08 20:40:11.863223 I | etcdmain: rejected connection from "172.10.10.13:17229" (error "EOF", ServerName "")
2019-01-08 20:40:11.880415 I | etcdmain: rejected connection from "172.10.10.12:32676" (error "EOF", ServerName "")
2019-01-08 20:40:11.915687 I | etcdmain: rejected connection from "172.10.10.11:18351" (error "EOF", ServerName "")
2019-01-08 20:44:31.138817 I | mvcc: store.index: compact 23682426
2019-01-08 20:44:31.142335 I | mvcc: finished scheduled compaction at 23682426 (took 2.583467ms)
2019-01-08 20:49:31.150687 I | mvcc: store.index: compact 23682965
2019-01-08 20:49:31.154064 I | mvcc: finished scheduled compaction at 23682965 (took 2.39469ms)
2019-01-08 20:54:31.162956 I | mvcc: store.index: compact 23683503
2019-01-08 20:54:31.166449 I | mvcc: finished scheduled compaction at 23683503 (took 2.505848ms)
2019-01-08 20:59:31.191949 I | mvcc: store.index: compact 23684043
2019-01-08 20:59:31.196749 I | mvcc: finished scheduled compaction at 23684043 (took 3.475937ms)
2019-01-08 21:01:14.196440 I | etcdmain: rejected connection from "172.10.10.12:28255" (error "EOF", ServerName "")
2019-01-08 21:01:14.228941 I | etcdmain: rejected connection from "172.10.10.13:19677" (error "EOF", ServerName "")
2019-01-08 21:01:14.250825 I | etcdmain: rejected connection from "172.10.10.11:23632" (error "EOF", ServerName "")
2019-01-08 21:04:31.231580 I | mvcc: store.index: compact 23684583
2019-01-08 21:04:31.235768 I | mvcc: finished scheduled compaction at 23684583 (took 3.202065ms)
2019-01-08 21:09:31.225815 I | mvcc: store.index: compact 23685122
2019-01-08 21:09:31.229120 I | mvcc: finished scheduled compaction at 23685122 (took 2.400029ms)
2019-01-08 21:14:31.240572 I | mvcc: store.index: compact 23685662
2019-01-08 21:14:31.243900 I | mvcc: finished scheduled compaction at 23685662 (took 2.37271ms)
2019-01-08 21:19:31.251580 I | mvcc: store.index: compact 23686202
2019-01-08 21:19:31.257180 I | mvcc: finished scheduled compaction at 23686202 (took 4.111563ms)

docker logs --tail=all kube-apiserver

+ echo kube-apiserver --insecure-bind-address=127.0.0.1 --requestheader-allowed-names=kube-apiserver-proxy-client --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --storage-backend=etcd3 --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-account-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --bind-address=0.0.0.0 --requestheader-group-headers=X-Remote-Group --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-prefix=/registry --allow-privileged=true --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --authorization-mode=Node,RBAC --secure-port=6443 --service-cluster-ip-range=10.43.0.0/16 --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --insecure-port=0 --requestheader-extra-headers-prefix=X-Remote-Extra- --cloud-provider= --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --requestheader-username-headers=X-Remote-User --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --etcd-servers=https://172.10.10.11:2379,https://172.10.10.12:2379,https://172.10.10.13:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem
+ grep -q cloud-provider=azure
+ '[' kube-apiserver = kubelet ']'
+ exec kube-apiserver --insecure-bind-address=127.0.0.1 --requestheader-allowed-names=kube-apiserver-proxy-client --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --storage-backend=etcd3 --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-account-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --bind-address=0.0.0.0 --requestheader-group-headers=X-Remote-Group --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-prefix=/registry --allow-privileged=true --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --authorization-mode=Node,RBAC --secure-port=6443 --service-cluster-ip-range=10.43.0.0/16 --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --insecure-port=0 --requestheader-extra-headers-prefix=X-Remote-Extra- --cloud-provider= --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --requestheader-username-headers=X-Remote-User --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --etcd-servers=https://172.10.10.11:2379,https://172.10.10.12:2379,https://172.10.10.13:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem
Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version.
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
I0108 20:29:30.892041       1 server.go:703] external host was not specified, using 172.10.10.11
I0108 20:29:30.892219       1 server.go:145] Version: v1.11.5
I0108 20:29:32.237368       1 plugins.go:158] Loaded 7 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0108 20:29:32.237402       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0108 20:29:32.238129       1 plugins.go:158] Loaded 7 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0108 20:29:32.238143       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0108 20:29:32.367120       1 master.go:234] Using reconciler: lease
W0108 20:29:34.640800       1 genericapiserver.go:319] Skipping API batch/v2alpha1 because it has no resources.
W0108 20:29:35.191776       1 genericapiserver.go:319] Skipping API rbac.authorization.k8s.io/v1alpha1 because it has no resources.
W0108 20:29:35.202565       1 genericapiserver.go:319] Skipping API scheduling.k8s.io/v1alpha1 because it has no resources.
W0108 20:29:35.235199       1 genericapiserver.go:319] Skipping API storage.k8s.io/v1alpha1 because it has no resources.
W0108 20:29:36.145623       1 genericapiserver.go:319] Skipping API admissionregistration.k8s.io/v1alpha1 because it has no resources.
[restful] 2019/01/08 20:29:36 log.go:33: [restful/swagger] listing is available at https://172.10.10.11:6443/swaggerapi
[restful] 2019/01/08 20:29:36 log.go:33: [restful/swagger] https://172.10.10.11:6443/swaggerui/ is mapped to folder /swagger-ui/
[restful] 2019/01/08 20:29:38 log.go:33: [restful/swagger] listing is available at https://172.10.10.11:6443/swaggerapi
[restful] 2019/01/08 20:29:38 log.go:33: [restful/swagger] https://172.10.10.11:6443/swaggerui/ is mapped to folder /swagger-ui/
I0108 20:29:38.288539       1 plugins.go:158] Loaded 7 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,Priority,DefaultTolerationSeconds,DefaultStorageClass,MutatingAdmissionWebhook.
I0108 20:29:38.288562       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
I0108 20:29:44.166264       1 serve.go:116] Serving securely on [::]:6443
I0108 20:29:44.166617       1 apiservice_controller.go:90] Starting APIServiceRegistrationController
I0108 20:29:44.166636       1 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
I0108 20:29:44.167071       1 controller.go:84] Starting OpenAPI AggregationController
I0108 20:29:44.167114       1 autoregister_controller.go:136] Starting autoregister controller
I0108 20:29:44.167123       1 cache.go:32] Waiting for caches to sync for autoregister controller
I0108 20:29:44.171085       1 crdregistration_controller.go:112] Starting crd-autoregister controller
I0108 20:29:44.171119       1 controller_utils.go:1025] Waiting for caches to sync for crd-autoregister controller
I0108 20:29:44.172374       1 available_controller.go:278] Starting AvailableConditionController
I0108 20:29:44.172389       1 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
I0108 20:29:44.172447       1 crd_finalizer.go:242] Starting CRDFinalizer
I0108 20:29:44.177082       1 log.go:172] http: TLS handshake error from 172.10.10.12:16932: EOF
I0108 20:29:44.180975       1 log.go:172] http: TLS handshake error from 172.10.10.12:17138: EOF
I0108 20:29:44.181067       1 log.go:172] http: TLS handshake error from 10.42.1.6:37228: EOF
I0108 20:29:44.184942       1 log.go:172] http: TLS handshake error from 10.42.1.6:37230: EOF
I0108 20:29:44.185577       1 log.go:172] http: TLS handshake error from 172.10.10.12:17142: EOF
I0108 20:29:44.188899       1 log.go:172] http: TLS handshake error from 172.10.10.11:42854: EOF
I0108 20:29:44.189765       1 log.go:172] http: TLS handshake error from 172.10.10.13:47110: EOF
I0108 20:29:44.192886       1 log.go:172] http: TLS handshake error from 172.10.10.11:42856: EOF
I0108 20:29:44.194168       1 log.go:172] http: TLS handshake error from 172.10.10.11:42952: EOF
I0108 20:29:44.196732       1 log.go:172] http: TLS handshake error from 172.10.10.11:42858: EOF
I0108 20:29:44.200554       1 log.go:172] http: TLS handshake error from 172.10.10.11:42860: EOF
I0108 20:29:44.212466       1 log.go:172] http: TLS handshake error from 172.10.10.11:42862: EOF
I0108 20:29:44.222151       1 log.go:172] http: TLS handshake error from 172.10.10.11:42864: EOF
I0108 20:29:44.226010       1 log.go:172] http: TLS handshake error from 172.10.10.11:42866: EOF
I0108 20:29:44.229863       1 log.go:172] http: TLS handshake error from 172.10.10.11:42868: EOF
I0108 20:29:44.233774       1 log.go:172] http: TLS handshake error from 10.42.1.6:37248: EOF
I0108 20:29:44.416513       1 log.go:172] http: TLS handshake error from 10.42.1.6:37252: EOF
I0108 20:29:44.422894       1 log.go:172] http: TLS handshake error from 172.10.10.11:42876: EOF
I0108 20:29:44.430718       1 log.go:172] http: TLS handshake error from 172.10.10.11:42878: EOF
I0108 20:29:44.435850       1 log.go:172] http: TLS handshake error from 172.10.10.11:42880: EOF
I0108 20:29:44.439751       1 log.go:172] http: TLS handshake error from 172.10.10.11:42882: EOF
I0108 20:29:44.443862       1 log.go:172] http: TLS handshake error from 172.10.10.11:42884: EOF
I0108 20:29:44.451824       1 log.go:172] http: TLS handshake error from 10.42.1.6:37266: EOF
I0108 20:29:44.455938       1 log.go:172] http: TLS handshake error from 10.42.1.6:37268: EOF
I0108 20:29:44.459920       1 log.go:172] http: TLS handshake error from 127.0.0.1:47642: EOF
I0108 20:29:44.463811       1 log.go:172] http: TLS handshake error from 172.10.10.12:16972: EOF
I0108 20:29:44.471551       1 log.go:172] http: TLS handshake error from 127.0.0.1:47648: EOF
I0108 20:29:44.475504       1 log.go:172] http: TLS handshake error from 172.10.10.11:42902: EOF
I0108 20:29:44.479393       1 log.go:172] http: TLS handshake error from 127.0.0.1:47652: EOF
I0108 20:29:44.483274       1 log.go:172] http: TLS handshake error from 127.0.0.1:47654: EOF
I0108 20:29:44.487204       1 log.go:172] http: TLS handshake error from 127.0.0.1:47656: EOF
I0108 20:29:44.491119       1 log.go:172] http: TLS handshake error from 172.10.10.11:42910: EOF
I0108 20:29:44.495045       1 log.go:172] http: TLS handshake error from 127.0.0.1:47660: EOF
I0108 20:29:44.498974       1 log.go:172] http: TLS handshake error from 127.0.0.1:47662: EOF
I0108 20:29:44.502913       1 log.go:172] http: TLS handshake error from 127.0.0.1:47664: EOF
I0108 20:29:44.506959       1 log.go:172] http: TLS handshake error from 127.0.0.1:47666: EOF
I0108 20:29:44.829814       1 log.go:172] http: TLS handshake error from 127.0.0.1:47668: EOF
I0108 20:29:44.835794       1 log.go:172] http: TLS handshake error from 127.0.0.1:47670: EOF
I0108 20:29:44.840011       1 log.go:172] http: TLS handshake error from 127.0.0.1:47672: EOF
I0108 20:29:44.853601       1 log.go:172] http: TLS handshake error from 127.0.0.1:47674: EOF
I0108 20:29:44.863038       1 log.go:172] http: TLS handshake error from 127.0.0.1:47676: EOF
I0108 20:29:44.873676       1 log.go:172] http: TLS handshake error from 127.0.0.1:47678: EOF
I0108 20:29:44.877849       1 log.go:172] http: TLS handshake error from 127.0.0.1:47680: EOF
I0108 20:29:44.883039       1 log.go:172] http: TLS handshake error from 127.0.0.1:47682: EOF
I0108 20:29:44.887049       1 log.go:172] http: TLS handshake error from 172.10.10.12:17062: EOF
I0108 20:29:44.897349       1 log.go:172] http: TLS handshake error from 172.10.10.11:42936: EOF
I0108 20:29:44.909639       1 log.go:172] http: TLS handshake error from 172.10.10.12:17116: EOF
I0108 20:29:44.913648       1 log.go:172] http: TLS handshake error from 172.10.10.12:17118: EOF
I0108 20:29:44.917585       1 log.go:172] http: TLS handshake error from 172.10.10.12:17120: EOF
I0108 20:29:44.929122       1 log.go:172] http: TLS handshake error from 172.10.10.12:17124: EOF
I0108 20:29:44.933872       1 log.go:172] http: TLS handshake error from 172.10.10.12:17126: EOF
I0108 20:29:44.940385       1 log.go:172] http: TLS handshake error from 172.10.10.12:17128: EOF
I0108 20:29:44.944833       1 log.go:172] http: TLS handshake error from 172.10.10.12:17130: EOF
I0108 20:29:44.949894       1 log.go:172] http: TLS handshake error from 172.10.10.12:17132: EOF
I0108 20:29:44.953896       1 log.go:172] http: TLS handshake error from 172.10.10.12:17136: EOF
I0108 20:29:44.953967       1 customresource_discovery_controller.go:199] Starting DiscoveryController
I0108 20:29:44.953996       1 naming_controller.go:284] Starting NamingConditionController
I0108 20:29:44.954016       1 establishing_controller.go:73] Starting EstablishingController
I0108 20:29:44.971167       1 log.go:172] http: TLS handshake error from 10.42.1.6:37250: EOF
I0108 20:29:45.053549       1 log.go:172] http: TLS handshake error from 172.10.10.12:17424: EOF
I0108 20:29:45.059258       1 log.go:172] http: TLS handshake error from 172.10.10.12:17122: EOF
E0108 20:29:45.075703       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
I0108 20:29:45.709163       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
I0108 20:29:45.748804       1 cache.go:39] Caches are synced for autoregister controller
I0108 20:29:45.749110       1 controller_utils.go:1032] Caches are synced for crd-autoregister controller
I0108 20:29:45.749638       1 cache.go:39] Caches are synced for AvailableConditionController controller
E0108 20:29:45.762819       1 memcache.go:147] couldn't get resource list for metrics.k8s.io/v1beta1: the server could not find the requested resource
I0108 20:29:45.911089       1 trace.go:76] Trace[538251335]: "Get /apis/scheduling.k8s.io/v1beta1/priorityclasses/system-cluster-critical" (started: 2019-01-08 20:29:45.321620857 +0000 UTC m=+14.792926218) (total time: 589.420739ms):
Trace[538251335]: [589.35128ms] [589.349526ms] About to write a response
I0108 20:29:45.911789       1 trace.go:76] Trace[1034464253]: "Get /api/v1/namespaces/kube-system" (started: 2019-01-08 20:29:45.256669923 +0000 UTC m=+14.727975304) (total time: 655.101681ms):
Trace[1034464253]: [655.051242ms] [655.049543ms] About to write a response
I0108 20:29:45.912064       1 trace.go:76] Trace[1112115680]: "Get /api/v1/namespaces/ingress-nginx/secrets/nginx-ingress-serviceaccount-token-254ms" (started: 2019-01-08 20:29:45.258003974 +0000 UTC m=+14.729309335) (total time: 654.03875ms):
Trace[1112115680]: [654.000392ms] [653.998668ms] About to write a response
I0108 20:29:45.912644       1 trace.go:76] Trace[348537002]: "Get /api/v1/namespaces/ingress-nginx/secrets/nginx-ingress-serviceaccount-token-254ms" (started: 2019-01-08 20:29:45.257860767 +0000 UTC m=+14.729166127) (total time: 654.758021ms):
Trace[348537002]: [654.368299ms] [654.366796ms] About to write a response
I0108 20:29:45.912871       1 trace.go:76] Trace[277573747]: "Get /api/v1/namespaces/ingress-nginx/secrets/nginx-ingress-serviceaccount-token-254ms" (started: 2019-01-08 20:29:45.257707499 +0000 UTC m=+14.729012875) (total time: 655.146645ms):
Trace[277573747]: [655.098186ms] [655.096464ms] About to write a response
I0108 20:29:45.913101       1 trace.go:76] Trace[1593194609]: "Get /api/v1/namespaces/kube-system/secrets/metrics-server-token-sg8kw" (started: 2019-01-08 20:29:45.258148495 +0000 UTC m=+14.729453856) (total time: 654.93605ms):
Trace[1593194609]: [654.893585ms] [654.892179ms] About to write a response
I0108 20:29:45.913384       1 trace.go:76] Trace[2002457097]: "Get /api/v1/namespaces/ingress-nginx/secrets/nginx-ingress-serviceaccount-token-254ms" (started: 2019-01-08 20:29:45.2575558 +0000 UTC m=+14.728861162) (total time: 655.810986ms):
Trace[2002457097]: [655.724123ms] [655.722029ms] About to write a response
I0108 20:29:45.913615       1 trace.go:76] Trace[204354824]: "Get /api/v1/namespaces/ingress-nginx/secrets/nginx-ingress-serviceaccount-token-254ms" (started: 2019-01-08 20:29:45.257149498 +0000 UTC m=+14.728454857) (total time: 656.449102ms):
Trace[204354824]: [656.410311ms] [656.407997ms] About to write a response
I0108 20:29:45.913889       1 trace.go:76] Trace[299852986]: "Get /api/v1/namespaces/ingress-nginx/secrets/nginx-ingress-serviceaccount-token-254ms" (started: 2019-01-08 20:29:45.2573377 +0000 UTC m=+14.728643093) (total time: 656.535189ms):
Trace[299852986]: [656.44289ms] [656.440764ms] About to write a response
I0108 20:29:45.924187       1 trace.go:76] Trace[664421271]: "Get /api/v1/namespaces/kube-system/serviceaccounts/canal" (started: 2019-01-08 20:29:45.318194278 +0000 UTC m=+14.789499645) (total time: 605.970929ms):
Trace[664421271]: [605.912937ms] [605.910809ms] About to write a response
I0108 20:29:45.924467       1 trace.go:76] Trace[902080218]: "Get /api/v1/namespaces/kube-system/serviceaccounts/canal" (started: 2019-01-08 20:29:45.311361864 +0000 UTC m=+14.782667224) (total time: 613.087493ms):
Trace[902080218]: [613.03465ms] [613.032532ms] About to write a response
I0108 20:29:45.924646       1 trace.go:76] Trace[250087854]: "Get /api/v1/namespaces/kube-system/serviceaccounts/kube-dns-autoscaler" (started: 2019-01-08 20:29:45.30786753 +0000 UTC m=+14.779172913) (total time: 616.75579ms):
Trace[250087854]: [616.725788ms] [616.724259ms] About to write a response
I0108 20:29:45.924794       1 trace.go:76] Trace[1346442210]: "Get /api/v1/namespaces/kube-system/serviceaccounts/canal" (started: 2019-01-08 20:29:45.307717248 +0000 UTC m=+14.779022670) (total time: 617.062005ms):
Trace[1346442210]: [617.035549ms] [617.033972ms] About to write a response
I0108 20:29:45.924941       1 trace.go:76] Trace[1594282935]: "Get /api/v1/namespaces/cattle-logging/serviceaccounts/fluentd" (started: 2019-01-08 20:29:45.30801821 +0000 UTC m=+14.779323571) (total time: 616.908927ms):
Trace[1594282935]: [616.883083ms] [616.881815ms] About to write a response
I0108 20:29:45.925173       1 trace.go:76] Trace[1929133905]: "Get /api/v1/namespaces/ingress-nginx/serviceaccounts/nginx-ingress-serviceaccount" (started: 2019-01-08 20:29:45.307543522 +0000 UTC m=+14.778848966) (total time: 617.613225ms):
Trace[1929133905]: [617.576246ms] [617.571966ms] About to write a response
I0108 20:29:45.931425       1 trace.go:76] Trace[1458231420]: "List /apis/rbac.authorization.k8s.io/v1/clusterrolebindings" (started: 2019-01-08 20:29:45.317997288 +0000 UTC m=+14.789302667) (total time: 613.404874ms):
Trace[1458231420]: [609.056524ms] [609.054541ms] Listing from storage done
I0108 20:29:45.931507       1 trace.go:76] Trace[944848865]: "List /api/v1/services" (started: 2019-01-08 20:29:45.276818178 +0000 UTC m=+14.748123539) (total time: 654.676347ms):
Trace[944848865]: [650.860984ms] [650.859136ms] Listing from storage done
I0108 20:29:45.931573       1 trace.go:76] Trace[497144353]: "Get /api/v1/nodes/172.10.10.11" (started: 2019-01-08 20:29:45.32902087 +0000 UTC m=+14.800326249) (total time: 602.54007ms):
Trace[497144353]: [599.023226ms] [598.997807ms] About to write a response
I0108 20:29:45.932332       1 trace.go:76] Trace[1972061241]: "Get /api/v1/namespaces/ingress-nginx/secrets/nginx-ingress-serviceaccount-token-254ms" (started: 2019-01-08 20:29:45.321329617 +0000 UTC m=+14.792634979) (total time: 610.985215ms):
Trace[1972061241]: [610.94318ms] [610.940915ms] About to write a response
I0108 20:29:45.935664       1 trace.go:76] Trace[1938971720]: "Get /api/v1/namespaces/cattle-logging/secrets/fluentd-token-7kk7d" (started: 2019-01-08 20:29:45.271729614 +0000 UTC m=+14.743034975) (total time: 663.915342ms):
Trace[1938971720]: [663.8794ms] [663.877662ms] About to write a response
I0108 20:29:45.935913       1 trace.go:76] Trace[762384217]: "Get /api/v1/namespaces/kube-system/secrets/cert-manager-token-n554m" (started: 2019-01-08 20:29:45.32580535 +0000 UTC m=+14.797110711) (total time: 610.090227ms):
Trace[762384217]: [610.022429ms] [610.020287ms] About to write a response
I0108 20:29:45.936113       1 trace.go:76] Trace[1298719998]: "Get /api/v1/namespaces/ingress-nginx/secrets/nginx-ingress-serviceaccount-token-254ms" (started: 2019-01-08 20:29:45.326554124 +0000 UTC m=+14.797859502) (total time: 609.53587ms):
Trace[1298719998]: [609.503814ms] [609.502067ms] About to write a response
I0108 20:29:45.936988       1 trace.go:76] Trace[1638872958]: "Get /api/v1/namespaces/kube-system/secrets/cert-manager-token-n554m" (started: 2019-01-08 20:29:45.326102723 +0000 UTC m=+14.797408084) (total time: 610.866125ms):
Trace[1638872958]: [610.807174ms] [610.805656ms] About to write a response
I0108 20:29:45.937230       1 trace.go:76] Trace[217231283]: "Get /api/v1/namespaces/kube-system/secrets/cert-manager-token-n554m" (started: 2019-01-08 20:29:45.327030322 +0000 UTC m=+14.798335682) (total time: 610.181881ms):
Trace[217231283]: [610.117471ms] [610.11536ms] About to write a response
I0108 20:29:45.937454       1 trace.go:76] Trace[742583003]: "Get /api/v1/namespaces/cattle-logging/secrets/fluentd-token-7kk7d" (started: 2019-01-08 20:29:45.325956065 +0000 UTC m=+14.797261424) (total time: 611.48177ms):
Trace[742583003]: [611.444015ms] [611.442312ms] About to write a response
I0108 20:29:45.937634       1 trace.go:76] Trace[1570758941]: "Get /api/v1/namespaces/ingress-nginx/secrets/nginx-ingress-serviceaccount-token-254ms" (started: 2019-01-08 20:29:45.325641699 +0000 UTC m=+14.796947060) (total time: 611.976872ms):
Trace[1570758941]: [611.947522ms] [611.945562ms] About to write a response
I0108 20:29:45.937826       1 trace.go:76] Trace[1996892821]: "Get /api/v1/namespaces/ingress-nginx/secrets/nginx-ingress-serviceaccount-token-254ms" (started: 2019-01-08 20:29:45.321494233 +0000 UTC m=+14.792799593) (total time: 616.316052ms):
Trace[1996892821]: [616.272266ms] [616.270675ms] About to write a response
I0108 20:29:45.939663       1 trace.go:76] Trace[880920607]: "Get /api/v1/namespaces/kube-system/secrets/cert-manager-token-n554m" (started: 2019-01-08 20:29:45.32084208 +0000 UTC m=+14.792147439) (total time: 618.708751ms):
Trace[880920607]: [618.653756ms] [618.651944ms] About to write a response
I0108 20:29:45.941709       1 trace.go:76] Trace[617322789]: "Get /api/v1/namespaces/ingress-nginx/secrets/nginx-ingress-serviceaccount-token-254ms" (started: 2019-01-08 20:29:45.325448877 +0000 UTC m=+14.796754238) (total time: 616.227317ms):
Trace[617322789]: [616.061472ms] [616.059675ms] About to write a response
I0108 20:29:45.946903       1 trace.go:76] Trace[342654281]: "Get /api/v1/namespaces/cattle-system/secrets/rancher-token-pqxfv" (started: 2019-01-08 20:29:45.322548063 +0000 UTC m=+14.793853775) (total time: 624.325249ms):
Trace[342654281]: [624.076953ms] [624.075092ms] About to write a response
I0108 20:29:45.947609       1 trace.go:76] Trace[1268593093]: "Get /api/v1/namespaces/kube-system/secrets/cert-manager-token-n554m" (started: 2019-01-08 20:29:45.322709512 +0000 UTC m=+14.794014872) (total time: 624.869194ms):
Trace[1268593093]: [624.663994ms] [624.662467ms] About to write a response
I0108 20:29:45.948359       1 trace.go:76] Trace[829079599]: "Get /api/v1/namespaces/ingress-nginx/secrets/nginx-ingress-serviceaccount-token-254ms" (started: 2019-01-08 20:29:45.309759784 +0000 UTC m=+14.781065160) (total time: 638.565796ms):
Trace[829079599]: [638.303154ms] [638.301633ms] About to write a response
I0108 20:29:45.949926       1 trace.go:76] Trace[1291150180]: "Get /api/v1/namespaces/kube-system/secrets/cert-manager-token-n554m" (started: 2019-01-08 20:29:45.309918929 +0000 UTC m=+14.781224307) (total time: 639.985313ms):
Trace[1291150180]: [639.737249ms] [639.735835ms] About to write a response
I0108 20:29:45.950600       1 trace.go:76] Trace[269029801]: "Get /api/v1/namespaces/kube-system/secrets/cert-manager-token-n554m" (started: 2019-01-08 20:29:45.309588038 +0000 UTC m=+14.780893416) (total time: 640.992495ms):
Trace[269029801]: [640.765754ms] [640.763927ms] About to write a response
I0108 20:29:45.951364       1 trace.go:76] Trace[1582313338]: "Get /api/v1/namespaces/kube-system/secrets/cert-manager-token-n554m" (started: 2019-01-08 20:29:45.306936894 +0000 UTC m=+14.778242256) (total time: 644.406955ms):
Trace[1582313338]: [644.086206ms] [644.083402ms] About to write a response
I0108 20:29:45.953025       1 trace.go:76] Trace[244236482]: "Get /api/v1/namespaces/kube-system/secrets/canal-token-mmxgp" (started: 2019-01-08 20:29:45.30142761 +0000 UTC m=+14.772732988) (total time: 651.570849ms):
Trace[244236482]: [651.341575ms] [651.339358ms] About to write a response
I0108 20:29:45.953831       1 trace.go:76] Trace[118085517]: "Get /api/v1/namespaces/kube-system/secrets/canal-token-mmxgp" (started: 2019-01-08 20:29:45.271897343 +0000 UTC m=+14.743202714) (total time: 681.901145ms):
Trace[118085517]: [681.645539ms] [681.643271ms] About to write a response
I0108 20:29:45.955408       1 storage_scheduling.go:100] all system priority classes are created successfully or already exist.
I0108 20:29:46.932927       1 trace.go:76] Trace[239295467]: "GuaranteedUpdate etcd3: *core.ConfigMap" (started: 2019-01-08 20:29:46.326778443 +0000 UTC m=+15.798083857) (total time: 606.120051ms):
Trace[239295467]: [606.120051ms] [606.078837ms] END
I0108 20:29:46.933047       1 trace.go:76] Trace[1126035321]: "Update /api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication" (started: 2019-01-08 20:29:46.326698019 +0000 UTC m=+15.798003450) (total time: 606.330481ms):
Trace[1126035321]: [606.252408ms] [606.201899ms] Object stored in database
I0108 20:29:46.933941       1 trace.go:76] Trace[1003606430]: "Get /api/v1/namespaces/kube-system/configmaps/kube-dns-autoscaler" (started: 2019-01-08 20:29:46.320441542 +0000 UTC m=+15.791746903) (total time: 613.481449ms):
Trace[1003606430]: [613.320349ms] [613.317382ms] About to write a response
I0108 20:29:46.951164       1 trace.go:76] Trace[185718876]: "Get /apis/crd.projectcalico.org/v1/clusterinformations/default" (started: 2019-01-08 20:29:46.315449078 +0000 UTC m=+15.786754468) (total time: 635.691383ms):
Trace[185718876]: [635.629876ms] [635.627784ms] About to write a response
I0108 20:29:46.951480       1 trace.go:76] Trace[485330009]: "Get /apis/crd.projectcalico.org/v1/clusterinformations/default" (started: 2019-01-08 20:29:46.315410119 +0000 UTC m=+15.786715480) (total time: 636.049351ms):
Trace[485330009]: [635.995792ms] [635.99245ms] About to write a response
I0108 20:29:47.419328       1 trace.go:76] Trace[1716720955]: "List /api/v1/namespaces/ingress-nginx/pods" (started: 2019-01-08 20:29:46.869544019 +0000 UTC m=+16.340849395) (total time: 549.711625ms):
Trace[1716720955]: [151.073256ms] [151.024498ms] Listing from storage done
Trace[1716720955]: [549.708473ms] [398.619905ms] Writing http response done (3 items)
I0108 20:29:47.420036       1 trace.go:76] Trace[311230024]: "List /api/v1/secrets" (started: 2019-01-08 20:29:46.862888362 +0000 UTC m=+16.334193741) (total time: 557.075308ms):
Trace[311230024]: [557.074127ms] [556.901788ms] Writing http response done (76 items)
I0108 20:29:47.420132       1 trace.go:76] Trace[747058973]: "List /api/v1/configmaps" (started: 2019-01-08 20:29:46.869151977 +0000 UTC m=+16.340457355) (total time: 550.967017ms):
Trace[747058973]: [550.965963ms] [550.851379ms] Writing http response done (21 items)
I0108 20:29:47.569317       1 trace.go:76] Trace[1157933846]: "GuaranteedUpdate etcd3: *core.Event" (started: 2019-01-08 20:29:47.021432365 +0000 UTC m=+16.492737777) (total time: 547.858276ms):
Trace[1157933846]: [547.845205ms] [547.845205ms] initial value restored
I0108 20:29:47.569421       1 trace.go:76] Trace[2072823108]: "Patch /api/v1/namespaces/cattle-system/events/rancher.155626a58f04143f" (started: 2019-01-08 20:29:47.021388792 +0000 UTC m=+16.492694199) (total time: 548.007319ms):
Trace[2072823108]: [547.891703ms] [547.881381ms] About to apply patch
I0108 20:29:47.652792       1 trace.go:76] Trace[449623081]: "GuaranteedUpdate etcd3: *core.ConfigMap" (started: 2019-01-08 20:29:47.041424226 +0000 UTC m=+16.512729637) (total time: 611.274503ms):
Trace[449623081]: [611.145741ms] [610.927409ms] Transaction committed
I0108 20:29:47.652930       1 trace.go:76] Trace[1306152395]: "Update /api/v1/namespaces/kube-system/configmaps/cert-manager-controller" (started: 2019-01-08 20:29:47.02005687 +0000 UTC m=+16.491362247) (total time: 632.85429ms):
Trace[1306152395]: [632.762941ms] [611.424405ms] Object stored in database
I0108 20:29:47.687219       1 trace.go:76] Trace[2059264320]: "Get /apis/extensions/v1beta1/namespaces/kube-system/deployments/kube-dns/scale" (started: 2019-01-08 20:29:47.043603693 +0000 UTC m=+16.514909071) (total time: 643.589924ms):
Trace[2059264320]: [643.34942ms] [643.346632ms] About to write a response
I0108 20:29:47.945435       1 trace.go:76] Trace[1659914373]: "List /api/v1/componentstatuses" (started: 2019-01-08 20:29:46.859150044 +0000 UTC m=+16.330455405) (total time: 1.086258258s):
Trace[1659914373]: [1.08579656s] [1.085765568s] Listing from storage done
I0108 20:29:47.948460       1 trace.go:76] Trace[229726629]: "Create /api/v1/namespaces/cattle-system/events" (started: 2019-01-08 20:29:47.444351889 +0000 UTC m=+16.915657287) (total time: 504.086615ms):
Trace[229726629]: [504.044653ms] [503.860732ms] Object stored in database
I0108 20:29:48.809922       1 trace.go:76] Trace[1926898508]: "Get /apis/rbac.authorization.k8s.io/v1/clusterroles/edit" (started: 2019-01-08 20:29:47.965595881 +0000 UTC m=+17.436901242) (total time: 844.288683ms):
Trace[1926898508]: [844.218682ms] [844.216628ms] About to write a response
I0108 20:29:48.821499       1 trace.go:76] Trace[120758483]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-01-08 20:29:47.733231779 +0000 UTC m=+17.204537168) (total time: 1.088242853s):
Trace[120758483]: [1.088242853s] [1.087801545s] END
I0108 20:29:48.821587       1 trace.go:76] Trace[2145345990]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.apps/status" (started: 2019-01-08 20:29:47.732660818 +0000 UTC m=+17.203966179) (total time: 1.088910895s):
Trace[2145345990]: [1.088857047s] [1.088303429s] Object stored in database
I0108 20:29:48.821903       1 trace.go:76] Trace[1768160008]: "Get /apis/crd.projectcalico.org/v1/felixconfigurations/node.172.10.10.13" (started: 2019-01-08 20:29:47.73103129 +0000 UTC m=+17.202336649) (total time: 1.090851791s):
Trace[1768160008]: [1.090851791s] [1.090849812s] END
I0108 20:29:48.832295       1 trace.go:76] Trace[1373364237]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-01-08 20:29:47.732859666 +0000 UTC m=+17.204165051) (total time: 1.089367973s):
Trace[1373364237]: [1.089367973s] [1.08932795s] END
I0108 20:29:48.832391       1 trace.go:76] Trace[886198616]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1.autoscaling/status" (started: 2019-01-08 20:29:47.732776844 +0000 UTC m=+17.204082221) (total time: 1.099597019s):
Trace[886198616]: [1.09953614s] [1.099473533s] Object stored in database
I0108 20:29:48.832609       1 trace.go:76] Trace[1681942013]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-01-08 20:29:47.732077919 +0000 UTC m=+17.203383302) (total time: 1.100516028s):
Trace[1681942013]: [1.100516028s] [1.100456061s] END
I0108 20:29:48.832677       1 trace.go:76] Trace[501688271]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1.apps/status" (started: 2019-01-08 20:29:47.731986734 +0000 UTC m=+17.203292093) (total time: 1.100676784s):
Trace[501688271]: [1.100636586s] [1.100568321s] Object stored in database
I0108 20:29:48.832900       1 trace.go:76] Trace[670215752]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-01-08 20:29:47.733939754 +0000 UTC m=+17.205245138) (total time: 1.098945775s):
Trace[670215752]: [1.098945775s] [1.09891128s] END
I0108 20:29:48.832967       1 trace.go:76] Trace[2009911888]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1.authorization.k8s.io/status" (started: 2019-01-08 20:29:47.73269973 +0000 UTC m=+17.204005091) (total time: 1.100252369s):
Trace[2009911888]: [1.100213536s] [1.098993915s] Object stored in database
I0108 20:29:48.833157       1 trace.go:76] Trace[670838360]: "GuaranteedUpdate etcd3: *apiregistration.APIService" (started: 2019-01-08 20:29:47.734199658 +0000 UTC m=+17.205505044) (total time: 1.098944296s):
Trace[670838360]: [1.098944296s] [1.098828542s] END
I0108 20:29:48.833225       1 trace.go:76] Trace[1102666968]: "Update /apis/apiregistration.k8s.io/v1/apiservices/v1alpha1.certmanager.k8s.io/status" (started: 2019-01-08 20:29:47.732735855 +0000 UTC m=+17.204041215) (total time: 1.100475794s):
Trace[1102666968]: [1.1004344s] [1.098987224s] Object stored in database
I0108 20:29:48.834492       1 trace.go:76] Trace[246086405]: "List /api/v1/nodes" (started: 2019-01-08 20:29:47.695617633 +0000 UTC m=+17.166923015) (total time: 1.138857587s):
Trace[246086405]: [1.137990099s] [1.137987805s] Listing from storage done
I0108 20:29:48.834945       1 trace.go:76] Trace[1812880337]: "Get /api/v1/nodes/172.10.10.12" (started: 2019-01-08 20:29:47.778697523 +0000 UTC m=+17.250002920) (total time: 1.056231207s):
Trace[1812880337]: [1.043205603s] [1.043202496s] About to write a response
I0108 20:29:48.843878       1 trace.go:76] Trace[370402172]: "Get /apis/crd.projectcalico.org/v1/felixconfigurations/node.172.10.10.11" (started: 2019-01-08 20:29:47.731319244 +0000 UTC m=+17.202624605) (total time: 1.112534042s):
Trace[370402172]: [1.112534042s] [1.112532407s] END
I0108 20:29:48.844150       1 trace.go:76] Trace[539931087]: "Get /apis/crd.projectcalico.org/v1/felixconfigurations/node.172.10.10.12" (started: 2019-01-08 20:29:47.790634162 +0000 UTC m=+17.261939553) (total time: 1.053495456s):
Trace[539931087]: [1.053495456s] [1.053493464s] END
I0108 20:29:48.940981       1 trace.go:76] Trace[946024438]: "GuaranteedUpdate etcd3: *core.Endpoints" (started: 2019-01-08 20:29:47.729563696 +0000 UTC m=+17.200869110) (total time: 1.211391676s):
Trace[946024438]: [1.09248679s] [1.09248679s] initial value restored
W0108 20:29:48.960719       1 lease.go:223] Resetting endpoints for master service "kubernetes" to [172.10.10.11 172.10.10.12 172.10.10.13]
I0108 20:29:49.624255       1 trace.go:76] Trace[597491189]: "GuaranteedUpdate etcd3: *core.Endpoints" (started: 2019-01-08 20:29:48.960746494 +0000 UTC m=+18.432051877) (total time: 663.457354ms):
Trace[597491189]: [663.43761ms] [654.242144ms] Transaction committed
I0108 20:29:49.639704       1 trace.go:76] Trace[64951410]: "Get /api/v1/nodes/172.10.10.13" (started: 2019-01-08 20:29:49.129877174 +0000 UTC m=+18.601182556) (total time: 509.796033ms):
Trace[64951410]: [509.30566ms] [509.30234ms] About to write a response
I0108 20:29:49.640603       1 trace.go:76] Trace[346357136]: "Get /api/v1/namespaces/kube-system/configmaps/kube-dns-autoscaler" (started: 2019-01-08 20:29:48.844945225 +0000 UTC m=+18.316250603) (total time: 795.640923ms):
Trace[346357136]: [795.584237ms] [795.581299ms] About to write a response
I0108 20:29:49.641917       1 trace.go:76] Trace[1153703552]: "Get /api/v1/namespaces/kube-system/endpoints/kube-scheduler" (started: 2019-01-08 20:29:49.02603333 +0000 UTC m=+18.497338708) (total time: 615.865642ms):
Trace[1153703552]: [615.83684ms] [615.833076ms] About to write a response
I0108 20:29:49.642144       1 trace.go:76] Trace[1199646544]: "Get /apis/rbac.authorization.k8s.io/v1/clusterroles/system:node-problem-detector" (started: 2019-01-08 20:29:48.969631243 +0000 UTC m=+18.440936603) (total time: 672.492231ms):
Trace[1199646544]: [672.455797ms] [672.453978ms] About to write a response
E0108 20:29:50.113048       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
I0108 20:29:52.950313       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:29:52.966814       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:29:52.966835       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0108 20:29:55.117937       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:30:00.120859       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:30:05.123577       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:30:10.125724       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:30:15.128366       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:30:20.130804       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:30:25.133229       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:30:30.135988       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
I0108 20:30:52.967018       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:30:52.969364       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:30:52.969383       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 20:32:34.566518       1 log.go:172] http: TLS handshake error from 172.10.10.12:29329: EOF
I0108 20:32:34.582046       1 log.go:172] http: TLS handshake error from 172.10.10.13:15770: EOF
I0108 20:32:34.650381       1 log.go:172] http: TLS handshake error from 172.10.10.11:8126: EOF
I0108 20:32:52.969555       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:32:52.971559       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:32:52.971577       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0108 20:33:07.568004       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:33:12.570537       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:33:17.572873       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:33:22.575567       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:33:27.577835       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:33:32.580288       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:33:37.582879       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:33:42.585164       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:33:47.587724       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:33:52.589982       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
I0108 20:34:49.090570       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:34:49.092534       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:34:49.092550       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 20:35:49.092764       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:35:49.094654       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:35:49.094668       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 20:35:58.657900       1 controller.go:597] quota admission added evaluator for: { serviceaccounts}
I0108 20:35:58.676005       1 controller.go:597] quota admission added evaluator for: {apps daemonsets}
I0108 20:35:59.103156       1 controller.go:597] quota admission added evaluator for: {management.cattle.io clusterpipelines}
I0108 20:36:28.687037       1 log.go:172] http: TLS handshake error from 172.10.10.11:18801: EOF
I0108 20:36:28.706354       1 log.go:172] http: TLS handshake error from 172.10.10.12:16331: EOF
I0108 20:36:28.743303       1 log.go:172] http: TLS handshake error from 172.10.10.13:17678: EOF
E0108 20:37:01.911476       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:37:06.914113       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:37:11.916636       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:37:16.918875       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:37:21.921280       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:37:26.923380       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:37:31.925730       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:37:36.928809       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:37:41.931276       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:37:46.934514       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
I0108 20:37:49.094845       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:37:49.096857       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:37:49.096880       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 20:39:49.129629       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:39:49.131695       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:39:49.131714       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 20:40:12.816867       1 log.go:172] http: TLS handshake error from 172.10.10.12:17953: EOF
I0108 20:40:12.847389       1 log.go:172] http: TLS handshake error from 172.10.10.13:23720: EOF
I0108 20:40:12.890770       1 log.go:172] http: TLS handshake error from 172.10.10.11:21465: EOF
E0108 20:40:47.489296       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
I0108 20:40:49.131987       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:40:49.134098       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:40:49.134113       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0108 20:40:52.491613       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:40:57.493835       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:41:02.496353       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:41:07.498897       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:41:12.501153       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:41:17.503464       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:41:22.505725       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:41:27.507978       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 20:41:32.510477       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
I0108 20:42:49.134334       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:42:49.136155       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:42:49.136170       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 20:44:49.093522       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:44:49.096020       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:44:49.096046       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 20:45:49.096280       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:45:49.098202       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:45:49.098219       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 20:47:49.098414       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:47:49.100855       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:47:49.100874       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 20:49:49.123166       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:49:49.125858       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:49:49.125872       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 20:50:49.126099       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:50:49.128231       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:50:49.128256       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 20:52:49.128479       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:52:49.130533       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:52:49.130552       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 20:54:49.166136       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:54:49.168560       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:54:49.168594       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 20:55:49.168777       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:55:49.170760       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:55:49.170775       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 20:57:49.171502       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:57:49.174818       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:57:49.174847       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 20:59:49.216916       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 20:59:49.219075       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 20:59:49.219095       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 21:00:49.219398       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 21:00:49.221483       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 21:00:49.221499       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 21:01:15.540933       1 log.go:172] http: TLS handshake error from 172.10.10.13:9963: EOF
I0108 21:01:15.565119       1 log.go:172] http: TLS handshake error from 172.10.10.11:25034: EOF
I0108 21:01:15.578440       1 log.go:172] http: TLS handshake error from 172.10.10.12:4259: EOF
E0108 21:01:49.031765       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 21:01:54.034329       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 21:01:59.036951       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 21:02:04.039716       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 21:02:09.042340       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 21:02:14.044707       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 21:02:19.047011       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 21:02:24.049510       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 21:02:29.052199       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
E0108 21:02:34.055359       1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")]
I0108 21:02:49.221712       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 21:02:49.223716       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 21:02:49.223732       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 21:04:49.152906       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 21:04:49.155722       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 21:04:49.155741       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 21:05:49.155964       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 21:05:49.158083       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 21:05:49.158099       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 21:07:49.158287       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 21:07:49.160151       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 21:07:49.160169       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 21:09:49.156324       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 21:09:49.160129       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 21:09:49.160163       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 21:10:49.160374       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 21:10:49.162372       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 21:10:49.162389       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 21:12:49.162648       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 21:12:49.164618       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 21:12:49.164633       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 21:14:49.098865       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 21:14:49.100838       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 21:14:49.100853       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 21:15:49.101066       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 21:15:49.102996       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 21:15:49.103015       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 21:17:49.104168       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 21:17:49.106145       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 21:17:49.106164       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 21:19:49.110518       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 21:19:49.113326       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 21:19:49.113342       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0108 21:20:49.113672       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0108 21:20:49.115946       1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0108 21:20:49.115966       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
rancher/rke

Answer questions frenchtoasters

@deniseschannon I was using them at the time to get rid of the security vulnerability that was found in the version that Rancher had defaulted to at the time I installed the cluster for version 1.11.1. When was that Kubernetes_version added into rke? I dont remember it being a value prior to this first security vulnerability being found.

@francesco2013 The ultimate fix for me on this issue was to blow away the cluster and start from scratch. I tried deleting the certs and then running rke on the cluster again to update and generate new certs, while that worked in my lab instance. In this production instance it did not work and instead of fighting with a half broken cluster I just deployed a new one and migrated workloads over.

useful!

Related questions

"Failed to reconcile etcd plane" when updating RKE binary hot 3
Failed to get /health for host - remote error: tls: bad certificate hot 2
Failed to rotate expired certificates on an RKE cluster: unable to reach api server to fetch CA hot 2
Error response from daemon: chown /etc/resolv.conf: operation not permitted hot 1
Pods can't reach networks outside of node hot 1
rke 0.1.17 Can't initiate NewClient: protocol not available hot 1
Calico node failed to start after upgrading the cluster hot 1
Job rke-network-plugin-deploy-job never completes (virtualbox) hot 1
rke up --local fails to deploy successfully hot 1
Job rke-network-plugin-deploy-job never completes (virtualbox) hot 1
go panic on intial rke up hot 1
Calico node failed to start after upgrading the cluster hot 1
pods in "CrashLoopBackOff" status after restoring from backup hot 1
[SOLVED] Failed to apply the ServiceAccount needed for job execution: Post https://10.102.X.X:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings: Forbidden hot 1
Failed to get /health for host - remote error: tls: bad certificate hot 1
Github User Rank List