profile
viewpoint

Ask questionskube controller manager refuses to connect after upgrading from 1.10.6 to 1.11.7

1. What kops version are you running? The command kops version, will display this information.

Kops 1.11.

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

Kubernetes 1.11.7

3. What cloud provider are you using?

AWS

After upgrading from Kubernetes 1.10.6 to 1.11.7 I have started getting this error in 2 of 3 of my kubernetes controller manager pods.

I0325 21:43:01.642353       1 controllermanager.go:123] Version: v1.11.7
W0325 21:43:01.643354       1 authentication.go:55] Authentication is disabled
I0325 21:43:01.643378       1 insecure_serving.go:49] Serving insecurely on [::]:10252
I0325 21:43:01.643615       1 leaderelection.go:203] attempting to acquire leader lease  kube-system/kube-controller-manager...
E0325 21:43:01.644524       1 leaderelection.go:252] error retrieving resource lock kube-system/kube-controller-manager: Get https://127.0.0.1/api/v1/namespaces/kube-system/endpoints/kube-controller-manager: dial tcp 127.0.0.1:443: connect: connection refused
E0325 21:43:05.383757       1 leaderelection.go:252] error retrieving resource lock kube-system/kube-controller-manager: Get https://127.0.0.1/api/v1/namespaces/kube-system/endpoints/kube-controller-manager: dial tcp 127.0.0.1:443: connect: connection refused
E0325 21:43:09.565787       1 leaderelection.go:252] error retrieving resource lock kube-system/kube-controller-manager: Get https://127.0.0.1/api/v1/namespaces/kube-system/endpoints/kube-controller-manager: dial tcp 127.0.0.1:443: connect: connection refused
E0325 21:43:12.001546       1 leaderelection.go:252] error retrieving resource lock kube-system/kube-controller-manager: Get https://127.0.0.1/api/v1/namespaces/kube-system/endpoints/kube-controller-manager: dial tcp 127.0.0.1:443: connect: connection refused
E0325 21:43:16.144831       1 leaderelection.go:252] error retrieving resource lock kube-system/kube-controller-manager: Get https://127.0.0.1/api/v1/namespaces/kube-system/endpoints/kube-controller-manager: dial tcp 127.0.0.1:443: connect: connection refused
E0325 21:43:18.734905       1 leaderelection.go:252] error retrieving resource lock kube-system/kube-controller-manager: Get https://127.0.0.1/api/v1/namespaces/kube-system/endpoints/kube-controller-manager: dial tcp 127.0.0.1:443: connect: connection refused
E0325 21:43:21.105232       1 leaderelection.go:252] error retrieving resource lock kube-system/kube-controller-manager: Get https://127.0.0.1/api/v1/namespaces/kube-system/endpoints/kube-controller-manager: dial tcp 127.0.0.1:443: connect: connection refused
E0325 21:43:25.483964       1 leaderelection.go:252] error retrieving resource lock kube-system/kube-controller-manager: Get https://127.0.0.1/api/v1/namespaces/kube-system/endpoints/kube-controller-manager: dial tcp 127.0.0.1:443: connect: connection refused
E0325 21:43:28.156557       1 leaderelection.go:252] error retrieving resource lock kube-system/kube-controller-manager: Get https://127.0.0.1/api/v1/namespaces/kube-system/endpoints/kube-controller-manager: dial tcp 127.0.0.1:443: connect: connection refused
E0325 21:43:38.311152       1 leaderelection.go:252] error retrieving resource lock kube-system/kube-controller-manager: endpoints "kube-controller-manager" is forbidden: User "system:kube-controller-manager" cannot get endpoints in the namespace "kube-system": RBAC: [clusterrole.rbac.authorization.k8s.io "system:basic-user" not found, clusterrole.rbac.authorization.k8s.io "system:discovery" not found, clusterrole.rbac.authorization.k8s.io "system:kube-controller-manager" not found]

I have checked and the cluster role that it is looking for is there and the other kube controller manager appears to be working correctly. The cluster also validates properly from kops validate cluster.

Any help would be much appreciated!

kubernetes/kops

Answer questions flmmartins

I also facing this issue currently

useful!

Related questions

Unable to use a local filesystem state store hot 2
Kops 1.12-beta.2 won't/can't bring up etcd server, manager or kube-api hot 1
Missing kops controller support for cloudproviders hot 1
InstanceGroup not found (for etcd ap-southeast-2a/main): "ap-southeast-2a" hot 1
Rolling-update fails due to calico-node with 1.12.0-beta.2 hot 1
Kubelet Unable To Apply Reserved Cgroup Limits because Cgroup does not exist hot 1
etcd3 and kube-apiserver fail on terraform apply after terraform destroying w/ kops generated config hot 1
Upgrade from Kops 1.11 to 1.12 has failed. hot 1
Couldn't find key etcd_endpoints in ConfigMap kube-system/calico-config hot 1
Protokube has sustained cpu usage above 100% hot 1
Allow just one instance type in mixedInstancesPolicy hot 1
kubectl command: Unable to connect to the server: EOF hot 1
DNS record for public API address not updated hot 1
etcd3 and kube-apiserver fail on terraform apply after terraform destroying w/ kops generated config hot 1
Issues encountered deploying to OpenStack hot 1
source:https://uonfu.com/
Github User Rank List