Kops 1.12-beta.2 won't/can't bring up etcd server, manager or kube-api

Ask questionsKops 1.12-beta.2 won't/can't bring up etcd server, manager or kube-api

1. What kops version are you running? The command kops version, will display this information. 1.12.0-beta.2 2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag. Updating from 1.11.9 to 1.12.7 3. What cloud provider are you using? AWS 4. What commands did you run? What is the simplest way to reproduce this issue? kops edit cluster changed kubernetes version from 1.11.9 to 1.12.7 kops update-cluster --yes kops rolling-update cluster --yes --instance-group master-us-east-1a --master-interval=2m After the failed update, I tried changing my cluster spec back to 1.11.9 to roll-back, but the master still cannot come back up 5. What happened after the commands executed? Master instance was killed and restarted, but never came back as member of the cluster 6. What did you expect to happen? Master coming back as Ready with kubernetes 1.12.7 OR 1.11.9 installed 7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2018-07-09T12:45:40Z
  name: kube.example.com
spec:
  additionalPolicies:
    master: |
      [
        {
          "Effect": "Allow",
          "Action": [
            "sts:AssumeRole"
          ],
          "Resource": "arn:aws:iam::<redacted>"
        }
      ]
  api:
    dns: {}
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://kops.example.com/kube.example.com
  etcdClusters:
  - enableEtcdTLS: true
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-us-east-1a
      name: us-east-1a
    - encryptedVolume: true
      instanceGroup: master-us-east-1d
      name: us-east-1d
    - encryptedVolume: true
      instanceGroup: master-us-east-1e
      name: us-east-1e
    name: main
    version: 3.2.18
  - enableEtcdTLS: true
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-us-east-1a
      name: us-east-1a
    - encryptedVolume: true
      instanceGroup: master-us-east-1d
      name: us-east-1d
    - encryptedVolume: true
      instanceGroup: master-us-east-1e
      name: us-east-1e
    name: events
    version: 3.2.18
  iam:
    legacy: false
  kubeAPIServer:
    disableBasicAuth: true
    oidcClientID: <redacted>.apps.googleusercontent.com
    oidcIssuerURL: https://accounts.google.com
    oidcUsernameClaim: email
  kubeControllerManager:
    horizontalPodAutoscalerDownscaleDelay: 5m0s
    horizontalPodAutoscalerSyncPeriod: 1m0s
    horizontalPodAutoscalerUpscaleDelay: 2m0s
  kubeDNS:
    provider: CoreDNS
  kubelet:
    anonymousAuth: false
    authenticationTokenWebhook: true
    authorizationMode: Webhook
  kubernetesApiAccess:
  - <redacted>
  kubernetesVersion: 1.11.9
  masterInternalName: api.internal.kube.example.com
  masterPublicName: api.kube.example.com
  networkCIDR: 10.20.0.0/16
  networking:
    kubenet: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - <redacted>
  subnets:
  - cidr: 10.20.32.0/19
    name: us-east-1a
    type: Public
    zone: us-east-1a
  - cidr: 10.20.64.0/19
    name: us-east-1d
    type: Public
    zone: us-east-1d
  - cidr: 10.20.96.0/19
    name: us-east-1e
    type: Public
    zone: us-east-1e
  topology:
    dns:
      type: Public
    masters: public
    nodes: public

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2018-07-09T12:45:42Z
  labels:
    kops.k8s.io/cluster: kube.example.com
  name: master-us-east-1a
spec:
  associatePublicIp: true
  image: ami-0de53d8956e8dcf80
  machineType: m4.large
  maxSize: 1
  minSize: 1
  role: Master
  rootVolumeOptimization: true
  rootVolumeSize: 64
  subnets:
  - us-east-1a

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here. Not sure if these are useful, but I can provide some more context information: ssh'ing into the node when rolling back to 1.11.9 I can see that both etcd-server containers are failing to start, as well as the kube-api container.

$ sudo docker ps -a
CONTAINER ID        IMAGE                                                                                                        COMMAND                  CREATED              STATUS                            PORTS               NAMES
3c67e4ddacf9        ef9a84756d40                                                                                                 "/bin/sh -c 'mkfif..."   About a minute ago   Exited (255) About a minute ago                       k8s_kube-apiserver_kube-apiserver-ip-10-20-68-128.ec2.internal_kube-system_329e569c44fb94d2ded0148837b45439_174
603dd269911b        b8df3b177be2                                                                                                 "/bin/sh -c 'mkfif..."   2 minutes ago        Exited (2) 2 minutes ago                              k8s_etcd-container_etcd-server-ip-10-20-68-128.ec2.internal_kube-system_2b458b36960a4d092e6477c4af21d58b_186
2f9ebb390d34        b8df3b177be2                                                                                                 "/bin/sh -c 'mkfif..."   5 minutes ago        Exited (2) 5 minutes ago                              k8s_etcd-container_etcd-server-events-ip-10-20-68-128.ec2.internal_kube-system_1f079d7b93918625d3c7298c9e17c3e7_185

Kube-api server logs show this:

I0417 08:01:37.868608       1 server.go:703] external host was not specified, using 10.20.68.128
I0417 08:01:37.868933       1 server.go:727] Initializing deserialization cache size based on 0MB limit
I0417 08:01:37.868941       1 server.go:746] Initializing cache sizes based on 0MB limit
I0417 08:01:37.869018       1 server.go:145] Version: v1.11.9
I0417 08:01:38.916515       1 oidc.go:280] OIDC: No x509 certificates provided, will use host's root CA set
W0417 08:01:38.917177       1 admission.go:71] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.
I0417 08:01:38.917475       1 feature_gate.go:194] feature gates: map[Initializers:true]
I0417 08:01:38.917510       1 initialization.go:90] enabled Initializers feature as part of admission plugin setup
I0417 08:01:38.917805       1 plugins.go:158] Loaded 10 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,Priority,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,MutatingAdmissionWebhook,Initializers.
I0417 08:01:38.917835       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
W0417 08:01:38.918369       1 admission.go:71] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.
I0417 08:01:38.918805       1 plugins.go:158] Loaded 10 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,Priority,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass,MutatingAdmissionWebhook,Initializers.
I0417 08:01:38.918818       1 plugins.go:161] Loaded 6 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,Priority,PersistentVolumeClaimResize,ValidatingAdmissionWebhook,ResourceQuota.
F0417 08:01:58.922758       1 storage_decorator.go:57] Unable to create storage backend: config (&{etcd3 /registry [https://127.0.0.1:4001] /srv/kubernetes/etcd-client-key.pem /srv/kubernetes/etcd-client.pem /srv/kubernetes/ca.crt true false 1000 0xc4202bec00 <nil> 5m0s 1m0s}), err (dial tcp 127.0.0.1:4001: connect: connection refused)

The etcd-server logs this:

2019-04-17 08:06:35.787087 I | etcdserver: starting server... [version: 3.2.18, cluster version: to_be_decided]
2019-04-17 08:06:35.787787 I | embed: ClientTLS: cert = /srv/kubernetes/etcd.pem, key = /srv/kubernetes/etcd-key.pem, ca = , trusted-ca = /srv/kubernetes/ca.crt, client-cert-auth = false
2019-04-17 08:06:35.789688 I | etcdserver/membership: added member ab06ef2367302e1 [https://etcd-us-east-1a.internal.kube.example.com:2380] to cluster 94c234d87a682b67
2019-04-17 08:06:35.789732 I | rafthttp: starting peer ab06ef2367302e1...
2019-04-17 08:06:35.789768 I | rafthttp: started HTTP pipelining with peer ab06ef2367302e1
2019-04-17 08:06:35.790295 I | rafthttp: started streaming with peer ab06ef2367302e1 (writer)
2019-04-17 08:06:35.795669 I | rafthttp: started peer ab06ef2367302e1
2019-04-17 08:06:35.795698 I | rafthttp: added peer ab06ef2367302e1
2019-04-17 08:06:35.795797 I | etcdserver/membership: added member 14565ad10ac7011b [https://etcd-us-east-1e.internal.kube.example.com:2380] to cluster 94c234d87a682b67
2019-04-17 08:06:35.795821 I | rafthttp: starting peer 14565ad10ac7011b...
2019-04-17 08:06:35.795837 I | rafthttp: started HTTP pipelining with peer 14565ad10ac7011b
2019-04-17 08:06:35.800098 I | rafthttp: started peer 14565ad10ac7011b
2019-04-17 08:06:35.800127 I | rafthttp: added peer 14565ad10ac7011b
2019-04-17 08:06:35.800232 I | etcdserver/membership: added member 7adadf7676897c01 [https://etcd-us-east-1d.internal.kube.example.com:2380] to cluster 94c234d87a682b67
2019-04-17 08:06:35.800276 I | rafthttp: started streaming with peer ab06ef2367302e1 (writer)
2019-04-17 08:06:35.800329 I | rafthttp: started streaming with peer ab06ef2367302e1 (stream MsgApp v2 reader)
2019-04-17 08:06:35.800684 I | rafthttp: started streaming with peer ab06ef2367302e1 (stream Message reader)
2019-04-17 08:06:35.800899 I | rafthttp: started streaming with peer 14565ad10ac7011b (writer)
2019-04-17 08:06:35.800924 I | rafthttp: started streaming with peer 14565ad10ac7011b (writer)
2019-04-17 08:06:35.800952 I | rafthttp: started streaming with peer 14565ad10ac7011b (stream MsgApp v2 reader)
2019-04-17 08:06:35.801176 I | rafthttp: started streaming with peer 14565ad10ac7011b (stream Message reader)
2019-04-17 08:06:35.806292 I | raft: 7adadf7676897c01 [term: 1] received a MsgHeartbeat message with higher term from 14565ad10ac7011b [term: 39]
2019-04-17 08:06:35.806313 I | raft: 7adadf7676897c01 became follower at term 39
2019-04-17 08:06:35.806346 C | raft: tocommit(105736474) is out of range [lastIndex(3)]. Was the raft log corrupted, truncated, or lost?
panic: tocommit(105736474) is out of range [lastIndex(3)]. Was the raft log corrupted, truncated, or lost?

goroutine 87 [running]:
github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog.(*PackageLogger).Panicf(0xc42015e9e0, 0xfaa6db, 0x5d, 0xc42014e180, 0x2, 0x2)
	/tmp/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog/pkg_logger.go:75 +0x15c
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.(*raftLog).commitTo(0xc4201f4e00, 0x64d691a)
	/tmp/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/log.go:191 +0x15c
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.(*raft).handleHeartbeat(0xc42021e3c0, 0x8, 0x7adadf7676897c01, 0x14565ad10ac7011b, 0x27, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/tmp/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:1100 +0x54
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.stepFollower(0xc42021e3c0, 0x8, 0x7adadf7676897c01, 0x14565ad10ac7011b, 0x27, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/tmp/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:1046 +0x2b3
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.(*raft).Step(0xc42021e3c0, 0x8, 0x7adadf7676897c01, 0x14565ad10ac7011b, 0x27, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/tmp/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:778 +0x10f7
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.(*node).run(0xc420228cc0, 0xc42021e3c0)
	/tmp/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/node.go:323 +0x67b
created by github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.RestartNode
	/tmp/etcd/release/etcd/gopath/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/node.go:223 +0x33e

If I switch back to 1.12 I don't have etcd-server but etcd-managers, which seem to also be in an error loop with

2019-04-17 09:21:43.165186 I | raft: 7adadf7676897c01 became candidate at term 4923
2019-04-17 09:21:43.165198 I | raft: 7adadf7676897c01 received MsgVoteResp from 7adadf7676897c01 at term 4923
2019-04-17 09:21:43.165322 I | raft: 7adadf7676897c01 [logterm: 39, index: 105736474] sent MsgVote request to ab06ef2367302e1 at term 4923
2019-04-17 09:21:43.165366 I | raft: 7adadf7676897c01 [logterm: 39, index: 105736474] sent MsgVote request to 14565ad10ac7011b at term 4923
W0417 09:21:44.368096    5976 controller.go:675] unable to reach member etcdClusterPeerInfo{peer=peer{id:"etcd-us-east-1d" endpoints:"10.20.65.72:3996" }, info=cluster_name:"etcd" node_configuration:<name:"etcd-us-east-1d" peer_urls:"https://etcd-us-east-1d.internal.kube.example.com:2380" client_urls:"https://etcd-us-east-1d.internal.kube.example.com:4001" quarantined_client_urls:"https://etcd-us-east-1d.internal.kube.example.com:3994" > etcd_state:<cluster:<desired_cluster_size:3 cluster_token:"etcd-cluster-token-etcd" nodes:<name:"etcd-us-east-1a" peer_urls:"https://etcd-us-east-1a.internal.kube.example.com:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > nodes:<name:"etcd-us-east-1d" peer_urls:"https://etcd-us-east-1d.internal.kube.example.com:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > nodes:<name:"etcd-us-east-1e" peer_urls:"https://etcd-us-east-1e.internal.kube.example.com:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > > etcd_version:"3.2.18" > }: error building etcd client for https://etcd-us-east-1d.internal.kube.example.com:4001: context deadline exceeded
I0417 09:21:44.368194    5976 controller.go:276] etcd cluster state: etcdClusterState
  members:
  peers:
    etcdClusterPeerInfo{peer=peer{id:"etcd-us-east-1d" endpoints:"10.20.65.72:3996" }, info=cluster_name:"etcd" node_configuration:<name:"etcd-us-east-1d" peer_urls:"https://etcd-us-east-1d.internal.kube.example.com:2380" client_urls:"https://etcd-us-east-1d.internal.kube.example.com:4001" quarantined_client_urls:"https://etcd-us-east-1d.internal.kube.example.com:3994" > etcd_state:<cluster:<desired_cluster_size:3 cluster_token:"etcd-cluster-token-etcd" nodes:<name:"etcd-us-east-1a" peer_urls:"https://etcd-us-east-1a.internal.kube.example.com:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > nodes:<name:"etcd-us-east-1d" peer_urls:"https://etcd-us-east-1d.internal.kube.example.com:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > nodes:<name:"etcd-us-east-1e" peer_urls:"https://etcd-us-east-1e.internal.kube.example.com:2380" client_urls:"https://0.0.0.0:4001" quarantined_client_urls:"https://0.0.0.0:3994" > > etcd_version:"3.2.18" > }
I0417 09:21:44.368255    5976 controller.go:277] etcd cluster members: map[]
I0417 09:21:44.368272    5976 controller.go:615] sending member map to all peers: members:<name:"etcd-us-east-1d" dns:"etcd-us-east-1d.internal.kube.example.com" addresses:"10.20.65.72" >
I0417 09:21:44.368670    5976 etcdserver.go:222] updating hosts: map[10.20.65.72:[etcd-us-east-1d.internal.kube.example.com]]
I0417 09:21:44.368699    5976 hosts.go:81] hosts update: primarymap[10.20.65.72:[etcd-us-east-1d.internal.kube.example.com]], fallbacks=map[etcd-us-east-1d.internal.kube.example.com:[10.20.65.72 10.20.65.72] etcd-us-east-1a.internal.kube.example.com:[10.20.56.139 10.20.56.139] etcd-us-east-1e.internal.kube.example.com:[10.20.108.220 10.20.108.220]], final=map[10.20.65.72:[etcd-us-east-1d.internal.kube.example.com] 10.20.56.139:[etcd-us-east-1a.internal.kube.example.com etcd-us-east-1a.internal.kube.example.com] 10.20.108.220:[etcd-us-east-1e.internal.kube.example.com etcd-us-east-1e.internal.kube.example.com]]
I0417 09:21:44.369233    5976 commands.go:22] not refreshing commands - TTL not hit
I0417 09:21:44.369256    5976 s3fs.go:219] Reading file "s3://kops.example.com/kube.example.com/backups/etcd/main/control/etcd-cluster-created"
I0417 09:21:44.464399    5976 controller.go:369] spec member_count:3 etcd_version:"3.2.18"
I0417 09:21:44.464463    5976 commands.go:25] refreshing commands
I0417 09:21:44.559080    5976 vfs.go:104] listed commands in s3://kops.example.com/kube.example.com/backups/etcd/main/control: 0 commands
I0417 09:21:44.559110    5976 s3fs.go:219] Reading file "s3://kops.example.com/kube.example.com/backups/etcd/main/control/etcd-cluster-spec"
W0417 09:21:44.905340    5976 controller.go:149] unexpected error running etcd cluster reconciliation loop: etcd has 0 members registered; must issue restore-backup command to proceed
2019-04-17 09:21:45.065140 I | raft: 7adadf7676897c01 is starting a new election at term 4923
2019-04-17 09:21:45.065168 I | raft: 7adadf7676897c01 became candidate at term 4924
2019-04-17 09:21:45.065180 I | raft: 7adadf7676897c01 received MsgVoteResp from 7adadf7676897c01 at term 4924
2019-04-17 09:21:45.065191 I | raft: 7adadf7676897c01 [logterm: 39, index: 105736474] sent MsgVote request to ab06ef2367302e1 at term 4924
2019-04-17 09:21:45.065201 I | raft: 7adadf7676897c01 [logterm: 39, index: 105736474] sent MsgVote request to 14565ad10ac7011b at term 4924
I0417 09:21:45.130969    5976 peers.go:281] connecting to peer "etcd-us-east-1e" with TLS policy, servername="etcd-manager-server-etcd-us-east-1e"
W0417 09:21:45.131987    5976 peers.go:325] unable to grpc-ping discovered peer 10.20.108.220:3996: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
I0417 09:21:45.132009    5976 peers.go:347] was not able to connect to peer etcd-us-east-1e: map[10.20.108.220:3996:true]
W0417 09:21:45.132023    5976 peers.go:215] unexpected error from peer intercommunications: unable to connect to peer etcd-us-east-1e
I0417 09:21:45.159279    5976 peers.go:281] connecting to peer "etcd-us-east-1a" with TLS policy, servername="etcd-manager-server-etcd-us-east-1a"
W0417 09:21:45.160541    5976 peers.go:325] unable to grpc-ping discovered peer 10.20.56.139:3996: rpc error: code = Unavailable desc = all SubConns are in TransientFailure
I0417 09:21:45.160568    5976 peers.go:347] was not able to connect to peer etcd-us-east-1a: map[10.20.56.139:3996:true]
W0417 09:21:45.160583    5976 peers.go:215] unexpected error from peer intercommunications: unable to connect to peer etcd-us-east-1a
2019-04-17 09:21:46.365129 I | raft: 7adadf7676897c01 is starting a new election at term 4924
2019-04-17 09:21:46.365158 I | raft: 7adadf7676897c01 became candidate at term 4925
2019-04-17 09:21:46.365169 I | raft: 7adadf7676897c01 received MsgVoteResp from 7adadf7676897c01 at term 4925
2019-04-17 09:21:46.365180 I | raft: 7adadf7676897c01 [logterm: 39, index: 105736474] sent MsgVote request to ab06ef2367302e1 at term 4925
2019-04-17 09:21:46.365189 I | raft: 7adadf7676897c01 [logterm: 39, index: 105736474] sent MsgVote request to 14565ad10ac7011b at term 4925
2019-04-17 09:21:46.697211 W | rafthttp: health check for peer ab06ef2367302e1 could not connect: x509: certificate signed by unknown authority
2019-04-17 09:21:46.699170 W | rafthttp: health check for peer 14565ad10ac7011b could not connect: x509: certificate signed by unknown authority
2019-04-17 09:21:47.465141 I | raft: 7adadf7676897c01 is starting a new election at term 4925
2019-04-17 09:21:47.465176 I | raft: 7adadf7676897c01 became candidate at term 4926

9. Anything else do we need to know? I first tried doing the rolling update to 1.12 with an in-cluster docker registry mirror configured (in pass-through cache mode) with the following piece of cluster spec:

  docker:
    registryMirrors:
    - http://100.66.100.161:5000

This worked fine on all previous updates, but this time I saw these errors in my journalctl:

Apr 16 14:36:41 ip-10-20-87-223.ec2.internal systemd[1]: protokube.service failed.
Apr 16 14:36:41 ip-10-20-87-223.ec2.internal systemd[1]: Unit protokube.service entered failed state.
Apr 16 14:36:41 ip-10-20-87-223.ec2.internal systemd[1]: protokube.service: main process exited, code=exited, status=125/n/a
Apr 16 14:36:41 ip-10-20-87-223.ec2.internal docker[4235]: See '/usr/bin/docker run --help'.
Apr 16 14:36:41 ip-10-20-87-223.ec2.internal docker[4235]: /usr/bin/docker: Error response from daemon: pull access denied for protokube, repository does not exist or may require 'docker login'.
Apr 16 14:36:41 ip-10-20-87-223.ec2.internal dockerd[4145]: time="2019-04-16T14:36:41.773681744Z" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthorized: aut
Apr 16 14:36:41 ip-10-20-87-223.ec2.internal dockerd[4145]: time="2019-04-16T14:36:41.631855265Z" level=warning msg="Error getting v2 registry: Get http://100.66.100.161:5000/v2/: net/http: request canceled while waiting for connec

I just use this mirror to prevent all my nodes from pulling the same images over and over from dockerhub during a rolling cluster update, so it was easy to remove. I had hoped this would solve the issue, but then all of the above happened.

kubernetes/kops

Answer questions marek-obuchowicz

@vainu-arto thanks for updating documentation and the link above, for me it explains the issue and steps needed to update. I might suggest adding a check for this and displaying an informational message (after executing kops edit cluster or kops update) for increased safety of the operation, but as you mentioned - it's documented now in a clear way.

useful!