profile
viewpoint

Ask questionsUpgrade from Kops 1.11 to 1.12 has failed.

1. What kops version are you running? The command kops version, will display this information.

Version 1.12.3 (git-e55205471)

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

A custom version of v1.13.3 with additional patches from PRs that were merged later.

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

kops update cluster --yes
kops rolling-update cluster --cloudonly --instance-group-roles master --master-interval=1s --yes

5. What happened after the commands executed?

The master all shutdown at once, and then new ones started up. All kubectl commands fail with:

Unable to connect to the server: EOF

The load balancer for the API points at the new instances but shows them as all being OutOfService.

I can ssh to the masters and only ssh (TCP:22) and RPC (TCP:111) are listening. Port 443 has nothing bound to it.

6. What did you expect to happen?

The upgrade to work of course ;)

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2018-10-02T08:14:39Z
  name: !!!REDACTED!!!
spec:
  additionalNetworkCIDRs:
  - 10.192.0.0/16
  additionalPolicies:
    master: |
      [
        !!!REDACTED!!!
      ]
    node: |
      [
        !!!REDACTED!!!
      ]
  api:
    loadBalancer:
      sslCertificate: arn:aws:acm:ap-southeast-2:!!!REDACTED!!!:certificate/be0f5d8e-c88a-402f-a952-1e7fb0dc2977
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudLabels:
    he:Application: kubernetes
  cloudProvider: aws
  configBase: s3://he-kube-hetest-cluster-state/!!!REDACTED!!!
  dnsZone: !!!REDACTED!!!
  docker:
    storage: overlay2
    version: 18.06.2
  etcdClusters:
  - etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-ap-southeast-2a
      kmsKeyId: arn:aws:kms:ap-southeast-2:!!!REDACTED!!!:key/398c0727-a3f3-4239-a1da-b8356d320dcc
      name: a
    - encryptedVolume: true
      instanceGroup: master-ap-southeast-2b
      kmsKeyId: arn:aws:kms:ap-southeast-2:!!!REDACTED!!!:key/398c0727-a3f3-4239-a1da-b8356d320dcc
      name: b
    - encryptedVolume: true
      instanceGroup: master-ap-southeast-2c
      kmsKeyId: arn:aws:kms:ap-southeast-2:!!!REDACTED!!!:key/398c0727-a3f3-4239-a1da-b8356d320dcc
      name: c
    name: main
    version: 3.2.24
  - etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-ap-southeast-2a
      kmsKeyId: arn:aws:kms:ap-southeast-2:!!!REDACTED!!!:key/398c0727-a3f3-4239-a1da-b8356d320dcc
      name: a
    - encryptedVolume: true
      instanceGroup: master-ap-southeast-2b
      kmsKeyId: arn:aws:kms:ap-southeast-2:!!!REDACTED!!!:key/398c0727-a3f3-4239-a1da-b8356d320dcc
      name: b
    - encryptedVolume: true
      instanceGroup: master-ap-southeast-2c
      kmsKeyId: arn:aws:kms:ap-southeast-2:!!!REDACTED!!!:key/398c0727-a3f3-4239-a1da-b8356d320dcc
      name: c
    name: events
    version: 3.2.24
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeAPIServer:
    oidcClientID: !!!REDACTED!!!
    oidcIssuerURL: https://accounts.google.com
    oidcUsernameClaim: email
  kubeDNS:
    provider: CoreDNS
  kubelet:
    allowedUnsafeSysctls:
    - net.ipv4.tcp_keepalive_time
    anonymousAuth: false
    clusterDNS: 169.254.20.10
  kubernetesApiAccess:
  - !!!REDACTED!!!
  - !!!REDACTED!!!
  kubernetesVersion: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/
  masterInternalName: api.internal.!!!REDACTED!!!
  masterPublicName: api.!!!REDACTED!!!
  networkCIDR: 10.200.0.0/16
  networkID: vpc-8bf3b9ef
  networking:
    cni: {}
    weave:
      mtu: 8192
      noMasqLocal: 1
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 127.0.0.1/32
  sshKeyName: kube-hetest
  subnets:
  - cidr: 10.192.32.0/19
    egress: nat-08de68609cdf6cc58
    name: ap-southeast-2a
    type: Private
    zone: ap-southeast-2a
  - cidr: 10.192.64.0/19
    egress: nat-00ef20ab2a9408b7c
    name: ap-southeast-2b
    type: Private
    zone: ap-southeast-2b
  - cidr: 10.192.96.0/19
    egress: nat-0032baadf115cfbde
    name: ap-southeast-2c
    type: Private
    zone: ap-southeast-2c
  - cidr: 10.192.0.0/22
    name: utility-ap-southeast-2a
    type: Utility
    zone: ap-southeast-2a
  - cidr: 10.192.4.0/22
    name: utility-ap-southeast-2b
    type: Utility
    zone: ap-southeast-2b
  - cidr: 10.192.8.0/22
    name: utility-ap-southeast-2c
    type: Utility
    zone: ap-southeast-2c
  topology:
    dns:
      type: Private
    masters: private
    nodes: private

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

The cluster was already running etcd3 prior to the upgrade, but in on-TLS mode.

The upgrade showed that it was going to perform the following changes:

Using cluster from kubectl context: !!!REDACTED!!!


*********************************************************************************

A new kubernetes version is available: 1.13.10
Upgrading is recommended (try kops upgrade cluster)

More information: https://github.com/kubernetes/kops/blob/master/permalinks/upgrade_k8s.md#1.13.10

*********************************************************************************

W0910 16:22:11.338804   13946 urls.go:91] Using base url from KOPS_BASE_URL env var: "https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/"
I0910 16:22:12.208716   13946 dns.go:92] Private DNS: skipping DNS validation
I0910 16:22:12.475590   13946 executor.go:103] Tasks: 0 done / 135 total; 51 can run
I0910 16:22:13.700247   13946 executor.go:103] Tasks: 51 done / 135 total; 30 can run
I0910 16:22:14.956002   13946 executor.go:103] Tasks: 81 done / 135 total; 38 can run
I0910 16:22:17.239992   13946 executor.go:103] Tasks: 119 done / 135 total; 13 can run
I0910 16:22:17.535428   13946 dnsname.go:111] AliasTarget for "api.!!!REDACTED!!!." is "api-!!!REDACTED!!!.ap-southeast-2.elb.amazonaws.com."
I0910 16:22:17.705135   13946 executor.go:103] Tasks: 132 done / 135 total; 3 can run
I0910 16:22:17.949346   13946 executor.go:103] Tasks: 135 done / 135 total; 0 can run
Will create resources:
  Keypair/etcd-clients-ca
  	Subject             	cn=etcd-clients-ca
  	Type                	ca
  	Format              	v1alpha2

  Keypair/etcd-manager-ca-events
  	Subject             	cn=etcd-manager-ca-events
  	Type                	ca
  	Format              	v1alpha2

  Keypair/etcd-manager-ca-main
  	Subject             	cn=etcd-manager-ca-main
  	Type                	ca
  	Format              	v1alpha2

  Keypair/etcd-peers-ca-events
  	Subject             	cn=etcd-peers-ca-events
  	Type                	ca
  	Format              	v1alpha2

  Keypair/etcd-peers-ca-main
  	Subject             	cn=etcd-peers-ca-main
  	Type                	ca
  	Format              	v1alpha2

  ManagedFile/!!!REDACTED!!!-addons-coredns.addons.k8s.io-k8s-1.12
  	Location            	addons/coredns.addons.k8s.io/k8s-1.12.yaml

  ManagedFile/!!!REDACTED!!!-addons-dns-controller.addons.k8s.io-k8s-1.12
  	Location            	addons/dns-controller.addons.k8s.io/k8s-1.12.yaml

  ManagedFile/!!!REDACTED!!!-addons-kubelet-api.rbac.addons.k8s.io-k8s-1.9
  	Location            	addons/kubelet-api.rbac.addons.k8s.io/k8s-1.9.yaml

  ManagedFile/!!!REDACTED!!!-addons-networking.weave-k8s-1.12
  	Location            	addons/networking.weave/k8s-1.12.yaml

Will modify resources:
  IAMRolePolicy/masters.!!!REDACTED!!!
  	PolicyDocument      
  	                    	...
  	                    	          "autoscaling:DescribeAutoScalingGroups",
  	                    	          "autoscaling:DescribeLaunchConfigurations",
  	                    	+         "autoscaling:DescribeTags",
  	                    	+         "ec2:DescribeLaunchTemplateVersions"
  	                    	-         "autoscaling:DescribeTags"
  	                    	        ],
  	                    	        "Resource": [
  	                    	...
  	                    	

  LaunchConfiguration/!!!REDACTED!!!
  	UserData            
  	                    	...
  	                    	  - 81a4ca8471fbb5980d0e5faac49473c21712bc8a@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubelet
  	                    	  - 5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubectl
  	                    	+ - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
  	                    	- - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
  	                    	  - e59df8727c65c3a36d18421deaa711888a1b2515@https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/linux/amd64/utils.tar.gz
  	                    	  ClusterName: !!!REDACTED!!!
  	                    	...
  	                    	  images:
  	                    	  - hash: 8984887e98a0a683c12cc3aa21ec2821c831ee7f
  	                    	+   sources:
  	                    	+   - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
  	                    	-   source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
  	                    	  protokubeImage:
  	                    	    hash: ba88d42afe960f328304da3f424ac40ec2b697e8
  	                    	+   name: protokube:1.12.3
  	                    	-   name: protokube:dev
  	                    	+   sources:
  	                    	+   - https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
  	                    	-   source: https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
  	                    	  
  	                    	  __EOF_KUBE_ENV
  	                    	...
  	                    	

  LaunchConfiguration/master-ap-southeast-2a.masters.!!!REDACTED!!!
  	UserData            
  	                    	...
  	                    	    cloudProvider: aws
  	                    	    enableAdmissionPlugins:
  	                    	-   - Initializers
  	                    	    - NamespaceLifecycle
  	                    	    - LimitRanger
  	                    	...
  	                    	  - 81a4ca8471fbb5980d0e5faac49473c21712bc8a@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubelet
  	                    	  - 5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubectl
  	                    	+ - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
  	                    	- - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
  	                    	  - e59df8727c65c3a36d18421deaa711888a1b2515@https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/linux/amd64/utils.tar.gz
  	                    	  ClusterName: !!!REDACTED!!!
  	                    	...
  	                    	  images:
  	                    	  - hash: 8984887e98a0a683c12cc3aa21ec2821c831ee7f
  	                    	+   sources:
  	                    	+   - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
  	                    	-   source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
  	                    	  - hash: 303c3347fe7db7118f302f4bf15f0015f1c4d86d
  	                    	+   sources:
  	                    	+   - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-apiserver.tar
  	                    	-   source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-apiserver.tar
  	                    	  - hash: 029d26f968def3dd24590dbdd22091c8bcfae5fe
  	                    	+   sources:
  	                    	+   - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-controller-manager.tar
  	                    	-   source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-controller-manager.tar
  	                    	  - hash: 1e6495ca0a5cbe5ffcac253621da88df5a69ac31
  	                    	+   sources:
  	                    	+   - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-scheduler.tar
  	                    	-   source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-scheduler.tar
  	                    	  protokubeImage:
  	                    	    hash: ba88d42afe960f328304da3f424ac40ec2b697e8
  	                    	+   name: protokube:1.12.3
  	                    	-   name: protokube:dev
  	                    	+   sources:
  	                    	+   - https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
  	                    	-   source: https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
  	                    	  
  	                    	  __EOF_KUBE_ENV
  	                    	...
  	                    	

  LaunchConfiguration/master-ap-southeast-2b.masters.!!!REDACTED!!!
  	UserData            
  	                    	...
  	                    	    cloudProvider: aws
  	                    	    enableAdmissionPlugins:
  	                    	-   - Initializers
  	                    	    - NamespaceLifecycle
  	                    	    - LimitRanger
  	                    	...
  	                    	  - 81a4ca8471fbb5980d0e5faac49473c21712bc8a@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubelet
  	                    	  - 5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubectl
  	                    	+ - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
  	                    	- - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
  	                    	  - e59df8727c65c3a36d18421deaa711888a1b2515@https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/linux/amd64/utils.tar.gz
  	                    	  ClusterName: !!!REDACTED!!!
  	                    	...
  	                    	  images:
  	                    	  - hash: 8984887e98a0a683c12cc3aa21ec2821c831ee7f
  	                    	+   sources:
  	                    	+   - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
  	                    	-   source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
  	                    	  - hash: 303c3347fe7db7118f302f4bf15f0015f1c4d86d
  	                    	+   sources:
  	                    	+   - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-apiserver.tar
  	                    	-   source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-apiserver.tar
  	                    	  - hash: 029d26f968def3dd24590dbdd22091c8bcfae5fe
  	                    	+   sources:
  	                    	+   - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-controller-manager.tar
  	                    	-   source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-controller-manager.tar
  	                    	  - hash: 1e6495ca0a5cbe5ffcac253621da88df5a69ac31
  	                    	+   sources:
  	                    	+   - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-scheduler.tar
  	                    	-   source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-scheduler.tar
  	                    	  protokubeImage:
  	                    	    hash: ba88d42afe960f328304da3f424ac40ec2b697e8
  	                    	+   name: protokube:1.12.3
  	                    	-   name: protokube:dev
  	                    	+   sources:
  	                    	+   - https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
  	                    	-   source: https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
  	                    	  
  	                    	  __EOF_KUBE_ENV
  	                    	...
  	                    	

  LaunchConfiguration/master-ap-southeast-2c.masters.!!!REDACTED!!!
  	UserData            
  	                    	...
  	                    	    cloudProvider: aws
  	                    	    enableAdmissionPlugins:
  	                    	-   - Initializers
  	                    	    - NamespaceLifecycle
  	                    	    - LimitRanger
  	                    	...
  	                    	  - 81a4ca8471fbb5980d0e5faac49473c21712bc8a@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubelet
  	                    	  - 5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubectl
  	                    	+ - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
  	                    	- - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
  	                    	  - e59df8727c65c3a36d18421deaa711888a1b2515@https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/linux/amd64/utils.tar.gz
  	                    	  ClusterName: !!!REDACTED!!!
  	                    	...
  	                    	  images:
  	                    	  - hash: 8984887e98a0a683c12cc3aa21ec2821c831ee7f
  	                    	+   sources:
  	                    	+   - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
  	                    	-   source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
  	                    	  - hash: 303c3347fe7db7118f302f4bf15f0015f1c4d86d
  	                    	+   sources:
  	                    	+   - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-apiserver.tar
  	                    	-   source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-apiserver.tar
  	                    	  - hash: 029d26f968def3dd24590dbdd22091c8bcfae5fe
  	                    	+   sources:
  	                    	+   - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-controller-manager.tar
  	                    	-   source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-controller-manager.tar
  	                    	  - hash: 1e6495ca0a5cbe5ffcac253621da88df5a69ac31
  	                    	+   sources:
  	                    	+   - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-scheduler.tar
  	                    	-   source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-scheduler.tar
  	                    	  protokubeImage:
  	                    	    hash: ba88d42afe960f328304da3f424ac40ec2b697e8
  	                    	+   name: protokube:1.12.3
  	                    	-   name: protokube:dev
  	                    	+   sources:
  	                    	+   - https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
  	                    	-   source: https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
  	                    	  
  	                    	  __EOF_KUBE_ENV
  	                    	...
  	                    	

  LaunchConfiguration/nodes-2a.!!!REDACTED!!!
  	UserData            
  	                    	...
  	                    	  - 81a4ca8471fbb5980d0e5faac49473c21712bc8a@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubelet
  	                    	  - 5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubectl
  	                    	+ - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
  	                    	- - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
  	                    	  - e59df8727c65c3a36d18421deaa711888a1b2515@https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/linux/amd64/utils.tar.gz
  	                    	  ClusterName: !!!REDACTED!!!
  	                    	...
  	                    	  images:
  	                    	  - hash: 8984887e98a0a683c12cc3aa21ec2821c831ee7f
  	                    	+   sources:
  	                    	+   - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
  	                    	-   source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
  	                    	  protokubeImage:
  	                    	    hash: ba88d42afe960f328304da3f424ac40ec2b697e8
  	                    	+   name: protokube:1.12.3
  	                    	-   name: protokube:dev
  	                    	+   sources:
  	                    	+   - https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
  	                    	-   source: https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
  	                    	  
  	                    	  __EOF_KUBE_ENV
  	                    	...
  	                    	

  LaunchConfiguration/nodes-2b.!!!REDACTED!!!
  	UserData            
  	                    	...
  	                    	  - 81a4ca8471fbb5980d0e5faac49473c21712bc8a@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubelet
  	                    	  - 5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubectl
  	                    	+ - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
  	                    	- - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
  	                    	  - e59df8727c65c3a36d18421deaa711888a1b2515@https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/linux/amd64/utils.tar.gz
  	                    	  ClusterName: !!!REDACTED!!!
  	                    	...
  	                    	  images:
  	                    	  - hash: 8984887e98a0a683c12cc3aa21ec2821c831ee7f
  	                    	+   sources:
  	                    	+   - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
  	                    	-   source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
  	                    	  protokubeImage:
  	                    	    hash: ba88d42afe960f328304da3f424ac40ec2b697e8
  	                    	+   name: protokube:1.12.3
  	                    	-   name: protokube:dev
  	                    	+   sources:
  	                    	+   - https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
  	                    	-   source: https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
  	                    	  
  	                    	  __EOF_KUBE_ENV
  	                    	...
  	                    	

  LaunchConfiguration/nodes-2c.!!!REDACTED!!!
  	UserData            
  	                    	...
  	                    	  - 81a4ca8471fbb5980d0e5faac49473c21712bc8a@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubelet
  	                    	  - 5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubectl
  	                    	+ - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
  	                    	- - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
  	                    	  - e59df8727c65c3a36d18421deaa711888a1b2515@https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/linux/amd64/utils.tar.gz
  	                    	  ClusterName: !!!REDACTED!!!
  	                    	...
  	                    	  images:
  	                    	  - hash: 8984887e98a0a683c12cc3aa21ec2821c831ee7f
  	                    	+   sources:
  	                    	+   - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
  	                    	-   source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
  	                    	  protokubeImage:
  	                    	    hash: ba88d42afe960f328304da3f424ac40ec2b697e8
  	                    	+   name: protokube:1.12.3
  	                    	-   name: protokube:dev
  	                    	+   sources:
  	                    	+   - https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
  	                    	-   source: https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
  	                    	  
  	                    	  __EOF_KUBE_ENV
  	                    	...
  	                    	

  LaunchConfiguration/nodes.!!!REDACTED!!!
  	UserData            
  	                    	...
  	                    	  - 81a4ca8471fbb5980d0e5faac49473c21712bc8a@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubelet
  	                    	  - 5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubectl
  	                    	+ - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
  	                    	- - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
  	                    	  - e59df8727c65c3a36d18421deaa711888a1b2515@https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/linux/amd64/utils.tar.gz
  	                    	  ClusterName: !!!REDACTED!!!
  	                    	...
  	                    	  images:
  	                    	  - hash: 8984887e98a0a683c12cc3aa21ec2821c831ee7f
  	                    	+   sources:
  	                    	+   - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
  	                    	-   source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
  	                    	  protokubeImage:
  	                    	    hash: ba88d42afe960f328304da3f424ac40ec2b697e8
  	                    	+   name: protokube:1.12.3
  	                    	-   name: protokube:dev
  	                    	+   sources:
  	                    	+   - https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
  	                    	-   source: https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
  	                    	  
  	                    	  __EOF_KUBE_ENV
  	                    	...
  	                    	

  LoadBalancer/api.!!!REDACTED!!!
  	Tags                	 {KubernetesCluster: !!!REDACTED!!!, Name: api.!!!REDACTED!!!} -> {Name: api.!!!REDACTED!!!, KubernetesCluster: !!!REDACTED!!!, kubernetes.io/cluster/!!!REDACTED!!!: owned, he:Application: kubernetes}

  ManagedFile/etcd-cluster-spec-events
  	Contents            
  	                    	  {
  	                    	+   "memberCount": 3,
  	                    	-   "member_count": 3,
  	                    	+   "etcdVersion": "3.2.24"
  	                    	-   "etcd_version": "3.2.24"
  	                    	  }
  	                    	

  ManagedFile/etcd-cluster-spec-main
  	Contents            
  	                    	  {
  	                    	+   "memberCount": 3,
  	                    	-   "member_count": 3,
  	                    	+   "etcdVersion": "3.2.24"
  	                    	-   "etcd_version": "3.2.24"
  	                    	  }
  	                    	

  ManagedFile/manifests-etcdmanager-events
  	Contents            
  	                    	...
  	                    	      - mkfifo /tmp/pipe; (tee -a /var/log/etcd.log < /tmp/pipe & ) ; exec /etcd-manager
  	                    	        --backup-store=s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/backups/etcd/events
  	                    	+       --client-urls=https://__name__:4002 --cluster-name=etcd-events --containerized=true
  	                    	-       --client-urls=http://__name__:4002 --cluster-name=etcd-events --containerized=true
  	                    	+       --dns-suffix=.internal.!!!REDACTED!!! --etcd-insecure=false --grpc-port=3997 --insecure=false
  	                    	-       --dns-suffix=.internal.!!!REDACTED!!! --grpc-port=3997 --peer-urls=http://__name__:2381
  	                    	+       --peer-urls=https://__name__:2381 --quarantine-client-urls=https://__name__:3995
  	                    	-       --quarantine-client-urls=http://__name__:3995 --v=8 --volume-name-tag=k8s.io/etcd/events
  	                    	+       --v=6 --volume-name-tag=k8s.io/etcd/events --volume-provider=aws --volume-tag=k8s.io/etcd/events
  	                    	-       --volume-provider=aws --volume-tag=k8s.io/etcd/events --volume-tag=k8s.io/role/master=1
  	                    	+       --volume-tag=k8s.io/role/master=1 --volume-tag=kubernetes.io/cluster/!!!REDACTED!!!=owned
  	                    	+       > /tmp/pipe 2>&1
  	                    	-       --volume-tag=kubernetes.io/cluster/!!!REDACTED!!!=owned > /tmp/pipe 2>&1
  	                    	+     image: kopeio/etcd-manager:3.0.20190801
  	                    	-     image: kopeio/etcd-manager:1.0.20181001
  	                    	      name: etcd-manager
  	                    	      resources:
  	                    	        requests:
  	                    	+         cpu: 200m
  	                    	+         memory: 100Mi
  	                    	-         cpu: 100m
  	                    	      securityContext:
  	                    	        privileged: true
  	                    	...
  	                    	      - mountPath: /etc/hosts
  	                    	        name: hosts
  	                    	+     - mountPath: /etc/kubernetes/pki/etcd-manager
  	                    	+       name: pki
  	                    	      - mountPath: /var/log/etcd.log
  	                    	        name: varlogetcd
  	                    	...
  	                    	      name: hosts
  	                    	    - hostPath:
  	                    	+       path: /etc/kubernetes/pki/etcd-manager-events
  	                    	+       type: DirectoryOrCreate
  	                    	+     name: pki
  	                    	+   - hostPath:
  	                    	        path: /var/log/etcd-events.log
  	                    	        type: FileOrCreate
  	                    	...
  	                    	

  ManagedFile/manifests-etcdmanager-main
  	Contents            
  	                    	...
  	                    	      - mkfifo /tmp/pipe; (tee -a /var/log/etcd.log < /tmp/pipe & ) ; exec /etcd-manager
  	                    	        --backup-store=s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/backups/etcd/main
  	                    	+       --client-urls=https://__name__:4001 --cluster-name=etcd --containerized=true
  	                    	-       --client-urls=http://__name__:4001 --cluster-name=etcd --containerized=true
  	                    	+       --dns-suffix=.internal.!!!REDACTED!!! --etcd-insecure=false --grpc-port=3996 --insecure=false
  	                    	-       --dns-suffix=.internal.!!!REDACTED!!! --grpc-port=3996 --peer-urls=http://__name__:2380
  	                    	+       --peer-urls=https://__name__:2380 --quarantine-client-urls=https://__name__:3994
  	                    	-       --quarantine-client-urls=http://__name__:3994 --v=8 --volume-name-tag=k8s.io/etcd/main
  	                    	+       --v=6 --volume-name-tag=k8s.io/etcd/main --volume-provider=aws --volume-tag=k8s.io/etcd/main
  	                    	-       --volume-provider=aws --volume-tag=k8s.io/etcd/main --volume-tag=k8s.io/role/master=1
  	                    	+       --volume-tag=k8s.io/role/master=1 --volume-tag=kubernetes.io/cluster/!!!REDACTED!!!=owned
  	                    	+       > /tmp/pipe 2>&1
  	                    	-       --volume-tag=kubernetes.io/cluster/!!!REDACTED!!!=owned > /tmp/pipe 2>&1
  	                    	+     image: kopeio/etcd-manager:3.0.20190801
  	                    	-     image: kopeio/etcd-manager:1.0.20181001
  	                    	      name: etcd-manager
  	                    	      resources:
  	                    	        requests:
  	                    	          cpu: 200m
  	                    	+         memory: 100Mi
  	                    	      securityContext:
  	                    	        privileged: true
  	                    	...
  	                    	      - mountPath: /etc/hosts
  	                    	        name: hosts
  	                    	+     - mountPath: /etc/kubernetes/pki/etcd-manager
  	                    	+       name: pki
  	                    	      - mountPath: /var/log/etcd.log
  	                    	        name: varlogetcd
  	                    	...
  	                    	      name: hosts
  	                    	    - hostPath:
  	                    	+       path: /etc/kubernetes/pki/etcd-manager-main
  	                    	+       type: DirectoryOrCreate
  	                    	+     name: pki
  	                    	+   - hostPath:
  	                    	        path: /var/log/etcd.log
  	                    	        type: FileOrCreate
  	                    	...
  	                    	

  ManagedFile/!!!REDACTED!!!-addons-bootstrap
  	Contents            
  	                    	...
  	                    	      version: 1.4.0
  	                    	    - id: k8s-1.6
  	                    	+     kubernetesVersion: '>=1.6.0 <1.12.0'
  	                    	-     kubernetesVersion: '>=1.6.0'
  	                    	      manifest: coredns.addons.k8s.io/k8s-1.6.yaml
  	                    	      name: coredns.addons.k8s.io
  	                    	      selector:
  	                    	        k8s-addon: coredns.addons.k8s.io
  	                    	+     version: 1.3.1-kops.3
  	                    	+   - id: k8s-1.12
  	                    	+     kubernetesVersion: '>=1.12.0'
  	                    	+     manifest: coredns.addons.k8s.io/k8s-1.12.yaml
  	                    	+     name: coredns.addons.k8s.io
  	                    	+     selector:
  	                    	+       k8s-addon: coredns.addons.k8s.io
  	                    	+     version: 1.3.0-kops.2
  	                    	-     version: 1.2.6-kops.1
  	                    	    - id: k8s-1.8
  	                    	      kubernetesVersion: '>=1.8.0'
  	                    	...
  	                    	        k8s-addon: rbac.addons.k8s.io
  	                    	      version: 1.8.0
  	                    	+   - id: k8s-1.9
  	                    	+     kubernetesVersion: '>=1.9.0'
  	                    	+     manifest: kubelet-api.rbac.addons.k8s.io/k8s-1.9.yaml
  	                    	+     name: kubelet-api.rbac.addons.k8s.io
  	                    	+     selector:
  	                    	+       k8s-addon: kubelet-api.rbac.addons.k8s.io
  	                    	+     version: v0.0.1
  	                    	    - manifest: limit-range.addons.k8s.io/v1.5.0.yaml
  	                    	      name: limit-range.addons.k8s.io
  	                    	...
  	                    	      selector:
  	                    	        k8s-addon: dns-controller.addons.k8s.io
  	                    	+     version: 1.12.3
  	                    	-     version: 1.11.0
  	                    	    - id: k8s-1.6
  	                    	+     kubernetesVersion: '>=1.6.0 <1.12.0'
  	                    	-     kubernetesVersion: '>=1.6.0'
  	                    	      manifest: dns-controller.addons.k8s.io/k8s-1.6.yaml
  	                    	      name: dns-controller.addons.k8s.io
  	                    	      selector:
  	                    	        k8s-addon: dns-controller.addons.k8s.io
  	                    	+     version: 1.12.3
  	                    	+   - id: k8s-1.12
  	                    	+     kubernetesVersion: '>=1.12.0'
  	                    	+     manifest: dns-controller.addons.k8s.io/k8s-1.12.yaml
  	                    	+     name: dns-controller.addons.k8s.io
  	                    	+     selector:
  	                    	+       k8s-addon: dns-controller.addons.k8s.io
  	                    	+     version: 1.12.3
  	                    	-     version: 1.11.0
  	                    	    - id: v1.7.0
  	                    	      kubernetesVersion: '>=1.7.0'
  	                    	...
  	                    	      selector:
  	                    	        role.kubernetes.io/networking: "1"
  	                    	+     version: 2.5.1-kops.1
  	                    	-     version: 2.5.0-kops.1
  	                    	    - id: k8s-1.8
  	                    	+     kubernetesVersion: '>=1.8.0 <1.12.0'
  	                    	-     kubernetesVersion: '>=1.8.0'
  	                    	      manifest: networking.weave/k8s-1.8.yaml
  	                    	      name: networking.weave
  	                    	      selector:
  	                    	        role.kubernetes.io/networking: "1"
  	                    	+     version: 2.5.1-kops.1
  	                    	+   - id: k8s-1.12
  	                    	+     kubernetesVersion: '>=1.12.0'
  	                    	+     manifest: networking.weave/k8s-1.12.yaml
  	                    	+     name: networking.weave
  	                    	+     selector:
  	                    	+       role.kubernetes.io/networking: "1"
  	                    	+     version: 2.5.1-kops.1
  	                    	-     version: 2.5.0-kops.1
  	                    	

  ManagedFile/!!!REDACTED!!!-addons-coredns.addons.k8s.io-k8s-1.6
  	Contents            
  	                    	...
  	                    	  ---
  	                    	  
  	                    	+ apiVersion: rbac.authorization.k8s.io/v1
  	                    	- apiVersion: rbac.authorization.k8s.io/v1beta1
  	                    	  kind: ClusterRole
  	                    	  metadata:
  	                    	...
  	                    	  ---
  	                    	  
  	                    	+ apiVersion: rbac.authorization.k8s.io/v1
  	                    	- apiVersion: rbac.authorization.k8s.io/v1beta1
  	                    	  kind: ClusterRoleBinding
  	                    	  metadata:
  	                    	...
  	                    	          }
  	                    	          prometheus :9153
  	                    	+         forward . /etc/resolv.conf
  	                    	-         proxy . /etc/resolv.conf
  	                    	          loop
  	                    	          cache 30
  	                    	...
  	                    	  ---
  	                    	  
  	                    	+ apiVersion: apps/v1
  	                    	- apiVersion: extensions/v1beta1
  	                    	  kind: Deployment
  	                    	  metadata:
  	                    	...
  	                    	          - -conf
  	                    	          - /etc/coredns/Corefile
  	                    	+         image: k8s.gcr.io/coredns:1.3.1
  	                    	-         image: k8s.gcr.io/coredns:1.2.6
  	                    	          imagePullPolicy: IfNotPresent
  	                    	          livenessProbe:
  	                    	...
  	                    	            name: metrics
  	                    	            protocol: TCP
  	                    	+         readinessProbe:
  	                    	+           httpGet:
  	                    	+             path: /health
  	                    	+             port: 8080
  	                    	+             scheme: HTTP
  	                    	          resources:
  	                    	            limits:
  	                    	...
  	                    	        nodeSelector:
  	                    	          beta.kubernetes.io/os: linux
  	                    	+       priorityClassName: system-cluster-critical
  	                    	        serviceAccountName: coredns
  	                    	        tolerations:
  	                    	...
  	                    	      port: 53
  	                    	      protocol: TCP
  	                    	+   - name: metrics
  	                    	+     port: 9153
  	                    	+     protocol: TCP
  	                    	    selector:
  	                    	      k8s-app: kube-dns
  	                    	

  ManagedFile/!!!REDACTED!!!-addons-dns-controller.addons.k8s.io-k8s-1.6
  	Contents            
  	                    	...
  	                    	      k8s-addon: dns-controller.addons.k8s.io
  	                    	      k8s-app: dns-controller
  	                    	+     version: v1.12.0
  	                    	-     version: v1.11.0
  	                    	    name: dns-controller
  	                    	    namespace: kube-system
  	                    	...
  	                    	          k8s-addon: dns-controller.addons.k8s.io
  	                    	          k8s-app: dns-controller
  	                    	+         version: v1.12.0
  	                    	-         version: v1.11.0
  	                    	      spec:
  	                    	        containers:
  	                    	...
  	                    	          - --zone=*/*
  	                    	          - -v=2
  	                    	+         image: kope/dns-controller:1.12.0
  	                    	-         image: kope/dns-controller:1.11.0
  	                    	          name: dns-controller
  	                    	          resources:
  	                    	...
  	                    	

  ManagedFile/!!!REDACTED!!!-addons-dns-controller.addons.k8s.io-pre-k8s-1.6
  	Contents            
  	                    	...
  	                    	      k8s-addon: dns-controller.addons.k8s.io
  	                    	      k8s-app: dns-controller
  	                    	+     version: v1.12.0
  	                    	-     version: v1.11.0
  	                    	    name: dns-controller
  	                    	    namespace: kube-system
  	                    	...
  	                    	          k8s-addon: dns-controller.addons.k8s.io
  	                    	          k8s-app: dns-controller
  	                    	+         version: v1.12.0
  	                    	-         version: v1.11.0
  	                    	      spec:
  	                    	        containers:
  	                    	...
  	                    	          - --zone=*/*
  	                    	          - -v=2
  	                    	+         image: kope/dns-controller:1.12.0
  	                    	-         image: kope/dns-controller:1.11.0
  	                    	          name: dns-controller
  	                    	          resources:
  	                    	...
  	                    	

  ManagedFile/!!!REDACTED!!!-addons-networking.weave-k8s-1.7
  	Contents            
  	                    	...
  	                    	          - name: WEAVE_MTU
  	                    	            value: "8192"
  	                    	+         image: weaveworks/weave-kube:2.5.1
  	                    	-         image: weaveworks/weave-kube:2.5.0
  	                    	          livenessProbe:
  	                    	            httpGet:
  	                    	...
  	                    	                apiVersion: v1
  	                    	                fieldPath: spec.nodeName
  	                    	+         image: weaveworks/weave-npc:2.5.1
  	                    	-         image: weaveworks/weave-npc:2.5.0
  	                    	          name: weave-npc
  	                    	          resources:
  	                    	...
  	                    	

  ManagedFile/!!!REDACTED!!!-addons-networking.weave-k8s-1.8
  	Contents            
  	                    	...
  	                    	          - name: NO_MASQ_LOCAL
  	                    	            value: "1"
  	                    	+         image: weaveworks/weave-kube:2.5.1
  	                    	-         image: weaveworks/weave-kube:2.5.0
  	                    	          livenessProbe:
  	                    	            httpGet:
  	                    	...
  	                    	                apiVersion: v1
  	                    	                fieldPath: spec.nodeName
  	                    	+         image: weaveworks/weave-npc:2.5.1
  	                    	-         image: weaveworks/weave-npc:2.5.0
  	                    	          name: weave-npc
  	                    	          resources:
  	                    	...

When I ssh to one of the masters and look in the journal the only error I see is:

Sep 10 08:25:56 ip-10-192-63-198 nodeup[834]: I0910 08:25:56.381489     834 s3fs.go:220] Reading file "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/pki/issued/etcd/keyset.yaml"
Sep 10 08:25:56 ip-10-192-63-198 nodeup[834]: W0910 08:25:56.390124     834 main.go:142] got error running nodeup (will retry in 30s): error building loader: certificate "etcd" not found

This error is repeated amongst other log lines over and over in the systemctl journal.

If I check the S3 bucket, the pki/issues/etcd/keyset.yaml file/object does not exist.

kubernetes/kops

Answer questions jim-barber-he

Additional information. The systemd journal is showing this group of operations being retried over and over on the master host:

nodeup[834]: I0910 09:40:51.042716     834 files.go:100] Hash matched for "/var/cache/nodeup/sha1:81a4ca8471fbb5980d0e5faac49473c21712bc8a_https___s3-ap-southeast-2_amazonaws_com_travisci-builds-public_kubernetes_custom-builds_v1_13_3-he_1_bin_linux_amd64_kubelet": sha1:81a4ca8471fbb5980d0e5faac49473c21712bc8a
nodeup[834]: I0910 09:40:51.042775     834 assetstore.go:203] added asset "kubelet" for &{"/var/cache/nodeup/sha1:81a4ca8471fbb5980d0e5faac49473c21712bc8a_https___s3-ap-southeast-2_amazonaws_com_travisci-builds-public_kubernetes_custom-builds_v1_13_3-he_1_bin_linux_amd64_kubelet"}
nodeup[834]: I0910 09:40:51.099936     834 files.go:100] Hash matched for "/var/cache/nodeup/sha1:5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7_https___s3-ap-southeast-2_amazonaws_com_travisci-builds-public_kubernetes_custom-builds_v1_13_3-he_1_bin_linux_amd64_kubectl": sha1:5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7
nodeup[834]: I0910 09:40:51.100002     834 assetstore.go:203] added asset "kubectl" for &{"/var/cache/nodeup/sha1:5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7_https___s3-ap-southeast-2_amazonaws_com_travisci-builds-public_kubernetes_custom-builds_v1_13_3-he_1_bin_linux_amd64_kubectl"}
nodeup[834]: I0910 09:40:51.124597     834 files.go:100] Hash matched for "/var/cache/nodeup/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz": sha1:52e9d2de8a5f927307d9397308735658ee44ab8d
nodeup[834]: I0910 09:40:51.124628     834 assetstore.go:203] added asset "cni-plugins-amd64-v0.7.5.tgz" for &{"/var/cache/nodeup/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz"}
nodeup[834]: I0910 09:40:51.124772     834 assetstore.go:313] added asset "bridge" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/bridge"}
nodeup[834]: I0910 09:40:51.124793     834 assetstore.go:313] added asset "dhcp" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/dhcp"}
nodeup[834]: I0910 09:40:51.124808     834 assetstore.go:313] added asset "flannel" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/flannel"}
nodeup[834]: I0910 09:40:51.124823     834 assetstore.go:313] added asset "host-device" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/host-device"}
nodeup[834]: I0910 09:40:51.124837     834 assetstore.go:313] added asset "host-local" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/host-local"}
nodeup[834]: I0910 09:40:51.124851     834 assetstore.go:313] added asset "ipvlan" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/ipvlan"}
nodeup[834]: I0910 09:40:51.124877     834 assetstore.go:313] added asset "loopback" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/loopback"}
nodeup[834]: I0910 09:40:51.124891     834 assetstore.go:313] added asset "macvlan" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/macvlan"}
nodeup[834]: I0910 09:40:51.124905     834 assetstore.go:313] added asset "portmap" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/portmap"}
nodeup[834]: I0910 09:40:51.124918     834 assetstore.go:313] added asset "ptp" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/ptp"}
nodeup[834]: I0910 09:40:51.124935     834 assetstore.go:313] added asset "sample" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/sample"}
nodeup[834]: I0910 09:40:51.124950     834 assetstore.go:313] added asset "tuning" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/tuning"}
nodeup[834]: I0910 09:40:51.124963     834 assetstore.go:313] added asset "vlan" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/vlan"}
nodeup[834]: I0910 09:40:51.126001     834 files.go:100] Hash matched for "/var/cache/nodeup/sha1:e59df8727c65c3a36d18421deaa711888a1b2515_https___!!!REDACTED!!!-kops_s3_amazonaws_com_kops_dev_linux_amd64_utils_tar_gz": sha1:e59df8727c65c3a36d18421deaa711888a1b2515
nodeup[834]: I0910 09:40:51.126015     834 assetstore.go:203] added asset "utils.tar.gz" for &{"/var/cache/nodeup/sha1:e59df8727c65c3a36d18421deaa711888a1b2515_https___!!!REDACTED!!!-kops_s3_amazonaws_com_kops_dev_linux_amd64_utils_tar_gz"}
nodeup[834]: I0910 09:40:51.126068     834 assetstore.go:313] added asset "conntrack" for &{"/var/cache/nodeup/extracted/sha1:e59df8727c65c3a36d18421deaa711888a1b2515_https___!!!REDACTED!!!-kops_s3_amazonaws_com_kops_dev_linux_amd64_utils_tar_gz/utils/conntrack"}
nodeup[834]: I0910 09:40:51.126084     834 assetstore.go:313] added asset "socat" for &{"/var/cache/nodeup/extracted/sha1:e59df8727c65c3a36d18421deaa711888a1b2515_https___!!!REDACTED!!!-kops_s3_amazonaws_com_kops_dev_linux_amd64_utils_tar_gz/utils/socat"}
nodeup[834]: I0910 09:40:51.126105     834 s3fs.go:220] Reading file "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/cluster.spec"
nodeup[834]: I0910 09:40:51.283487     834 s3fs.go:220] Reading file "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/instancegroup/master-ap-southeast-2a"
nodeup[834]: I0910 09:40:51.293038     834 context.go:159] Performing HTTP request: GET http://169.254.169.254/latest//meta-data/local-hostname
nodeup[834]: I0910 09:40:51.293818     834 command.go:456] Using hostname from AWS metadata service: ip-10-192-63-198.ap-southeast-2.compute.internal
nodeup[834]: I0910 09:40:51.293844     834 context.go:159] Performing HTTP request: GET http://169.254.169.254/latest//meta-data/local-hostname
nodeup[834]: I0910 09:40:51.294499     834 command.go:456] Using hostname from AWS metadata service: ip-10-192-63-198.ap-southeast-2.compute.internal
nodeup[834]: I0910 09:40:51.294515     834 context.go:159] Performing HTTP request: GET http://169.254.169.254/latest//meta-data/local-hostname
nodeup[834]: I0910 09:40:51.295128     834 command.go:456] Using hostname from AWS metadata service: ip-10-192-63-198.ap-southeast-2.compute.internal
nodeup[834]: I0910 09:40:51.295181     834 command.go:180] Config tags: [_automatic_upgrades _aws]
nodeup[834]: I0910 09:40:51.295191     834 command.go:181] OS tags: [_debian_family _systemd]
nodeup[834]: I0910 09:40:51.295198     834 command.go:193] Building SecretStore at "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/secrets"
nodeup[834]: I0910 09:40:51.295268     834 command.go:205] Building KeyStore at "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/pki"
nodeup[834]: I0910 09:40:51.295287     834 versions.go:79] Kubernetes version "https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/" string matched to 1.13.0
nodeup[834]: I0910 09:40:51.295297     834 command.go:582] Doing modprobe for module br_netfilter
nodeup[834]: I0910 09:40:51.296432     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades"
nodeup[834]: I0910 09:40:51.296450     834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/_automatic_upgrades"
nodeup[834]: I0910 09:40:51.296458     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family"
nodeup[834]: I0910 09:40:51.296467     834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/_automatic_upgrades/_debian_family"
nodeup[834]: I0910 09:40:51.296474     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files"
nodeup[834]: I0910 09:40:51.296484     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files/etc"
nodeup[834]: I0910 09:40:51.296494     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files/etc/apt"
nodeup[834]: I0910 09:40:51.296506     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files/etc/apt/apt.conf.d"
nodeup[834]: I0910 09:40:51.296516     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files/etc/apt/apt.conf.d/20auto-upgrades"
nodeup[834]: I0910 09:40:51.296526     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/packages"
nodeup[834]: I0910 09:40:51.296536     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/packages/unattended-upgrades"
nodeup[834]: I0910 09:40:51.296544     834 tree_walker.go:98] visit "nodeup/docker"
nodeup[834]: I0910 09:40:51.296553     834 tree_walker.go:98] visit "nodeup/docker/_systemd"
nodeup[834]: I0910 09:40:51.296618     834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/docker/_systemd"
nodeup[834]: I0910 09:40:51.296625     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family"
nodeup[834]: I0910 09:40:51.296635     834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/docker/_systemd/_debian_family"
nodeup[834]: I0910 09:40:51.296642     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files"
nodeup[834]: I0910 09:40:51.296651     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt"
nodeup[834]: I0910 09:40:51.296661     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes"
nodeup[834]: I0910 09:40:51.296671     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers"
nodeup[834]: I0910 09:40:51.296686     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-prestart.meta"
nodeup[834]: I0910 09:40:51.296742     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-healthcheck"
nodeup[834]: I0910 09:40:51.296802     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-healthcheck.meta"
nodeup[834]: I0910 09:40:51.296859     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-prestart"
nodeup[834]: I0910 09:40:51.296882     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/services"
nodeup[834]: I0910 09:40:51.296895     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/services/docker-healthcheck.service"
nodeup[834]: I0910 09:40:51.296904     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/services/docker-healthcheck.timer"
nodeup[834]: I0910 09:40:51.296930     834 tree_walker.go:98] visit "nodeup/ntp"
nodeup[834]: I0910 09:40:51.296942     834 tree_walker.go:98] visit "nodeup/ntp/_aws"
nodeup[834]: I0910 09:40:51.296950     834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/ntp/_aws"
nodeup[834]: I0910 09:40:51.296956     834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family"
nodeup[834]: I0910 09:40:51.296966     834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/ntp/_aws/_debian_family"
nodeup[834]: I0910 09:40:51.296973     834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family/services"
nodeup[834]: I0910 09:40:51.296982     834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family/services/ntp"
nodeup[834]: I0910 09:40:51.296990     834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family/packages"
nodeup[834]: I0910 09:40:51.296998     834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family/packages/ntp"
nodeup[834]: I0910 09:40:51.297006     834 tree_walker.go:98] visit "nodeup/ntp/_aws/_rhel_family"
nodeup[834]: I0910 09:40:51.297015     834 tree_walker.go:121] Skipping directory "nodeup/ntp/_aws/_rhel_family" as tag "_rhel_family" not present
nodeup[834]: I0910 09:40:51.297022     834 tree_walker.go:98] visit "nodeup/resources"
nodeup[834]: I0910 09:40:51.297032     834 tree_walker.go:98] visit "nodeup/resources/_lyft_vpc_cni"
nodeup[834]: I0910 09:40:51.297040     834 tree_walker.go:121] Skipping directory "nodeup/resources/_lyft_vpc_cni" as tag "_lyft_vpc_cni" not present
nodeup[834]: I0910 09:40:51.297046     834 tree_walker.go:98] visit "nodeup/top"
nodeup[834]: I0910 09:40:51.297056     834 tree_walker.go:98] visit "nodeup/top/_debian_family"
nodeup[834]: I0910 09:40:51.297063     834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/top/_debian_family"
nodeup[834]: I0910 09:40:51.297069     834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages"
nodeup[834]: I0910 09:40:51.297097     834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/curl"
nodeup[834]: I0910 09:40:51.297107     834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/nfs-common"
nodeup[834]: I0910 09:40:51.297116     834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/python-apt"
nodeup[834]: I0910 09:40:51.297124     834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/socat"
nodeup[834]: I0910 09:40:51.297131     834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/apt-transport-https"
nodeup[834]: I0910 09:40:51.297139     834 tree_walker.go:98] visit "nodeup/top/_redhat_family"
nodeup[834]: I0910 09:40:51.297148     834 tree_walker.go:121] Skipping directory "nodeup/top/_redhat_family" as tag "_redhat_family" not present
nodeup[834]: I0910 09:40:51.297154     834 tree_walker.go:98] visit "nodeup/top/_ubuntu"
nodeup[834]: I0910 09:40:51.297162     834 tree_walker.go:121] Skipping directory "nodeup/top/_ubuntu" as tag "_ubuntu" not present
nodeup[834]: I0910 09:40:51.297173     834 tree_walker.go:98] visit "nodeup/ntp"
nodeup[834]: I0910 09:40:51.297181     834 tree_walker.go:98] visit "nodeup/ntp/_aws"
nodeup[834]: I0910 09:40:51.297189     834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/ntp/_aws"
nodeup[834]: I0910 09:40:51.297195     834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family"
nodeup[834]: I0910 09:40:51.297203     834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/ntp/_aws/_debian_family"
nodeup[834]: I0910 09:40:51.297210     834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family/packages"
nodeup[834]: I0910 09:40:51.297218     834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family/packages/ntp"
nodeup[834]: I0910 09:40:51.297257     834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family/services"
nodeup[834]: I0910 09:40:51.297270     834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family/services/ntp"
nodeup[834]: I0910 09:40:51.297291     834 tree_walker.go:98] visit "nodeup/ntp/_aws/_rhel_family"
nodeup[834]: I0910 09:40:51.297301     834 tree_walker.go:121] Skipping directory "nodeup/ntp/_aws/_rhel_family" as tag "_rhel_family" not present
nodeup[834]: I0910 09:40:51.297307     834 tree_walker.go:98] visit "nodeup/resources"
nodeup[834]: I0910 09:40:51.297315     834 tree_walker.go:98] visit "nodeup/resources/_lyft_vpc_cni"
nodeup[834]: I0910 09:40:51.297323     834 tree_walker.go:121] Skipping directory "nodeup/resources/_lyft_vpc_cni" as tag "_lyft_vpc_cni" not present
nodeup[834]: I0910 09:40:51.297329     834 tree_walker.go:98] visit "nodeup/top"
nodeup[834]: I0910 09:40:51.297338     834 tree_walker.go:98] visit "nodeup/top/_debian_family"
nodeup[834]: I0910 09:40:51.297347     834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/top/_debian_family"
nodeup[834]: I0910 09:40:51.297354     834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages"
nodeup[834]: I0910 09:40:51.297367     834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/socat"
nodeup[834]: I0910 09:40:51.297385     834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/apt-transport-https"
nodeup[834]: I0910 09:40:51.297426     834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/curl"
nodeup[834]: I0910 09:40:51.297446     834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/nfs-common"
nodeup[834]: I0910 09:40:51.297466     834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/python-apt"
nodeup[834]: I0910 09:40:51.297486     834 tree_walker.go:98] visit "nodeup/top/_redhat_family"
nodeup[834]: I0910 09:40:51.297496     834 tree_walker.go:121] Skipping directory "nodeup/top/_redhat_family" as tag "_redhat_family" not present
nodeup[834]: I0910 09:40:51.297503     834 tree_walker.go:98] visit "nodeup/top/_ubuntu"
nodeup[834]: I0910 09:40:51.297511     834 tree_walker.go:121] Skipping directory "nodeup/top/_ubuntu" as tag "_ubuntu" not present
nodeup[834]: I0910 09:40:51.297516     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades"
nodeup[834]: I0910 09:40:51.297525     834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/_automatic_upgrades"
nodeup[834]: I0910 09:40:51.297531     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family"
nodeup[834]: I0910 09:40:51.297539     834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/_automatic_upgrades/_debian_family"
nodeup[834]: I0910 09:40:51.297546     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files"
nodeup[834]: I0910 09:40:51.297573     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files/etc"
nodeup[834]: I0910 09:40:51.297584     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files/etc/apt"
nodeup[834]: I0910 09:40:51.297593     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files/etc/apt/apt.conf.d"
nodeup[834]: I0910 09:40:51.297604     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files/etc/apt/apt.conf.d/20auto-upgrades"
nodeup[834]: I0910 09:40:51.297618     834 loader.go:257] path "nodeup/_automatic_upgrades/_debian_family/files/etc/apt/apt.conf.d/20auto-upgrades" -> task File: "/etc/apt/apt.conf.d/20auto-upgrades"
nodeup[834]: I0910 09:40:51.297629     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/packages"
nodeup[834]: I0910 09:40:51.297639     834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/packages/unattended-upgrades"
nodeup[834]: I0910 09:40:51.297660     834 tree_walker.go:98] visit "nodeup/docker"
nodeup[834]: I0910 09:40:51.297670     834 tree_walker.go:98] visit "nodeup/docker/_systemd"
nodeup[834]: I0910 09:40:51.297679     834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/docker/_systemd"
nodeup[834]: I0910 09:40:51.297685     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family"
nodeup[834]: I0910 09:40:51.297694     834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/docker/_systemd/_debian_family"
nodeup[834]: I0910 09:40:51.297702     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files"
nodeup[834]: I0910 09:40:51.297711     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt"
nodeup[834]: I0910 09:40:51.297738     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes"
nodeup[834]: I0910 09:40:51.297748     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers"
nodeup[834]: I0910 09:40:51.297763     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-healthcheck"
nodeup[834]: I0910 09:40:51.297820     834 loader.go:257] path "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-healthcheck" -> task File: "/opt/kubernetes/helpers/docker-healthcheck"
nodeup[834]: I0910 09:40:51.297833     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-healthcheck.meta"
nodeup[834]: I0910 09:40:51.297915     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-prestart"
nodeup[834]: I0910 09:40:51.297966     834 loader.go:257] path "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-prestart" -> task File: "/opt/kubernetes/helpers/docker-prestart"
nodeup[834]: I0910 09:40:51.297978     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-prestart.meta"
nodeup[834]: I0910 09:40:51.298025     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/services"
nodeup[834]: I0910 09:40:51.298039     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/services/docker-healthcheck.service"
nodeup[834]: I0910 09:40:51.298092     834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/services/docker-healthcheck.timer"
nodeup[834]: I0910 09:40:51.298127     834 update_service.go:42] UpdatePolicy not set in Cluster Spec; skipping creation of update-service
nodeup[834]: I0910 09:40:51.298137     834 task.go:98] task *nodetasks.File does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.298149     834 task.go:98] task *nodetasks.Package does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.298155     834 task.go:98] task *nodetasks.Package does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.298160     834 task.go:98] task *nodetasks.Package does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.298165     834 task.go:98] task *nodetasks.Package does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.298170     834 task.go:98] task *nodetasks.Package does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.298201     834 docker.go:902] Built service manifest "docker"
nodeup[834]: [Unit]
nodeup[834]: Description=Docker Application Container Engine
nodeup[834]: Documentation=https://docs.docker.com
nodeup[834]: After=network.target docker.socket
nodeup[834]: Requires=docker.socket
nodeup[834]: [Service]
nodeup[834]: Type=notify
nodeup[834]: EnvironmentFile=/etc/sysconfig/docker
nodeup[834]: EnvironmentFile=/etc/environment
nodeup[834]: ExecStart=/usr/bin/dockerd -H fd:// "$DOCKER_OPTS"
nodeup[834]: ExecReload=/bin/kill -s HUP $MAINPID
nodeup[834]: KillMode=process
nodeup[834]: TimeoutStartSec=0
nodeup[834]: LimitNOFILE=1048576
nodeup[834]: LimitNPROC=1048576
nodeup[834]: LimitCORE=infinity
nodeup[834]: TasksMax=infinity
nodeup[834]: Restart=always
nodeup[834]: RestartSec=2s
nodeup[834]: StartLimitInterval=0
nodeup[834]: Delegate=yes
nodeup[834]: ExecStartPre=/opt/kubernetes/helpers/docker-prestart
nodeup[834]: [Install]
nodeup[834]: WantedBy=multi-user.target
nodeup[834]: I0910 09:40:51.298226     834 task.go:98] task *nodetasks.Service does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.298240     834 build_flags.go:50] ignoring non-field:
nodeup[834]: I0910 09:40:51.298248     834 build_flags.go:50] ignoring non-field:
nodeup[834]: I0910 09:40:51.298257     834 build_flags.go:50] ignoring non-field: .AuthorizationPlugins
nodeup[834]: I0910 09:40:51.298269     834 build_flags.go:50] ignoring non-field: .DefaultUlimit
nodeup[834]: I0910 09:40:51.298277     834 build_flags.go:50] ignoring non-field: .Hosts
nodeup[834]: I0910 09:40:51.298291     834 build_flags.go:50] ignoring non-field: .LiveRestore
nodeup[834]: I0910 09:40:51.298304     834 build_flags.go:50] ignoring non-field: .MTU
nodeup[834]: I0910 09:40:51.298312     834 build_flags.go:50] ignoring non-field: .RegistryMirrors
nodeup[834]: I0910 09:40:51.298321     834 build_flags.go:50] ignoring non-field: .StorageOpts
nodeup[834]: I0910 09:40:51.298330     834 build_flags.go:55] not writing field with no flag tag: .Version
nodeup[834]: I0910 09:40:51.298336     834 build_flags.go:50] ignoring non-field: .Version
nodeup[834]: I0910 09:40:51.298342     834 build_flags.go:50] ignoring non-field: .Version
nodeup[834]: I0910 09:40:51.298350     834 task.go:98] task *nodetasks.File does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.298361     834 s3fs.go:220] Reading file "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/pki/issued/ca/keyset.yaml"
nodeup[834]: I0910 09:40:51.308739     834 s3fs.go:220] Reading file "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/pki/issued/kops/keyset.yaml"
nodeup[834]: I0910 09:40:51.319676     834 s3fs.go:220] Reading file "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/pki/private/kops/keyset.yaml"
nodeup[834]: I0910 09:40:51.328794     834 task.go:98] task *nodetasks.File does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.328812     834 s3fs.go:220] Reading file "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/pki/issued/etcd/keyset.yaml"
nodeup[834]: W0910 09:40:51.334638     834 main.go:142] got error running nodeup (will retry in 30s): error building loader: certificate "etcd" not found
useful!

Related questions

Unable to use a local filesystem state store hot 2
Kops 1.12-beta.2 won't/can't bring up etcd server, manager or kube-api hot 1
kube controller manager refuses to connect after upgrading from 1.10.6 to 1.11.7 hot 1
Missing kops controller support for cloudproviders hot 1
InstanceGroup not found (for etcd ap-southeast-2a/main): "ap-southeast-2a" hot 1
Rolling-update fails due to calico-node with 1.12.0-beta.2 hot 1
Kubelet Unable To Apply Reserved Cgroup Limits because Cgroup does not exist hot 1
etcd3 and kube-apiserver fail on terraform apply after terraform destroying w/ kops generated config hot 1
Couldn't find key etcd_endpoints in ConfigMap kube-system/calico-config hot 1
Protokube has sustained cpu usage above 100% hot 1
Allow just one instance type in mixedInstancesPolicy hot 1
kubectl command: Unable to connect to the server: EOF hot 1
DNS record for public API address not updated hot 1
etcd3 and kube-apiserver fail on terraform apply after terraform destroying w/ kops generated config hot 1
Issues encountered deploying to OpenStack hot 1
source:https://uonfu.com/
Github User Rank List