Ask questionsUpgrade from Kops 1.11 to 1.12 has failed.
1. What kops version are you running? The command kops version, will display
this information.
Version 1.12.3 (git-e55205471)
2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
A custom version of v1.13.3 with additional patches from PRs that were merged later.
3. What cloud provider are you using?
AWS
4. What commands did you run? What is the simplest way to reproduce this issue?
kops update cluster --yes
kops rolling-update cluster --cloudonly --instance-group-roles master --master-interval=1s --yes
5. What happened after the commands executed?
The master all shutdown at once, and then new ones started up. All kubectl commands fail with:
Unable to connect to the server: EOF
The load balancer for the API points at the new instances but shows them as all being OutOfService.
I can ssh to the masters and only ssh (TCP:22) and RPC (TCP:111) are listening. Port 443 has nothing bound to it.
6. What did you expect to happen?
The upgrade to work of course ;)
7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.
apiVersion: kops/v1alpha2
kind: Cluster
metadata:
creationTimestamp: 2018-10-02T08:14:39Z
name: !!!REDACTED!!!
spec:
additionalNetworkCIDRs:
- 10.192.0.0/16
additionalPolicies:
master: |
[
!!!REDACTED!!!
]
node: |
[
!!!REDACTED!!!
]
api:
loadBalancer:
sslCertificate: arn:aws:acm:ap-southeast-2:!!!REDACTED!!!:certificate/be0f5d8e-c88a-402f-a952-1e7fb0dc2977
type: Public
authorization:
rbac: {}
channel: stable
cloudLabels:
he:Application: kubernetes
cloudProvider: aws
configBase: s3://he-kube-hetest-cluster-state/!!!REDACTED!!!
dnsZone: !!!REDACTED!!!
docker:
storage: overlay2
version: 18.06.2
etcdClusters:
- etcdMembers:
- encryptedVolume: true
instanceGroup: master-ap-southeast-2a
kmsKeyId: arn:aws:kms:ap-southeast-2:!!!REDACTED!!!:key/398c0727-a3f3-4239-a1da-b8356d320dcc
name: a
- encryptedVolume: true
instanceGroup: master-ap-southeast-2b
kmsKeyId: arn:aws:kms:ap-southeast-2:!!!REDACTED!!!:key/398c0727-a3f3-4239-a1da-b8356d320dcc
name: b
- encryptedVolume: true
instanceGroup: master-ap-southeast-2c
kmsKeyId: arn:aws:kms:ap-southeast-2:!!!REDACTED!!!:key/398c0727-a3f3-4239-a1da-b8356d320dcc
name: c
name: main
version: 3.2.24
- etcdMembers:
- encryptedVolume: true
instanceGroup: master-ap-southeast-2a
kmsKeyId: arn:aws:kms:ap-southeast-2:!!!REDACTED!!!:key/398c0727-a3f3-4239-a1da-b8356d320dcc
name: a
- encryptedVolume: true
instanceGroup: master-ap-southeast-2b
kmsKeyId: arn:aws:kms:ap-southeast-2:!!!REDACTED!!!:key/398c0727-a3f3-4239-a1da-b8356d320dcc
name: b
- encryptedVolume: true
instanceGroup: master-ap-southeast-2c
kmsKeyId: arn:aws:kms:ap-southeast-2:!!!REDACTED!!!:key/398c0727-a3f3-4239-a1da-b8356d320dcc
name: c
name: events
version: 3.2.24
iam:
allowContainerRegistry: true
legacy: false
kubeAPIServer:
oidcClientID: !!!REDACTED!!!
oidcIssuerURL: https://accounts.google.com
oidcUsernameClaim: email
kubeDNS:
provider: CoreDNS
kubelet:
allowedUnsafeSysctls:
- net.ipv4.tcp_keepalive_time
anonymousAuth: false
clusterDNS: 169.254.20.10
kubernetesApiAccess:
- !!!REDACTED!!!
- !!!REDACTED!!!
kubernetesVersion: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/
masterInternalName: api.internal.!!!REDACTED!!!
masterPublicName: api.!!!REDACTED!!!
networkCIDR: 10.200.0.0/16
networkID: vpc-8bf3b9ef
networking:
cni: {}
weave:
mtu: 8192
noMasqLocal: 1
nonMasqueradeCIDR: 100.64.0.0/10
sshAccess:
- 127.0.0.1/32
sshKeyName: kube-hetest
subnets:
- cidr: 10.192.32.0/19
egress: nat-08de68609cdf6cc58
name: ap-southeast-2a
type: Private
zone: ap-southeast-2a
- cidr: 10.192.64.0/19
egress: nat-00ef20ab2a9408b7c
name: ap-southeast-2b
type: Private
zone: ap-southeast-2b
- cidr: 10.192.96.0/19
egress: nat-0032baadf115cfbde
name: ap-southeast-2c
type: Private
zone: ap-southeast-2c
- cidr: 10.192.0.0/22
name: utility-ap-southeast-2a
type: Utility
zone: ap-southeast-2a
- cidr: 10.192.4.0/22
name: utility-ap-southeast-2b
type: Utility
zone: ap-southeast-2b
- cidr: 10.192.8.0/22
name: utility-ap-southeast-2c
type: Utility
zone: ap-southeast-2c
topology:
dns:
type: Private
masters: private
nodes: private
8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.
9. Anything else do we need to know?
The cluster was already running etcd3 prior to the upgrade, but in on-TLS mode.
The upgrade showed that it was going to perform the following changes:
Using cluster from kubectl context: !!!REDACTED!!!
*********************************************************************************
A new kubernetes version is available: 1.13.10
Upgrading is recommended (try kops upgrade cluster)
More information: https://github.com/kubernetes/kops/blob/master/permalinks/upgrade_k8s.md#1.13.10
*********************************************************************************
W0910 16:22:11.338804 13946 urls.go:91] Using base url from KOPS_BASE_URL env var: "https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/"
I0910 16:22:12.208716 13946 dns.go:92] Private DNS: skipping DNS validation
I0910 16:22:12.475590 13946 executor.go:103] Tasks: 0 done / 135 total; 51 can run
I0910 16:22:13.700247 13946 executor.go:103] Tasks: 51 done / 135 total; 30 can run
I0910 16:22:14.956002 13946 executor.go:103] Tasks: 81 done / 135 total; 38 can run
I0910 16:22:17.239992 13946 executor.go:103] Tasks: 119 done / 135 total; 13 can run
I0910 16:22:17.535428 13946 dnsname.go:111] AliasTarget for "api.!!!REDACTED!!!." is "api-!!!REDACTED!!!.ap-southeast-2.elb.amazonaws.com."
I0910 16:22:17.705135 13946 executor.go:103] Tasks: 132 done / 135 total; 3 can run
I0910 16:22:17.949346 13946 executor.go:103] Tasks: 135 done / 135 total; 0 can run
Will create resources:
Keypair/etcd-clients-ca
Subject cn=etcd-clients-ca
Type ca
Format v1alpha2
Keypair/etcd-manager-ca-events
Subject cn=etcd-manager-ca-events
Type ca
Format v1alpha2
Keypair/etcd-manager-ca-main
Subject cn=etcd-manager-ca-main
Type ca
Format v1alpha2
Keypair/etcd-peers-ca-events
Subject cn=etcd-peers-ca-events
Type ca
Format v1alpha2
Keypair/etcd-peers-ca-main
Subject cn=etcd-peers-ca-main
Type ca
Format v1alpha2
ManagedFile/!!!REDACTED!!!-addons-coredns.addons.k8s.io-k8s-1.12
Location addons/coredns.addons.k8s.io/k8s-1.12.yaml
ManagedFile/!!!REDACTED!!!-addons-dns-controller.addons.k8s.io-k8s-1.12
Location addons/dns-controller.addons.k8s.io/k8s-1.12.yaml
ManagedFile/!!!REDACTED!!!-addons-kubelet-api.rbac.addons.k8s.io-k8s-1.9
Location addons/kubelet-api.rbac.addons.k8s.io/k8s-1.9.yaml
ManagedFile/!!!REDACTED!!!-addons-networking.weave-k8s-1.12
Location addons/networking.weave/k8s-1.12.yaml
Will modify resources:
IAMRolePolicy/masters.!!!REDACTED!!!
PolicyDocument
...
"autoscaling:DescribeAutoScalingGroups",
"autoscaling:DescribeLaunchConfigurations",
+ "autoscaling:DescribeTags",
+ "ec2:DescribeLaunchTemplateVersions"
- "autoscaling:DescribeTags"
],
"Resource": [
...
LaunchConfiguration/!!!REDACTED!!!
UserData
...
- 81a4ca8471fbb5980d0e5faac49473c21712bc8a@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubelet
- 5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubectl
+ - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
- - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
- e59df8727c65c3a36d18421deaa711888a1b2515@https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/linux/amd64/utils.tar.gz
ClusterName: !!!REDACTED!!!
...
images:
- hash: 8984887e98a0a683c12cc3aa21ec2821c831ee7f
+ sources:
+ - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
- source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
protokubeImage:
hash: ba88d42afe960f328304da3f424ac40ec2b697e8
+ name: protokube:1.12.3
- name: protokube:dev
+ sources:
+ - https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
- source: https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
__EOF_KUBE_ENV
...
LaunchConfiguration/master-ap-southeast-2a.masters.!!!REDACTED!!!
UserData
...
cloudProvider: aws
enableAdmissionPlugins:
- - Initializers
- NamespaceLifecycle
- LimitRanger
...
- 81a4ca8471fbb5980d0e5faac49473c21712bc8a@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubelet
- 5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubectl
+ - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
- - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
- e59df8727c65c3a36d18421deaa711888a1b2515@https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/linux/amd64/utils.tar.gz
ClusterName: !!!REDACTED!!!
...
images:
- hash: 8984887e98a0a683c12cc3aa21ec2821c831ee7f
+ sources:
+ - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
- source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
- hash: 303c3347fe7db7118f302f4bf15f0015f1c4d86d
+ sources:
+ - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-apiserver.tar
- source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-apiserver.tar
- hash: 029d26f968def3dd24590dbdd22091c8bcfae5fe
+ sources:
+ - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-controller-manager.tar
- source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-controller-manager.tar
- hash: 1e6495ca0a5cbe5ffcac253621da88df5a69ac31
+ sources:
+ - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-scheduler.tar
- source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-scheduler.tar
protokubeImage:
hash: ba88d42afe960f328304da3f424ac40ec2b697e8
+ name: protokube:1.12.3
- name: protokube:dev
+ sources:
+ - https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
- source: https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
__EOF_KUBE_ENV
...
LaunchConfiguration/master-ap-southeast-2b.masters.!!!REDACTED!!!
UserData
...
cloudProvider: aws
enableAdmissionPlugins:
- - Initializers
- NamespaceLifecycle
- LimitRanger
...
- 81a4ca8471fbb5980d0e5faac49473c21712bc8a@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubelet
- 5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubectl
+ - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
- - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
- e59df8727c65c3a36d18421deaa711888a1b2515@https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/linux/amd64/utils.tar.gz
ClusterName: !!!REDACTED!!!
...
images:
- hash: 8984887e98a0a683c12cc3aa21ec2821c831ee7f
+ sources:
+ - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
- source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
- hash: 303c3347fe7db7118f302f4bf15f0015f1c4d86d
+ sources:
+ - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-apiserver.tar
- source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-apiserver.tar
- hash: 029d26f968def3dd24590dbdd22091c8bcfae5fe
+ sources:
+ - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-controller-manager.tar
- source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-controller-manager.tar
- hash: 1e6495ca0a5cbe5ffcac253621da88df5a69ac31
+ sources:
+ - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-scheduler.tar
- source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-scheduler.tar
protokubeImage:
hash: ba88d42afe960f328304da3f424ac40ec2b697e8
+ name: protokube:1.12.3
- name: protokube:dev
+ sources:
+ - https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
- source: https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
__EOF_KUBE_ENV
...
LaunchConfiguration/master-ap-southeast-2c.masters.!!!REDACTED!!!
UserData
...
cloudProvider: aws
enableAdmissionPlugins:
- - Initializers
- NamespaceLifecycle
- LimitRanger
...
- 81a4ca8471fbb5980d0e5faac49473c21712bc8a@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubelet
- 5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubectl
+ - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
- - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
- e59df8727c65c3a36d18421deaa711888a1b2515@https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/linux/amd64/utils.tar.gz
ClusterName: !!!REDACTED!!!
...
images:
- hash: 8984887e98a0a683c12cc3aa21ec2821c831ee7f
+ sources:
+ - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
- source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
- hash: 303c3347fe7db7118f302f4bf15f0015f1c4d86d
+ sources:
+ - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-apiserver.tar
- source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-apiserver.tar
- hash: 029d26f968def3dd24590dbdd22091c8bcfae5fe
+ sources:
+ - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-controller-manager.tar
- source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-controller-manager.tar
- hash: 1e6495ca0a5cbe5ffcac253621da88df5a69ac31
+ sources:
+ - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-scheduler.tar
- source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-scheduler.tar
protokubeImage:
hash: ba88d42afe960f328304da3f424ac40ec2b697e8
+ name: protokube:1.12.3
- name: protokube:dev
+ sources:
+ - https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
- source: https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
__EOF_KUBE_ENV
...
LaunchConfiguration/nodes-2a.!!!REDACTED!!!
UserData
...
- 81a4ca8471fbb5980d0e5faac49473c21712bc8a@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubelet
- 5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubectl
+ - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
- - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
- e59df8727c65c3a36d18421deaa711888a1b2515@https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/linux/amd64/utils.tar.gz
ClusterName: !!!REDACTED!!!
...
images:
- hash: 8984887e98a0a683c12cc3aa21ec2821c831ee7f
+ sources:
+ - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
- source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
protokubeImage:
hash: ba88d42afe960f328304da3f424ac40ec2b697e8
+ name: protokube:1.12.3
- name: protokube:dev
+ sources:
+ - https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
- source: https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
__EOF_KUBE_ENV
...
LaunchConfiguration/nodes-2b.!!!REDACTED!!!
UserData
...
- 81a4ca8471fbb5980d0e5faac49473c21712bc8a@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubelet
- 5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubectl
+ - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
- - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
- e59df8727c65c3a36d18421deaa711888a1b2515@https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/linux/amd64/utils.tar.gz
ClusterName: !!!REDACTED!!!
...
images:
- hash: 8984887e98a0a683c12cc3aa21ec2821c831ee7f
+ sources:
+ - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
- source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
protokubeImage:
hash: ba88d42afe960f328304da3f424ac40ec2b697e8
+ name: protokube:1.12.3
- name: protokube:dev
+ sources:
+ - https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
- source: https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
__EOF_KUBE_ENV
...
LaunchConfiguration/nodes-2c.!!!REDACTED!!!
UserData
...
- 81a4ca8471fbb5980d0e5faac49473c21712bc8a@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubelet
- 5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubectl
+ - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
- - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
- e59df8727c65c3a36d18421deaa711888a1b2515@https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/linux/amd64/utils.tar.gz
ClusterName: !!!REDACTED!!!
...
images:
- hash: 8984887e98a0a683c12cc3aa21ec2821c831ee7f
+ sources:
+ - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
- source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
protokubeImage:
hash: ba88d42afe960f328304da3f424ac40ec2b697e8
+ name: protokube:1.12.3
- name: protokube:dev
+ sources:
+ - https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
- source: https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
__EOF_KUBE_ENV
...
LaunchConfiguration/nodes.!!!REDACTED!!!
UserData
...
- 81a4ca8471fbb5980d0e5faac49473c21712bc8a@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubelet
- 5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7@https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kubectl
+ - 52e9d2de8a5f927307d9397308735658ee44ab8d@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.7.5.tgz
- - d595d3ded6499a64e8dac02466e2f5f2ce257c9f@https://storage.googleapis.com/kubernetes-release/network-plugins/cni-plugins-amd64-v0.6.0.tgz
- e59df8727c65c3a36d18421deaa711888a1b2515@https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/linux/amd64/utils.tar.gz
ClusterName: !!!REDACTED!!!
...
images:
- hash: 8984887e98a0a683c12cc3aa21ec2821c831ee7f
+ sources:
+ - https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
- source: https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/bin/linux/amd64/kube-proxy.tar
protokubeImage:
hash: ba88d42afe960f328304da3f424ac40ec2b697e8
+ name: protokube:1.12.3
- name: protokube:dev
+ sources:
+ - https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
- source: https://!!!REDACTED!!!.s3.amazonaws.com/kops/dev/images/protokube.tar.gz
__EOF_KUBE_ENV
...
LoadBalancer/api.!!!REDACTED!!!
Tags {KubernetesCluster: !!!REDACTED!!!, Name: api.!!!REDACTED!!!} -> {Name: api.!!!REDACTED!!!, KubernetesCluster: !!!REDACTED!!!, kubernetes.io/cluster/!!!REDACTED!!!: owned, he:Application: kubernetes}
ManagedFile/etcd-cluster-spec-events
Contents
{
+ "memberCount": 3,
- "member_count": 3,
+ "etcdVersion": "3.2.24"
- "etcd_version": "3.2.24"
}
ManagedFile/etcd-cluster-spec-main
Contents
{
+ "memberCount": 3,
- "member_count": 3,
+ "etcdVersion": "3.2.24"
- "etcd_version": "3.2.24"
}
ManagedFile/manifests-etcdmanager-events
Contents
...
- mkfifo /tmp/pipe; (tee -a /var/log/etcd.log < /tmp/pipe & ) ; exec /etcd-manager
--backup-store=s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/backups/etcd/events
+ --client-urls=https://__name__:4002 --cluster-name=etcd-events --containerized=true
- --client-urls=http://__name__:4002 --cluster-name=etcd-events --containerized=true
+ --dns-suffix=.internal.!!!REDACTED!!! --etcd-insecure=false --grpc-port=3997 --insecure=false
- --dns-suffix=.internal.!!!REDACTED!!! --grpc-port=3997 --peer-urls=http://__name__:2381
+ --peer-urls=https://__name__:2381 --quarantine-client-urls=https://__name__:3995
- --quarantine-client-urls=http://__name__:3995 --v=8 --volume-name-tag=k8s.io/etcd/events
+ --v=6 --volume-name-tag=k8s.io/etcd/events --volume-provider=aws --volume-tag=k8s.io/etcd/events
- --volume-provider=aws --volume-tag=k8s.io/etcd/events --volume-tag=k8s.io/role/master=1
+ --volume-tag=k8s.io/role/master=1 --volume-tag=kubernetes.io/cluster/!!!REDACTED!!!=owned
+ > /tmp/pipe 2>&1
- --volume-tag=kubernetes.io/cluster/!!!REDACTED!!!=owned > /tmp/pipe 2>&1
+ image: kopeio/etcd-manager:3.0.20190801
- image: kopeio/etcd-manager:1.0.20181001
name: etcd-manager
resources:
requests:
+ cpu: 200m
+ memory: 100Mi
- cpu: 100m
securityContext:
privileged: true
...
- mountPath: /etc/hosts
name: hosts
+ - mountPath: /etc/kubernetes/pki/etcd-manager
+ name: pki
- mountPath: /var/log/etcd.log
name: varlogetcd
...
name: hosts
- hostPath:
+ path: /etc/kubernetes/pki/etcd-manager-events
+ type: DirectoryOrCreate
+ name: pki
+ - hostPath:
path: /var/log/etcd-events.log
type: FileOrCreate
...
ManagedFile/manifests-etcdmanager-main
Contents
...
- mkfifo /tmp/pipe; (tee -a /var/log/etcd.log < /tmp/pipe & ) ; exec /etcd-manager
--backup-store=s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/backups/etcd/main
+ --client-urls=https://__name__:4001 --cluster-name=etcd --containerized=true
- --client-urls=http://__name__:4001 --cluster-name=etcd --containerized=true
+ --dns-suffix=.internal.!!!REDACTED!!! --etcd-insecure=false --grpc-port=3996 --insecure=false
- --dns-suffix=.internal.!!!REDACTED!!! --grpc-port=3996 --peer-urls=http://__name__:2380
+ --peer-urls=https://__name__:2380 --quarantine-client-urls=https://__name__:3994
- --quarantine-client-urls=http://__name__:3994 --v=8 --volume-name-tag=k8s.io/etcd/main
+ --v=6 --volume-name-tag=k8s.io/etcd/main --volume-provider=aws --volume-tag=k8s.io/etcd/main
- --volume-provider=aws --volume-tag=k8s.io/etcd/main --volume-tag=k8s.io/role/master=1
+ --volume-tag=k8s.io/role/master=1 --volume-tag=kubernetes.io/cluster/!!!REDACTED!!!=owned
+ > /tmp/pipe 2>&1
- --volume-tag=kubernetes.io/cluster/!!!REDACTED!!!=owned > /tmp/pipe 2>&1
+ image: kopeio/etcd-manager:3.0.20190801
- image: kopeio/etcd-manager:1.0.20181001
name: etcd-manager
resources:
requests:
cpu: 200m
+ memory: 100Mi
securityContext:
privileged: true
...
- mountPath: /etc/hosts
name: hosts
+ - mountPath: /etc/kubernetes/pki/etcd-manager
+ name: pki
- mountPath: /var/log/etcd.log
name: varlogetcd
...
name: hosts
- hostPath:
+ path: /etc/kubernetes/pki/etcd-manager-main
+ type: DirectoryOrCreate
+ name: pki
+ - hostPath:
path: /var/log/etcd.log
type: FileOrCreate
...
ManagedFile/!!!REDACTED!!!-addons-bootstrap
Contents
...
version: 1.4.0
- id: k8s-1.6
+ kubernetesVersion: '>=1.6.0 <1.12.0'
- kubernetesVersion: '>=1.6.0'
manifest: coredns.addons.k8s.io/k8s-1.6.yaml
name: coredns.addons.k8s.io
selector:
k8s-addon: coredns.addons.k8s.io
+ version: 1.3.1-kops.3
+ - id: k8s-1.12
+ kubernetesVersion: '>=1.12.0'
+ manifest: coredns.addons.k8s.io/k8s-1.12.yaml
+ name: coredns.addons.k8s.io
+ selector:
+ k8s-addon: coredns.addons.k8s.io
+ version: 1.3.0-kops.2
- version: 1.2.6-kops.1
- id: k8s-1.8
kubernetesVersion: '>=1.8.0'
...
k8s-addon: rbac.addons.k8s.io
version: 1.8.0
+ - id: k8s-1.9
+ kubernetesVersion: '>=1.9.0'
+ manifest: kubelet-api.rbac.addons.k8s.io/k8s-1.9.yaml
+ name: kubelet-api.rbac.addons.k8s.io
+ selector:
+ k8s-addon: kubelet-api.rbac.addons.k8s.io
+ version: v0.0.1
- manifest: limit-range.addons.k8s.io/v1.5.0.yaml
name: limit-range.addons.k8s.io
...
selector:
k8s-addon: dns-controller.addons.k8s.io
+ version: 1.12.3
- version: 1.11.0
- id: k8s-1.6
+ kubernetesVersion: '>=1.6.0 <1.12.0'
- kubernetesVersion: '>=1.6.0'
manifest: dns-controller.addons.k8s.io/k8s-1.6.yaml
name: dns-controller.addons.k8s.io
selector:
k8s-addon: dns-controller.addons.k8s.io
+ version: 1.12.3
+ - id: k8s-1.12
+ kubernetesVersion: '>=1.12.0'
+ manifest: dns-controller.addons.k8s.io/k8s-1.12.yaml
+ name: dns-controller.addons.k8s.io
+ selector:
+ k8s-addon: dns-controller.addons.k8s.io
+ version: 1.12.3
- version: 1.11.0
- id: v1.7.0
kubernetesVersion: '>=1.7.0'
...
selector:
role.kubernetes.io/networking: "1"
+ version: 2.5.1-kops.1
- version: 2.5.0-kops.1
- id: k8s-1.8
+ kubernetesVersion: '>=1.8.0 <1.12.0'
- kubernetesVersion: '>=1.8.0'
manifest: networking.weave/k8s-1.8.yaml
name: networking.weave
selector:
role.kubernetes.io/networking: "1"
+ version: 2.5.1-kops.1
+ - id: k8s-1.12
+ kubernetesVersion: '>=1.12.0'
+ manifest: networking.weave/k8s-1.12.yaml
+ name: networking.weave
+ selector:
+ role.kubernetes.io/networking: "1"
+ version: 2.5.1-kops.1
- version: 2.5.0-kops.1
ManagedFile/!!!REDACTED!!!-addons-coredns.addons.k8s.io-k8s-1.6
Contents
...
---
+ apiVersion: rbac.authorization.k8s.io/v1
- apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
...
---
+ apiVersion: rbac.authorization.k8s.io/v1
- apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
...
}
prometheus :9153
+ forward . /etc/resolv.conf
- proxy . /etc/resolv.conf
loop
cache 30
...
---
+ apiVersion: apps/v1
- apiVersion: extensions/v1beta1
kind: Deployment
metadata:
...
- -conf
- /etc/coredns/Corefile
+ image: k8s.gcr.io/coredns:1.3.1
- image: k8s.gcr.io/coredns:1.2.6
imagePullPolicy: IfNotPresent
livenessProbe:
...
name: metrics
protocol: TCP
+ readinessProbe:
+ httpGet:
+ path: /health
+ port: 8080
+ scheme: HTTP
resources:
limits:
...
nodeSelector:
beta.kubernetes.io/os: linux
+ priorityClassName: system-cluster-critical
serviceAccountName: coredns
tolerations:
...
port: 53
protocol: TCP
+ - name: metrics
+ port: 9153
+ protocol: TCP
selector:
k8s-app: kube-dns
ManagedFile/!!!REDACTED!!!-addons-dns-controller.addons.k8s.io-k8s-1.6
Contents
...
k8s-addon: dns-controller.addons.k8s.io
k8s-app: dns-controller
+ version: v1.12.0
- version: v1.11.0
name: dns-controller
namespace: kube-system
...
k8s-addon: dns-controller.addons.k8s.io
k8s-app: dns-controller
+ version: v1.12.0
- version: v1.11.0
spec:
containers:
...
- --zone=*/*
- -v=2
+ image: kope/dns-controller:1.12.0
- image: kope/dns-controller:1.11.0
name: dns-controller
resources:
...
ManagedFile/!!!REDACTED!!!-addons-dns-controller.addons.k8s.io-pre-k8s-1.6
Contents
...
k8s-addon: dns-controller.addons.k8s.io
k8s-app: dns-controller
+ version: v1.12.0
- version: v1.11.0
name: dns-controller
namespace: kube-system
...
k8s-addon: dns-controller.addons.k8s.io
k8s-app: dns-controller
+ version: v1.12.0
- version: v1.11.0
spec:
containers:
...
- --zone=*/*
- -v=2
+ image: kope/dns-controller:1.12.0
- image: kope/dns-controller:1.11.0
name: dns-controller
resources:
...
ManagedFile/!!!REDACTED!!!-addons-networking.weave-k8s-1.7
Contents
...
- name: WEAVE_MTU
value: "8192"
+ image: weaveworks/weave-kube:2.5.1
- image: weaveworks/weave-kube:2.5.0
livenessProbe:
httpGet:
...
apiVersion: v1
fieldPath: spec.nodeName
+ image: weaveworks/weave-npc:2.5.1
- image: weaveworks/weave-npc:2.5.0
name: weave-npc
resources:
...
ManagedFile/!!!REDACTED!!!-addons-networking.weave-k8s-1.8
Contents
...
- name: NO_MASQ_LOCAL
value: "1"
+ image: weaveworks/weave-kube:2.5.1
- image: weaveworks/weave-kube:2.5.0
livenessProbe:
httpGet:
...
apiVersion: v1
fieldPath: spec.nodeName
+ image: weaveworks/weave-npc:2.5.1
- image: weaveworks/weave-npc:2.5.0
name: weave-npc
resources:
...
When I ssh to one of the masters and look in the journal the only error I see is:
Sep 10 08:25:56 ip-10-192-63-198 nodeup[834]: I0910 08:25:56.381489 834 s3fs.go:220] Reading file "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/pki/issued/etcd/keyset.yaml"
Sep 10 08:25:56 ip-10-192-63-198 nodeup[834]: W0910 08:25:56.390124 834 main.go:142] got error running nodeup (will retry in 30s): error building loader: certificate "etcd" not found
This error is repeated amongst other log lines over and over in the systemctl journal.
If I check the S3 bucket, the pki/issues/etcd/keyset.yaml file/object does not exist.
Answer
questions
jim-barber-he
Additional information. The systemd journal is showing this group of operations being retried over and over on the master host:
nodeup[834]: I0910 09:40:51.042716 834 files.go:100] Hash matched for "/var/cache/nodeup/sha1:81a4ca8471fbb5980d0e5faac49473c21712bc8a_https___s3-ap-southeast-2_amazonaws_com_travisci-builds-public_kubernetes_custom-builds_v1_13_3-he_1_bin_linux_amd64_kubelet": sha1:81a4ca8471fbb5980d0e5faac49473c21712bc8a
nodeup[834]: I0910 09:40:51.042775 834 assetstore.go:203] added asset "kubelet" for &{"/var/cache/nodeup/sha1:81a4ca8471fbb5980d0e5faac49473c21712bc8a_https___s3-ap-southeast-2_amazonaws_com_travisci-builds-public_kubernetes_custom-builds_v1_13_3-he_1_bin_linux_amd64_kubelet"}
nodeup[834]: I0910 09:40:51.099936 834 files.go:100] Hash matched for "/var/cache/nodeup/sha1:5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7_https___s3-ap-southeast-2_amazonaws_com_travisci-builds-public_kubernetes_custom-builds_v1_13_3-he_1_bin_linux_amd64_kubectl": sha1:5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7
nodeup[834]: I0910 09:40:51.100002 834 assetstore.go:203] added asset "kubectl" for &{"/var/cache/nodeup/sha1:5ddb7b1be23f5e4aecc8bf894d480664acfa5fc7_https___s3-ap-southeast-2_amazonaws_com_travisci-builds-public_kubernetes_custom-builds_v1_13_3-he_1_bin_linux_amd64_kubectl"}
nodeup[834]: I0910 09:40:51.124597 834 files.go:100] Hash matched for "/var/cache/nodeup/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz": sha1:52e9d2de8a5f927307d9397308735658ee44ab8d
nodeup[834]: I0910 09:40:51.124628 834 assetstore.go:203] added asset "cni-plugins-amd64-v0.7.5.tgz" for &{"/var/cache/nodeup/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz"}
nodeup[834]: I0910 09:40:51.124772 834 assetstore.go:313] added asset "bridge" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/bridge"}
nodeup[834]: I0910 09:40:51.124793 834 assetstore.go:313] added asset "dhcp" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/dhcp"}
nodeup[834]: I0910 09:40:51.124808 834 assetstore.go:313] added asset "flannel" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/flannel"}
nodeup[834]: I0910 09:40:51.124823 834 assetstore.go:313] added asset "host-device" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/host-device"}
nodeup[834]: I0910 09:40:51.124837 834 assetstore.go:313] added asset "host-local" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/host-local"}
nodeup[834]: I0910 09:40:51.124851 834 assetstore.go:313] added asset "ipvlan" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/ipvlan"}
nodeup[834]: I0910 09:40:51.124877 834 assetstore.go:313] added asset "loopback" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/loopback"}
nodeup[834]: I0910 09:40:51.124891 834 assetstore.go:313] added asset "macvlan" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/macvlan"}
nodeup[834]: I0910 09:40:51.124905 834 assetstore.go:313] added asset "portmap" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/portmap"}
nodeup[834]: I0910 09:40:51.124918 834 assetstore.go:313] added asset "ptp" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/ptp"}
nodeup[834]: I0910 09:40:51.124935 834 assetstore.go:313] added asset "sample" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/sample"}
nodeup[834]: I0910 09:40:51.124950 834 assetstore.go:313] added asset "tuning" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/tuning"}
nodeup[834]: I0910 09:40:51.124963 834 assetstore.go:313] added asset "vlan" for &{"/var/cache/nodeup/extracted/sha1:52e9d2de8a5f927307d9397308735658ee44ab8d_https___storage_googleapis_com_kubernetes-release_network-plugins_cni-plugins-amd64-v0_7_5_tgz/vlan"}
nodeup[834]: I0910 09:40:51.126001 834 files.go:100] Hash matched for "/var/cache/nodeup/sha1:e59df8727c65c3a36d18421deaa711888a1b2515_https___!!!REDACTED!!!-kops_s3_amazonaws_com_kops_dev_linux_amd64_utils_tar_gz": sha1:e59df8727c65c3a36d18421deaa711888a1b2515
nodeup[834]: I0910 09:40:51.126015 834 assetstore.go:203] added asset "utils.tar.gz" for &{"/var/cache/nodeup/sha1:e59df8727c65c3a36d18421deaa711888a1b2515_https___!!!REDACTED!!!-kops_s3_amazonaws_com_kops_dev_linux_amd64_utils_tar_gz"}
nodeup[834]: I0910 09:40:51.126068 834 assetstore.go:313] added asset "conntrack" for &{"/var/cache/nodeup/extracted/sha1:e59df8727c65c3a36d18421deaa711888a1b2515_https___!!!REDACTED!!!-kops_s3_amazonaws_com_kops_dev_linux_amd64_utils_tar_gz/utils/conntrack"}
nodeup[834]: I0910 09:40:51.126084 834 assetstore.go:313] added asset "socat" for &{"/var/cache/nodeup/extracted/sha1:e59df8727c65c3a36d18421deaa711888a1b2515_https___!!!REDACTED!!!-kops_s3_amazonaws_com_kops_dev_linux_amd64_utils_tar_gz/utils/socat"}
nodeup[834]: I0910 09:40:51.126105 834 s3fs.go:220] Reading file "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/cluster.spec"
nodeup[834]: I0910 09:40:51.283487 834 s3fs.go:220] Reading file "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/instancegroup/master-ap-southeast-2a"
nodeup[834]: I0910 09:40:51.293038 834 context.go:159] Performing HTTP request: GET http://169.254.169.254/latest//meta-data/local-hostname
nodeup[834]: I0910 09:40:51.293818 834 command.go:456] Using hostname from AWS metadata service: ip-10-192-63-198.ap-southeast-2.compute.internal
nodeup[834]: I0910 09:40:51.293844 834 context.go:159] Performing HTTP request: GET http://169.254.169.254/latest//meta-data/local-hostname
nodeup[834]: I0910 09:40:51.294499 834 command.go:456] Using hostname from AWS metadata service: ip-10-192-63-198.ap-southeast-2.compute.internal
nodeup[834]: I0910 09:40:51.294515 834 context.go:159] Performing HTTP request: GET http://169.254.169.254/latest//meta-data/local-hostname
nodeup[834]: I0910 09:40:51.295128 834 command.go:456] Using hostname from AWS metadata service: ip-10-192-63-198.ap-southeast-2.compute.internal
nodeup[834]: I0910 09:40:51.295181 834 command.go:180] Config tags: [_automatic_upgrades _aws]
nodeup[834]: I0910 09:40:51.295191 834 command.go:181] OS tags: [_debian_family _systemd]
nodeup[834]: I0910 09:40:51.295198 834 command.go:193] Building SecretStore at "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/secrets"
nodeup[834]: I0910 09:40:51.295268 834 command.go:205] Building KeyStore at "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/pki"
nodeup[834]: I0910 09:40:51.295287 834 versions.go:79] Kubernetes version "https://s3-ap-southeast-2.amazonaws.com/travisci-builds-public/kubernetes/custom-builds/v1.13.3-he.1/" string matched to 1.13.0
nodeup[834]: I0910 09:40:51.295297 834 command.go:582] Doing modprobe for module br_netfilter
nodeup[834]: I0910 09:40:51.296432 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades"
nodeup[834]: I0910 09:40:51.296450 834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/_automatic_upgrades"
nodeup[834]: I0910 09:40:51.296458 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family"
nodeup[834]: I0910 09:40:51.296467 834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/_automatic_upgrades/_debian_family"
nodeup[834]: I0910 09:40:51.296474 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files"
nodeup[834]: I0910 09:40:51.296484 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files/etc"
nodeup[834]: I0910 09:40:51.296494 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files/etc/apt"
nodeup[834]: I0910 09:40:51.296506 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files/etc/apt/apt.conf.d"
nodeup[834]: I0910 09:40:51.296516 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files/etc/apt/apt.conf.d/20auto-upgrades"
nodeup[834]: I0910 09:40:51.296526 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/packages"
nodeup[834]: I0910 09:40:51.296536 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/packages/unattended-upgrades"
nodeup[834]: I0910 09:40:51.296544 834 tree_walker.go:98] visit "nodeup/docker"
nodeup[834]: I0910 09:40:51.296553 834 tree_walker.go:98] visit "nodeup/docker/_systemd"
nodeup[834]: I0910 09:40:51.296618 834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/docker/_systemd"
nodeup[834]: I0910 09:40:51.296625 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family"
nodeup[834]: I0910 09:40:51.296635 834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/docker/_systemd/_debian_family"
nodeup[834]: I0910 09:40:51.296642 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files"
nodeup[834]: I0910 09:40:51.296651 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt"
nodeup[834]: I0910 09:40:51.296661 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes"
nodeup[834]: I0910 09:40:51.296671 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers"
nodeup[834]: I0910 09:40:51.296686 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-prestart.meta"
nodeup[834]: I0910 09:40:51.296742 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-healthcheck"
nodeup[834]: I0910 09:40:51.296802 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-healthcheck.meta"
nodeup[834]: I0910 09:40:51.296859 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-prestart"
nodeup[834]: I0910 09:40:51.296882 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/services"
nodeup[834]: I0910 09:40:51.296895 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/services/docker-healthcheck.service"
nodeup[834]: I0910 09:40:51.296904 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/services/docker-healthcheck.timer"
nodeup[834]: I0910 09:40:51.296930 834 tree_walker.go:98] visit "nodeup/ntp"
nodeup[834]: I0910 09:40:51.296942 834 tree_walker.go:98] visit "nodeup/ntp/_aws"
nodeup[834]: I0910 09:40:51.296950 834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/ntp/_aws"
nodeup[834]: I0910 09:40:51.296956 834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family"
nodeup[834]: I0910 09:40:51.296966 834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/ntp/_aws/_debian_family"
nodeup[834]: I0910 09:40:51.296973 834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family/services"
nodeup[834]: I0910 09:40:51.296982 834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family/services/ntp"
nodeup[834]: I0910 09:40:51.296990 834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family/packages"
nodeup[834]: I0910 09:40:51.296998 834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family/packages/ntp"
nodeup[834]: I0910 09:40:51.297006 834 tree_walker.go:98] visit "nodeup/ntp/_aws/_rhel_family"
nodeup[834]: I0910 09:40:51.297015 834 tree_walker.go:121] Skipping directory "nodeup/ntp/_aws/_rhel_family" as tag "_rhel_family" not present
nodeup[834]: I0910 09:40:51.297022 834 tree_walker.go:98] visit "nodeup/resources"
nodeup[834]: I0910 09:40:51.297032 834 tree_walker.go:98] visit "nodeup/resources/_lyft_vpc_cni"
nodeup[834]: I0910 09:40:51.297040 834 tree_walker.go:121] Skipping directory "nodeup/resources/_lyft_vpc_cni" as tag "_lyft_vpc_cni" not present
nodeup[834]: I0910 09:40:51.297046 834 tree_walker.go:98] visit "nodeup/top"
nodeup[834]: I0910 09:40:51.297056 834 tree_walker.go:98] visit "nodeup/top/_debian_family"
nodeup[834]: I0910 09:40:51.297063 834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/top/_debian_family"
nodeup[834]: I0910 09:40:51.297069 834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages"
nodeup[834]: I0910 09:40:51.297097 834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/curl"
nodeup[834]: I0910 09:40:51.297107 834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/nfs-common"
nodeup[834]: I0910 09:40:51.297116 834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/python-apt"
nodeup[834]: I0910 09:40:51.297124 834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/socat"
nodeup[834]: I0910 09:40:51.297131 834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/apt-transport-https"
nodeup[834]: I0910 09:40:51.297139 834 tree_walker.go:98] visit "nodeup/top/_redhat_family"
nodeup[834]: I0910 09:40:51.297148 834 tree_walker.go:121] Skipping directory "nodeup/top/_redhat_family" as tag "_redhat_family" not present
nodeup[834]: I0910 09:40:51.297154 834 tree_walker.go:98] visit "nodeup/top/_ubuntu"
nodeup[834]: I0910 09:40:51.297162 834 tree_walker.go:121] Skipping directory "nodeup/top/_ubuntu" as tag "_ubuntu" not present
nodeup[834]: I0910 09:40:51.297173 834 tree_walker.go:98] visit "nodeup/ntp"
nodeup[834]: I0910 09:40:51.297181 834 tree_walker.go:98] visit "nodeup/ntp/_aws"
nodeup[834]: I0910 09:40:51.297189 834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/ntp/_aws"
nodeup[834]: I0910 09:40:51.297195 834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family"
nodeup[834]: I0910 09:40:51.297203 834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/ntp/_aws/_debian_family"
nodeup[834]: I0910 09:40:51.297210 834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family/packages"
nodeup[834]: I0910 09:40:51.297218 834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family/packages/ntp"
nodeup[834]: I0910 09:40:51.297257 834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family/services"
nodeup[834]: I0910 09:40:51.297270 834 tree_walker.go:98] visit "nodeup/ntp/_aws/_debian_family/services/ntp"
nodeup[834]: I0910 09:40:51.297291 834 tree_walker.go:98] visit "nodeup/ntp/_aws/_rhel_family"
nodeup[834]: I0910 09:40:51.297301 834 tree_walker.go:121] Skipping directory "nodeup/ntp/_aws/_rhel_family" as tag "_rhel_family" not present
nodeup[834]: I0910 09:40:51.297307 834 tree_walker.go:98] visit "nodeup/resources"
nodeup[834]: I0910 09:40:51.297315 834 tree_walker.go:98] visit "nodeup/resources/_lyft_vpc_cni"
nodeup[834]: I0910 09:40:51.297323 834 tree_walker.go:121] Skipping directory "nodeup/resources/_lyft_vpc_cni" as tag "_lyft_vpc_cni" not present
nodeup[834]: I0910 09:40:51.297329 834 tree_walker.go:98] visit "nodeup/top"
nodeup[834]: I0910 09:40:51.297338 834 tree_walker.go:98] visit "nodeup/top/_debian_family"
nodeup[834]: I0910 09:40:51.297347 834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/top/_debian_family"
nodeup[834]: I0910 09:40:51.297354 834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages"
nodeup[834]: I0910 09:40:51.297367 834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/socat"
nodeup[834]: I0910 09:40:51.297385 834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/apt-transport-https"
nodeup[834]: I0910 09:40:51.297426 834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/curl"
nodeup[834]: I0910 09:40:51.297446 834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/nfs-common"
nodeup[834]: I0910 09:40:51.297466 834 tree_walker.go:98] visit "nodeup/top/_debian_family/packages/python-apt"
nodeup[834]: I0910 09:40:51.297486 834 tree_walker.go:98] visit "nodeup/top/_redhat_family"
nodeup[834]: I0910 09:40:51.297496 834 tree_walker.go:121] Skipping directory "nodeup/top/_redhat_family" as tag "_redhat_family" not present
nodeup[834]: I0910 09:40:51.297503 834 tree_walker.go:98] visit "nodeup/top/_ubuntu"
nodeup[834]: I0910 09:40:51.297511 834 tree_walker.go:121] Skipping directory "nodeup/top/_ubuntu" as tag "_ubuntu" not present
nodeup[834]: I0910 09:40:51.297516 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades"
nodeup[834]: I0910 09:40:51.297525 834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/_automatic_upgrades"
nodeup[834]: I0910 09:40:51.297531 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family"
nodeup[834]: I0910 09:40:51.297539 834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/_automatic_upgrades/_debian_family"
nodeup[834]: I0910 09:40:51.297546 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files"
nodeup[834]: I0910 09:40:51.297573 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files/etc"
nodeup[834]: I0910 09:40:51.297584 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files/etc/apt"
nodeup[834]: I0910 09:40:51.297593 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files/etc/apt/apt.conf.d"
nodeup[834]: I0910 09:40:51.297604 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/files/etc/apt/apt.conf.d/20auto-upgrades"
nodeup[834]: I0910 09:40:51.297618 834 loader.go:257] path "nodeup/_automatic_upgrades/_debian_family/files/etc/apt/apt.conf.d/20auto-upgrades" -> task File: "/etc/apt/apt.conf.d/20auto-upgrades"
nodeup[834]: I0910 09:40:51.297629 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/packages"
nodeup[834]: I0910 09:40:51.297639 834 tree_walker.go:98] visit "nodeup/_automatic_upgrades/_debian_family/packages/unattended-upgrades"
nodeup[834]: I0910 09:40:51.297660 834 tree_walker.go:98] visit "nodeup/docker"
nodeup[834]: I0910 09:40:51.297670 834 tree_walker.go:98] visit "nodeup/docker/_systemd"
nodeup[834]: I0910 09:40:51.297679 834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/docker/_systemd"
nodeup[834]: I0910 09:40:51.297685 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family"
nodeup[834]: I0910 09:40:51.297694 834 tree_walker.go:125] Descending into directory, as tag is present: "nodeup/docker/_systemd/_debian_family"
nodeup[834]: I0910 09:40:51.297702 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files"
nodeup[834]: I0910 09:40:51.297711 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt"
nodeup[834]: I0910 09:40:51.297738 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes"
nodeup[834]: I0910 09:40:51.297748 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers"
nodeup[834]: I0910 09:40:51.297763 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-healthcheck"
nodeup[834]: I0910 09:40:51.297820 834 loader.go:257] path "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-healthcheck" -> task File: "/opt/kubernetes/helpers/docker-healthcheck"
nodeup[834]: I0910 09:40:51.297833 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-healthcheck.meta"
nodeup[834]: I0910 09:40:51.297915 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-prestart"
nodeup[834]: I0910 09:40:51.297966 834 loader.go:257] path "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-prestart" -> task File: "/opt/kubernetes/helpers/docker-prestart"
nodeup[834]: I0910 09:40:51.297978 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/files/opt/kubernetes/helpers/docker-prestart.meta"
nodeup[834]: I0910 09:40:51.298025 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/services"
nodeup[834]: I0910 09:40:51.298039 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/services/docker-healthcheck.service"
nodeup[834]: I0910 09:40:51.298092 834 tree_walker.go:98] visit "nodeup/docker/_systemd/_debian_family/services/docker-healthcheck.timer"
nodeup[834]: I0910 09:40:51.298127 834 update_service.go:42] UpdatePolicy not set in Cluster Spec; skipping creation of update-service
nodeup[834]: I0910 09:40:51.298137 834 task.go:98] task *nodetasks.File does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.298149 834 task.go:98] task *nodetasks.Package does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.298155 834 task.go:98] task *nodetasks.Package does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.298160 834 task.go:98] task *nodetasks.Package does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.298165 834 task.go:98] task *nodetasks.Package does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.298170 834 task.go:98] task *nodetasks.Package does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.298201 834 docker.go:902] Built service manifest "docker"
nodeup[834]: [Unit]
nodeup[834]: Description=Docker Application Container Engine
nodeup[834]: Documentation=https://docs.docker.com
nodeup[834]: After=network.target docker.socket
nodeup[834]: Requires=docker.socket
nodeup[834]: [Service]
nodeup[834]: Type=notify
nodeup[834]: EnvironmentFile=/etc/sysconfig/docker
nodeup[834]: EnvironmentFile=/etc/environment
nodeup[834]: ExecStart=/usr/bin/dockerd -H fd:// "$DOCKER_OPTS"
nodeup[834]: ExecReload=/bin/kill -s HUP $MAINPID
nodeup[834]: KillMode=process
nodeup[834]: TimeoutStartSec=0
nodeup[834]: LimitNOFILE=1048576
nodeup[834]: LimitNPROC=1048576
nodeup[834]: LimitCORE=infinity
nodeup[834]: TasksMax=infinity
nodeup[834]: Restart=always
nodeup[834]: RestartSec=2s
nodeup[834]: StartLimitInterval=0
nodeup[834]: Delegate=yes
nodeup[834]: ExecStartPre=/opt/kubernetes/helpers/docker-prestart
nodeup[834]: [Install]
nodeup[834]: WantedBy=multi-user.target
nodeup[834]: I0910 09:40:51.298226 834 task.go:98] task *nodetasks.Service does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.298240 834 build_flags.go:50] ignoring non-field:
nodeup[834]: I0910 09:40:51.298248 834 build_flags.go:50] ignoring non-field:
nodeup[834]: I0910 09:40:51.298257 834 build_flags.go:50] ignoring non-field: .AuthorizationPlugins
nodeup[834]: I0910 09:40:51.298269 834 build_flags.go:50] ignoring non-field: .DefaultUlimit
nodeup[834]: I0910 09:40:51.298277 834 build_flags.go:50] ignoring non-field: .Hosts
nodeup[834]: I0910 09:40:51.298291 834 build_flags.go:50] ignoring non-field: .LiveRestore
nodeup[834]: I0910 09:40:51.298304 834 build_flags.go:50] ignoring non-field: .MTU
nodeup[834]: I0910 09:40:51.298312 834 build_flags.go:50] ignoring non-field: .RegistryMirrors
nodeup[834]: I0910 09:40:51.298321 834 build_flags.go:50] ignoring non-field: .StorageOpts
nodeup[834]: I0910 09:40:51.298330 834 build_flags.go:55] not writing field with no flag tag: .Version
nodeup[834]: I0910 09:40:51.298336 834 build_flags.go:50] ignoring non-field: .Version
nodeup[834]: I0910 09:40:51.298342 834 build_flags.go:50] ignoring non-field: .Version
nodeup[834]: I0910 09:40:51.298350 834 task.go:98] task *nodetasks.File does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.298361 834 s3fs.go:220] Reading file "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/pki/issued/ca/keyset.yaml"
nodeup[834]: I0910 09:40:51.308739 834 s3fs.go:220] Reading file "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/pki/issued/kops/keyset.yaml"
nodeup[834]: I0910 09:40:51.319676 834 s3fs.go:220] Reading file "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/pki/private/kops/keyset.yaml"
nodeup[834]: I0910 09:40:51.328794 834 task.go:98] task *nodetasks.File does not implement HasLifecycle
nodeup[834]: I0910 09:40:51.328812 834 s3fs.go:220] Reading file "s3://!!!REDACTED!!!-cluster-state/!!!REDACTED!!!/pki/issued/etcd/keyset.yaml"
nodeup[834]: W0910 09:40:51.334638 834 main.go:142] got error running nodeup (will retry in 30s): error building loader: certificate "etcd" not found
Related questions