profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/divyenpatel/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Divyen Patel divyenpatel @vmware Palo Alto, CA

vmware-tanzu/vm-operator-api 45

A client API for the VM Operator project, designed to allow for integration with vSphere 7 with Kubernetes

divyenpatel/kubernetes.github.io 1

Kubernetes website and documentation repo:

divyenpatel/cloud-provider-vsphere 0

Kubernetes Cloud Provider for vSphere https://cloud-provider-vsphere.sigs.k8s.io

divyenpatel/community 0

Kubernetes community content

divyenpatel/csi-translation-lib 0

Staging repo for CSI Migration/Translation libraries

divyenpatel/docs 0

Documentation for CSI integration with Kubernetes

divyenpatel/external-attacher 0

Sidecar container that watches Kubernetes VolumeAttachment objects and triggers ControllerPublish/Unpublish against a CSI endpoint

divyenpatel/external-provisioner 0

Sidecar container that watches Kubernetes PersistentVolumeClaim objects and triggers CreateVolume/DeleteVolume against a CSI endpoint

divyenpatel/external-resizer 0

Sidecar container that watches Kubernetes PersistentVolumeClaims objects and triggers controller side expansion operation against a CSI endpoint

pull request commentkubernetes-sigs/vsphere-csi-driver

Add healthz port for CSI controller and use emptyDir for CSI socket file

Started WCP block pre-checkin pipeline... Build Number: 67

chethanv28

comment created time in a minute

PR merged kubernetes-sigs/vsphere-csi-driver

Reviewers
Cherry-pick PR (#1013) to rel 2.3 - Add check for CSI volume spec in PVUpdated method approved cncf-cla: yes lgtm ok-to-test size/XS
  • Add check for volume spec to avoid NPE

<!-- Thanks for sending a pull request! -->

What this PR does / why we need it: This PR is cherry-picking https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1013 from master

Release note: <!-- Steps to write your release note:

  1. Use the release-note-* labels to set the release note state (if you have access)
  2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write NONE. -->
Cherry-pick PR (#1013) - Add check for CSI volume spec in PVUpdated method 
+5 -2

3 comments

1 changed file

chethanv28

pr closed time in 34 minutes

push eventkubernetes-sigs/vsphere-csi-driver

Chethan Venkatesh

commit sha d132c75fd52df9cc233ca412fe2fb30c626f3b00

Add check for CSI volume spec in PVUpdated method (#1013) (#1015) * Add check for volume spec to avoid NPE * Rename the variable used for checking if it is a dynamic PV * Add an extra check to ensure code block is for static CSI volume

view details

push time in 34 minutes

pull request commentkubernetes-sigs/vsphere-csi-driver

Add benchmarks for LogNewError methods

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1014#pullrequestreview-692063365" title="Approved">divyenpatel</a>, <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1014#" title="Author self-approved">zhelongp</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

zhelongp

comment created time in 37 minutes

pull request commentkubernetes-sigs/vsphere-csi-driver

Add benchmarks for LogNewError methods

@zhelongp do you intend to run these tests every time as part of PR validation?

I see tests are not executed for this PR - https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_vsphere-csi-driver/1014/pull-vsphere-csi-driver-unit-test/1408113957868146688/build-log.txt

Thanks for the review, @divyenpatel . These tests are benchmarks, which do not run by default (e.g. via make unit). I do not mean to trigger those tests automatically for any PR, which would be a waste of resource. It is good for now.

zhelongp

comment created time in an hour

pull request commentkubernetes-sigs/vsphere-csi-driver

Cherry-pick PR (#1013) to rel 2.3 - Add check for CSI volume spec in PVUpdated method

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1015#" title="Author self-approved">chethanv28</a>, <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1015#pullrequestreview-692038956" title="Approved">divyenpatel</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

  • OWNERS [chethanv28,divyenpatel]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

chethanv28

comment created time in an hour

pull request commentkubernetes-sigs/vsphere-csi-driver

Clean up white spaces in utils.go

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1009#issuecomment-867818520" title="Approved">divyenpatel</a>, <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1009#" title="Author self-approved">zhelongp</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

zhelongp

comment created time in an hour

pull request commentkubernetes-sigs/vsphere-csi-driver

Clean up white spaces in migration.go

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1007#issuecomment-867818379" title="Approved">divyenpatel</a>, <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1007#" title="Author self-approved">zhelongp</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

zhelongp

comment created time in an hour

pull request commentkubernetes-sigs/vsphere-csi-driver

Clean up white spaces in cns-lib

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1006#issuecomment-867817981" title="Approved">divyenpatel</a>, <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1006#" title="Author self-approved">zhelongp</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

zhelongp

comment created time in an hour

pull request commentkubernetes-sigs/vsphere-csi-driver

Clean white spaces in triggercsifullsync_controller.go

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1002#issuecomment-867817438" title="Approved">divyenpatel</a>, <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1002#" title="Author self-approved">zhelongp</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

zhelongp

comment created time in an hour

pull request commentkubernetes-sigs/vsphere-csi-driver

Add healthz port for CSI controller and use emptyDir for CSI socket file

Build ID: 12
WCP build status: FAILURE 
Stage before exit: null 
Jenkins E2E Test Results: 
Ran 14 of 195 Specs in 2822.795 seconds
FAIL! -- 9 Passed | 5 Failed | 0 Pending | 181 Skipped
--- FAIL: TestE2E (2822.91s)
FAIL

Ginkgo ran 1 suite in 47m42.798649647s
Test Suite Failed
make: Leaving directory `/home/worker/workspace/csi-wcp-pre-check-in-trials/Results/12/vsphere-csi-driver`

chethanv28

comment created time in an hour

pull request commentkubernetes-sigs/vsphere-csi-driver

Add healthz port for CSI controller and use emptyDir for CSI socket file

Started WCP block pre-checkin pipeline... Build Number: 12

chethanv28

comment created time in 2 hours

PR closed kubernetes-sigs/vsphere-csi-driver

Reviewers
Update CSI Spec to v1.4.0 approved cncf-cla: yes needs-rebase size/L

<!-- Thanks for sending a pull request! -->

What this PR does / why we need it: This PR is updating CSI Spec to v1.4.0 The latest CSI spec has few limitations relaxed, such as volume name length and node id. The list of improvements can be found here: https://github.com/container-storage-interface/spec/releases/tag/v1.4.0

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:

Release note: <!-- Steps to write your release note:

  1. Use the release-note-* labels to set the release note state (if you have access)
  2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write NONE. -->
Update CSI Spec to v1.4.0
+27 -138

15 comments

5 changed files

chethanv28

pr closed time in 3 hours

pull request commentkubernetes-sigs/vsphere-csi-driver

Update CSI Spec to v1.4.0

Not needed since CSI spec has been updated along with go-csi removal work items

chethanv28

comment created time in 3 hours

pull request commentkubernetes-sigs/vsphere-csi-driver

Cherry-pick PR (#1013) to rel 2.3 - Add check for CSI volume spec in PVUpdated method

cc: @divyenpatel Can you help merge this cherry-pick PR

chethanv28

comment created time in 3 hours

MemberEvent

issue openedkubernetes-sigs/vsphere-csi-driver

no datastores found to create file volume // empty nodeUUID

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened: If I try to create a pvc I receive the error "no datastores found to create file volume" or sometimes I get "failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager". I think the main reason is that the node UUIDs are empty for some reason.

In the logs of the vsphere-csi-controller container I found the following entries:

{"level":"info","time":"2021-06-23T17:14:39.377247663Z","caller":"logger/logger.go:37","msg":"Setting default log level to :\"PRODUCTION\""}
{"level":"info","time":"2021-06-23T17:14:39.377457762Z","caller":"vsphere-csi/main.go:54","msg":"Version : v2.2.1","TraceId":"de2d5508-f9e7-4659-b426-b4dad397629b"}
{"level":"info","time":"2021-06-23T17:14:39.379655703Z","caller":"logger/logger.go:37","msg":"Setting default log level to :\"PRODUCTION\""}
{"level":"info","time":"2021-06-23T17:14:39.384817331Z","caller":"k8sorchestrator/k8sorchestrator.go:122","msg":"Initializing k8sOrchestratorInstance","TraceId":"29b35dba-cd21-4c6c-8e85-33c96158773c"}
{"level":"info","time":"2021-06-23T17:14:39.385285147Z","caller":"kubernetes/kubernetes.go:91","msg":"k8s client using in-cluster config","TraceId":"29b35dba-cd21-4c6c-8e85-33c96158773c"}
{"level":"info","time":"2021-06-23T17:14:39.388263712Z","caller":"kubernetes/kubernetes.go:304","msg":"Setting client QPS to 100.000000 and Burst to 100.","TraceId":"29b35dba-cd21-4c6c-8e85-33c96158773c"}
{"level":"info","time":"2021-06-23T17:14:39.42948329Z","caller":"k8sorchestrator/k8sorchestrator.go:234","msg":"New internal feature states values stored successfully: map[csi-auth-check:true csi-migration:false online-volume-extend:true]","TraceId":"29b35dba-cd21-4c6c-8e85-33c96158773c"}
{"level":"info","time":"2021-06-23T17:14:39.429767408Z","caller":"k8sorchestrator/k8sorchestrator.go:148","msg":"k8sOrchestratorInstance initialized","TraceId":"29b35dba-cd21-4c6c-8e85-33c96158773c"}
{"level":"info","time":"2021-06-23T17:14:39.431977646Z","caller":"config/config.go:302","msg":"No Net Permissions given in Config. Using default permissions.","TraceId":"29b35dba-cd21-4c6c-8e85-33c96158773c"}
{"level":"info","time":"2021-06-23T17:14:39.4326364Z","caller":"vanilla/controller.go:79","msg":"Initializing CNS controller","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.432688233Z","caller":"vsphere/utils.go:132","msg":"Defaulting timeout for vCenter Client to 5 minutes","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.432712115Z","caller":"vsphere/virtualcentermanager.go:72","msg":"Initializing defaultVirtualCenterManager...","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.43272412Z","caller":"vsphere/virtualcentermanager.go:74","msg":"Successfully initialized defaultVirtualCenterManager","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.432742534Z","caller":"vsphere/virtualcentermanager.go:118","msg":"Successfully registered VC \"****.vcsa.01.***\"","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.432755507Z","caller":"volume/manager.go:111","msg":"Initializing new volume.defaultManager...","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.44806408Z","caller":"k8sorchestrator/k8sorchestrator.go:273","msg":"New feature states values from \"internal-feature-states.csi.vsphere.vmware.com\" stored successfully: map[csi-auth-check:true csi-migration:false online-volume-extend:true]","TraceId":"0b7eac10-7e14-4bb8-984d-f5939891039b"}
{"level":"info","time":"2021-06-23T17:14:39.550063804Z","caller":"vsphere/virtualcenter.go:169","msg":"New session ID for '***\\***' = 5230da53-6841-4bfe-8bb1-f72e8a4b4c19","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.550125363Z","caller":"node/manager.go:75","msg":"Initializing node.defaultManager...","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.550137425Z","caller":"node/manager.go:79","msg":"node.defaultManager initialized","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.550153808Z","caller":"kubernetes/kubernetes.go:91","msg":"k8s client using in-cluster config","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.550717029Z","caller":"kubernetes/kubernetes.go:304","msg":"Setting client QPS to 100.000000 and Burst to 100.","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.558542125Z","caller":"vanilla/controller.go:142","msg":"CSIAuthCheck feature is enabled, loading AuthorizationService","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.558581435Z","caller":"common/authmanager.go:76","msg":"Initializing authorization service...","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.558592813Z","caller":"common/authmanager.go:86","msg":"authorization service initialized","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.55863975Z","caller":"vanilla/controller.go:197","msg":"Adding watch on path: \"/etc/cloud\"","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.558674841Z","caller":"service/service.go:106","msg":"configured: \"csi.vsphere.vmware.com\" with clusterFlavor: \"VANILLA\" and mode: \"controller\"","TraceId":"29b35dba-cd21-4c6c-8e85-33c96158773c"}
time="2021-06-23T17:14:39Z" level=info msg="identity service registered"
time="2021-06-23T17:14:39Z" level=info msg="controller service registered"
time="2021-06-23T17:14:39Z" level=info msg=serving endpoint="unix:///csi/csi.sock"
{"level":"info","time":"2021-06-23T17:14:39.559394784Z","caller":"common/authmanager.go:156","msg":"auth manager: ComputeDatastoreMapForBlockVolumes enter"}
{"level":"info","time":"2021-06-23T17:14:39.562355896Z","caller":"common/authmanager.go:166","msg":"auth manager: ComputeDatastoreMapForFileVolumes enter"}
{"level":"info","time":"2021-06-23T17:14:39.572984454Z","caller":"vanilla/controller.go:215","msg":"Starting the http server to expose Prometheus metrics..","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.62088948Z","caller":"vanilla/controller.go:1069","msg":"ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"79725ae3-386e-4f14-96bb-ea19f3e686f0"}
{"level":"info","time":"2021-06-23T17:14:39.624403738Z","caller":"node/manager.go:103","msg":"Successfully registered node: \"k8s-worker-1-01.***.***\" with nodeUUID \"\"","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110"}
{"level":"info","time":"2021-06-23T17:14:39.62478924Z","caller":"vsphere/virtualmachine.go:123","msg":"Initiating asynchronous datacenter listing with uuid ","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110"}
{"level":"info","time":"2021-06-23T17:14:39.631695568Z","caller":"vanilla/controller.go:1069","msg":"ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"92dee706-80d2-4856-b752-e3b17e8cfb2a"}
{"level":"info","time":"2021-06-23T17:14:39.646434025Z","caller":"vsphere/datacenter.go:145","msg":"Publishing datacenter Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***]","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110"}
{"level":"info","time":"2021-06-23T17:14:39.646610493Z","caller":"vsphere/virtualmachine.go:160","msg":"AsyncGetAllDatacenters with uuid  sent a dc Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***]","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110"}
{"level":"error","time":"2021-06-23T17:14:39.652400548Z","caller":"vsphere/datacenter.go:98","msg":"Couldn't find VM given uuid ","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.(*Datacenter).GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/datacenter.go:98\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID.func1\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:161"}
{"level":"warn","time":"2021-06-23T17:14:39.65263142Z","caller":"vsphere/virtualmachine.go:164","msg":"Couldn't find VM given uuid  on DC Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***] with err: virtual machine wasn't found, continuing search","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110"}
{"level":"error","time":"2021-06-23T17:14:39.652723572Z","caller":"vsphere/virtualmachine.go:191","msg":"Returning VM not found err for UUID ","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:191\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:117\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:73"}
{"level":"error","time":"2021-06-23T17:14:39.652877692Z","caller":"node/manager.go:119","msg":"Couldn't find VM instance with nodeUUID , failed to discover with err: virtual machine wasn't found","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:119\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:73"}
{"level":"error","time":"2021-06-23T17:14:39.653048955Z","caller":"node/manager.go:106","msg":"failed to discover VM with uuid: \"\" for node: \"k8s-worker-1-01.***.***\"","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:106\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:73"}
{"level":"warn","time":"2021-06-23T17:14:39.653167976Z","caller":"vanilla/nodes.go:67","msg":"failed to register node:\"k8s-worker-1-01.***.***\". err=virtual machine wasn't found","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110"}
{"level":"info","time":"2021-06-23T17:14:39.653677111Z","caller":"node/manager.go:103","msg":"Successfully registered node: \"k8s-master-1-01.***.***\" with nodeUUID \"\"","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15"}
{"level":"info","time":"2021-06-23T17:14:39.655341817Z","caller":"vsphere/virtualmachine.go:123","msg":"Initiating asynchronous datacenter listing with uuid ","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15"}
{"level":"info","time":"2021-06-23T17:14:39.680181413Z","caller":"vsphere/datacenter.go:145","msg":"Publishing datacenter Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***]","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15"}
{"level":"info","time":"2021-06-23T17:14:39.680636976Z","caller":"vsphere/virtualmachine.go:160","msg":"AsyncGetAllDatacenters with uuid  sent a dc Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***]","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15"}
{"level":"error","time":"2021-06-23T17:14:39.687238379Z","caller":"vsphere/datacenter.go:98","msg":"Couldn't find VM given uuid ","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.(*Datacenter).GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/datacenter.go:98\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID.func1\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:161"}
{"level":"warn","time":"2021-06-23T17:14:39.687338753Z","caller":"vsphere/virtualmachine.go:164","msg":"Couldn't find VM given uuid  on DC Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***] with err: virtual machine wasn't found, continuing search","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15"}
{"level":"error","time":"2021-06-23T17:14:39.687361172Z","caller":"vsphere/virtualmachine.go:191","msg":"Returning VM not found err for UUID ","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:191\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:117\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:73"}
{"level":"error","time":"2021-06-23T17:14:39.687416376Z","caller":"node/manager.go:119","msg":"Couldn't find VM instance with nodeUUID , failed to discover with err: virtual machine wasn't found","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:119\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:73"}
{"level":"error","time":"2021-06-23T17:14:39.687474217Z","caller":"node/manager.go:106","msg":"failed to discover VM with uuid: \"\" for node: \"k8s-master-1-01.***.***\"","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:106\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:73"}
{"level":"warn","time":"2021-06-23T17:14:39.687530348Z","caller":"vanilla/nodes.go:67","msg":"failed to register node:\"k8s-master-1-01.***.***\". err=virtual machine wasn't found","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15"}
{"level":"info","time":"2021-06-23T17:14:39.687764295Z","caller":"node/manager.go:103","msg":"Successfully registered node: \"k8s-master-2-01.***.***\" with nodeUUID \"\"","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812"}
{"level":"info","time":"2021-06-23T17:14:39.687792615Z","caller":"vsphere/virtualmachine.go:123","msg":"Initiating asynchronous datacenter listing with uuid ","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812"}
{"level":"info","time":"2021-06-23T17:14:39.697016254Z","caller":"vsphere/datacenter.go:145","msg":"Publishing datacenter Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***]","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812"}
{"level":"info","time":"2021-06-23T17:14:39.697270216Z","caller":"vsphere/virtualmachine.go:160","msg":"AsyncGetAllDatacenters with uuid  sent a dc Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***]","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812"}
{"level":"error","time":"2021-06-23T17:14:39.706040552Z","caller":"vsphere/datacenter.go:98","msg":"Couldn't find VM given uuid ","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.(*Datacenter).GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/datacenter.go:98\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID.func1\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:161"}
{"level":"warn","time":"2021-06-23T17:14:39.706358026Z","caller":"vsphere/virtualmachine.go:164","msg":"Couldn't find VM given uuid  on DC Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***] with err: virtual machine wasn't found, continuing search","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812"}
{"level":"error","time":"2021-06-23T17:14:39.706466123Z","caller":"vsphere/virtualmachine.go:191","msg":"Returning VM not found err for UUID ","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:191\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:117\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:73"}
{"level":"error","time":"2021-06-23T17:14:39.706588692Z","caller":"node/manager.go:119","msg":"Couldn't find VM instance with nodeUUID , failed to discover with err: virtual machine wasn't found","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:119\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:73"}
{"level":"error","time":"2021-06-23T17:14:39.706664133Z","caller":"node/manager.go:106","msg":"failed to discover VM with uuid: \"\" for node: \"k8s-master-2-01.***.***\"","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:106\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:73"}
{"level":"warn","time":"2021-06-23T17:14:39.706738905Z","caller":"vanilla/nodes.go:67","msg":"failed to register node:\"k8s-master-2-01.***.***\". err=virtual machine wasn't found","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812"}
{"level":"info","time":"2021-06-23T17:14:39.706966564Z","caller":"node/manager.go:103","msg":"Successfully registered node: \"k8s-master-3-01.***.***\" with nodeUUID \"\"","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163"}
{"level":"info","time":"2021-06-23T17:14:39.707034283Z","caller":"vsphere/virtualmachine.go:123","msg":"Initiating asynchronous datacenter listing with uuid ","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163"}
{"level":"info","time":"2021-06-23T17:14:39.716086532Z","caller":"vsphere/datacenter.go:145","msg":"Publishing datacenter Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***]","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163"}
{"level":"info","time":"2021-06-23T17:14:39.716200395Z","caller":"vsphere/virtualmachine.go:160","msg":"AsyncGetAllDatacenters with uuid  sent a dc Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***]","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163"}
{"level":"error","time":"2021-06-23T17:14:39.721503507Z","caller":"vsphere/datacenter.go:98","msg":"Couldn't find VM given uuid ","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.(*Datacenter).GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/datacenter.go:98\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID.func1\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:161"}
{"level":"warn","time":"2021-06-23T17:14:39.721577793Z","caller":"vsphere/virtualmachine.go:164","msg":"Couldn't find VM given uuid  on DC Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***] with err: virtual machine wasn't found, continuing search","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163"}
{"level":"error","time":"2021-06-23T17:14:39.721597247Z","caller":"vsphere/virtualmachine.go:191","msg":"Returning VM not found err for UUID ","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:191\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:117\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:73"}
{"level":"error","time":"2021-06-23T17:14:39.721664846Z","caller":"node/manager.go:119","msg":"Couldn't find VM instance with nodeUUID , failed to discover with err: virtual machine wasn't found","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:119\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:73"}
{"level":"error","time":"2021-06-23T17:14:39.721712072Z","caller":"node/manager.go:106","msg":"failed to discover VM with uuid: \"\" for node: \"k8s-master-3-01.***.***\"","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:106\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/client-go@v0.18.5/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.5/pkg/util/wait/wait.go:73"}
{"level":"warn","time":"2021-06-23T17:14:39.721755623Z","caller":"vanilla/nodes.go:67","msg":"failed to register node:\"k8s-master-3-01.***.***\". err=virtual machine wasn't found","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163"}
{"level":"info","time":"2021-06-23T17:14:39.809397819Z","caller":"vanilla/controller.go:1069","msg":"ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"743deeba-68ef-4637-8feb-41f77bcf4aa2"}
{"level":"info","time":"2021-06-23T17:14:41.158843676Z","caller":"vanilla/controller.go:1069","msg":"ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"f87052c2-17de-4f90-a15c-81bb579b30ea"}
{"level":"info","time":"2021-06-23T17:44:39.44905192Z","caller":"k8sorchestrator/k8sorchestrator.go:293","msg":"New feature states values from \"internal-feature-states.csi.vsphere.vmware.com\" stored successfully: map[csi-auth-check:true csi-migration:false online-volume-extend:true]","TraceId":"766f18fd-94fd-4f4e-9322-49cc9a0a6931"}
{"level":"info","time":"2021-06-24T03:44:39.506774985Z","caller":"k8sorchestrator/k8sorchestrator.go:293","msg":"New feature states values from \"internal-feature-states.csi.vsphere.vmware.com\" stored successfully: map[csi-auth-check:true csi-migration:false online-volume-extend:true]","TraceId":"7ce83dbf-ab35-4619-bf74-20db0134dfa7"}
{"level":"info","time":"2021-06-24T03:55:52.000394761Z","caller":"vanilla/controller.go:1069","msg":"ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"d3acbc20-1e5f-482c-9418-7ab6ed5a3586"}
{"level":"info","time":"2021-06-24T04:00:22.916524464Z","caller":"vanilla/controller.go:1069","msg":"ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"0929c14c-9f0b-497b-ba64-6cc94766d227"}
{"level":"info","time":"2021-06-24T04:14:39.503498331Z","caller":"k8sorchestrator/k8sorchestrator.go:293","msg":"New feature states values from \"internal-feature-states.csi.vsphere.vmware.com\" stored successfully: map[csi-auth-check:true csi-migration:false online-volume-extend:true]","TraceId":"1c0113f2-8ee5-4c21-af09-d139959ee159"}
{"level":"info","time":"2021-06-24T07:14:39.49835871Z","caller":"k8sorchestrator/k8sorchestrator.go:293","msg":"New feature states values from \"internal-feature-states.csi.vsphere.vmware.com\" stored successfully: map[csi-auth-check:true csi-migration:false online-volume-extend:true]","TraceId":"dcb58eb4-399b-4dba-88ec-a045ae7ddcd8"}
{"level":"info","time":"2021-06-24T07:35:39.517171263Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-6ef59515-e2ae-41e7-b506-1d729b1d7a9d CapacityRange:required_bytes:1073741824  VolumeCapabilities:[block:<> access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.517223011Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.517234728Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.531998917Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"error","time":"2021-06-24T07:35:39.532040552Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:39.53213164Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.532144002Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.546474831Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"error","time":"2021-06-24T07:35:39.546523548Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:39.546625075Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.546638392Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.554171081Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"error","time":"2021-06-24T07:35:39.554211076Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:39.554299506Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.554312086Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.560668452Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"error","time":"2021-06-24T07:35:39.560713428Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:35:39.560837658Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:35:39.560890413Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:40.562424896Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-6ef59515-e2ae-41e7-b506-1d729b1d7a9d CapacityRange:required_bytes:1073741824  VolumeCapabilities:[block:<> access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.562636273Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.562677201Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.569787296Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"error","time":"2021-06-24T07:35:40.569949985Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:40.570085971Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.570182137Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.576102603Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"error","time":"2021-06-24T07:35:40.576217073Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:40.576357269Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.576398489Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.582433874Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"error","time":"2021-06-24T07:35:40.582515763Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:40.582601509Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.58261388Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.597313293Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"error","time":"2021-06-24T07:35:40.59737204Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:35:40.597450419Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:35:40.598421879Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:42.600379178Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-6ef59515-e2ae-41e7-b506-1d729b1d7a9d CapacityRange:required_bytes:1073741824  VolumeCapabilities:[block:<> access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.600427404Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.600452334Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.616491396Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"error","time":"2021-06-24T07:35:42.616531646Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"da350121-741b-4af5-8306-e67e5585228e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:42.616640356Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.616652322Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.642052498Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"error","time":"2021-06-24T07:35:42.642100282Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"da350121-741b-4af5-8306-e67e5585228e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:42.642198273Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.642210124Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.663517728Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"error","time":"2021-06-24T07:35:42.663647082Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"da350121-741b-4af5-8306-e67e5585228e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:42.663780904Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.663831735Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.676333554Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"error","time":"2021-06-24T07:35:42.676517963Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"da350121-741b-4af5-8306-e67e5585228e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:35:42.676756044Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"da350121-741b-4af5-8306-e67e5585228e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:35:42.676845696Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"da350121-741b-4af5-8306-e67e5585228e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:46.680692707Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-6ef59515-e2ae-41e7-b506-1d729b1d7a9d CapacityRange:required_bytes:1073741824  VolumeCapabilities:[block:<> access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.680752247Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.680768598Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.696105192Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"error","time":"2021-06-24T07:35:46.696263941Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:46.69645028Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.696494523Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.718759556Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"error","time":"2021-06-24T07:35:46.719198657Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:46.71949466Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.719514382Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.732399548Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"error","time":"2021-06-24T07:35:46.732440905Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:46.732517863Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.732529239Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.744510827Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"error","time":"2021-06-24T07:35:46.744584083Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:35:46.744693871Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:35:46.744809166Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:02.263369522Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-650ee0f9-6939-4599-a6c7-8104563cd094 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.263475695Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.267481858Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.274630112Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"error","time":"2021-06-24T07:36:02.274781601Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:02.274953389Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.27501909Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.285935614Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"error","time":"2021-06-24T07:36:02.28597211Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:02.286063464Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.286075098Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.291910385Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"error","time":"2021-06-24T07:36:02.29194672Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:02.292022881Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.292033973Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.297246316Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"error","time":"2021-06-24T07:36:02.297358462Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:02.297466229Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:02.297520002Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:03.299757413Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-650ee0f9-6939-4599-a6c7-8104563cd094 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.299876238Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.299933615Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.312288413Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"error","time":"2021-06-24T07:36:03.312342912Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:03.312449124Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.312460949Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.319093193Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"error","time":"2021-06-24T07:36:03.31914127Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:03.319241097Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.31925331Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.324215441Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"error","time":"2021-06-24T07:36:03.324250327Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:03.324335731Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.324346784Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.338195221Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"error","time":"2021-06-24T07:36:03.338236454Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:03.338311386Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:03.338383805Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:05.34414704Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-650ee0f9-6939-4599-a6c7-8104563cd094 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.34438201Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.3444321Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.358437025Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"error","time":"2021-06-24T07:36:05.358589374Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:05.358739173Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.358794526Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.368311311Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"error","time":"2021-06-24T07:36:05.368373198Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:05.368458019Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.368470212Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.373226807Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"error","time":"2021-06-24T07:36:05.373386349Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:05.373562725Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.373613147Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.379161465Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"error","time":"2021-06-24T07:36:05.379293022Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:05.379469257Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:05.379603273Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:09.383009985Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-650ee0f9-6939-4599-a6c7-8104563cd094 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.383075308Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.383449201Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.392964784Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"error","time":"2021-06-24T07:36:09.393004611Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:09.393077416Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.393088711Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.398475254Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"error","time":"2021-06-24T07:36:09.398604675Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:09.398807996Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.398864397Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.403210165Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"error","time":"2021-06-24T07:36:09.403332577Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:09.403453893Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.403491092Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.408316534Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"error","time":"2021-06-24T07:36:09.408441619Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:09.408525195Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:09.408598948Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:17.411103961Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-650ee0f9-6939-4599-a6c7-8104563cd094 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.412257567Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.413229955Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.420589296Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"error","time":"2021-06-24T07:36:17.420710833Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:17.420940961Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.420990529Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.429127075Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"error","time":"2021-06-24T07:36:17.42916452Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:17.429462977Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.429502926Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.4351766Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"error","time":"2021-06-24T07:36:17.435311744Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:17.435519221Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.435530836Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.441886822Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"error","time":"2021-06-24T07:36:17.441930122Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:17.442006352Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:17.442062864Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/gocsi@v1.2.2/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/spec@v1.2.0/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722"}

What you expected to happen: I want to create successfully create a pvc.

How to reproduce it (as minimally and precisely as possible):

create csi-vsphere.conf

cat << EOF > csi-vsphere.conf
[Global]
cluster-id = "csi-vsphere-cluster"
cluster-distribution = "Openshift"

[VirtualCenter "****.vcsa.01.***"]
insecure-flag = "true"
user = "***"
password = "***"
port = "443"
datacenters = "W***"
EOF

Create csi-vsphere secret oc create secret generic vsphere-config-secret --from-file=csi-vsphere.conf --namespace=kube-system

Apply required vsphere-csi-driver yamls

oc apply -f https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v2.2.1/manifests/v2.2.1/rbac/vsphere-csi-controller-rbac.yaml
oc apply -f https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v2.2.1/manifests/v2.2.1/rbac/vsphere-csi-node-rbac.yaml
oc apply -f https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v2.2.1/manifests/v2.2.1/deploy/vsphere-csi-controller-deployment.yaml
oc apply -f https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v2.2.1/manifests/v2.2.1/deploy/vsphere-csi-node-ds.yaml

EnableUUID for all nodes

govc vm.change -e="disk.enableUUID=TRUE" -vm k8s-master-1-01.***.***
govc vm.change -e="disk.enableUUID=TRUE" -vm k8s-master-2-01.***.***
govc vm.change -e="disk.enableUUID=TRUE" -vm k8s-master-3-01.***.***
govc vm.change -e="disk.enableUUID=TRUE" -vm k8s-worker-1-01.***.***

I restarted all nodes after this change.

Create some test namespace oc create namespace test

Add storageclass

(
cat <<EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: csi-vsphere-vmware
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: csi.vsphere.vmware.com
parameters:
  storagepolicyname: "OKD-Test Storage Policy"
  datastoreurl: "ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/"
  csi.storage.k8s.io/fstype: "nfs4"
EOF
 ) | oc create -f -

Create PVC

(
cat <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc1
  namespace: test
  annotations:
    volume.beta.kubernetes.io/storage-class: csi-vsphere-vmware
  labels:
    appname: testapp
spec:
  accessModes:
    - ReadWriteMany 
  resources:
   requests:
     storage: "1Gi"
  storageClassName: csi-vsphere-vmware
EOF
 ) | oc create -f -

Anything else we need to know?:

  • We do not have a vSAN-Cluster, we just want to use our storage from our HPE Primera-Cluster.
  • All nodes are in compatibility mode "ESXi 7.0 U2 and later (VM version 19)" (I also tried with ESXi 6.7 U2 and later (VM version 15), but no diference at all)
oc get CSINode
NAME                                 DRIVERS   AGE
k8s-master-1-01.***.***   1         47h
k8s-master-2-01.***.***   1         47h
k8s-master-3-01.***.***   1         47h
k8s-worker-1-01.***.***   1         42h

oc describe csidrivers
Name:         csi.vsphere.vmware.com
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  storage.k8s.io/v1
Kind:         CSIDriver
Metadata:
  Creation Timestamp:  2021-06-22T08:39:12Z
  Managed Fields:
    API Version:  storage.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        f:attachRequired:
        f:fsGroupPolicy:
        f:podInfoOnMount:
        f:volumeLifecycleModes:
          .:
          v:"Persistent":
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-06-22T08:39:12Z
  Resource Version:  25040
  Self Link:         /apis/storage.k8s.io/v1/csidrivers/csi.vsphere.vmware.com
  UID:               4ccbe69f-4693-4092-b8a6-6c5bbb2d71ba
Spec:
  Attach Required:    true
  Fs Group Policy:    ReadWriteOnceWithFSType
  Pod Info On Mount:  false
  Volume Lifecycle Modes:
    Persistent
Events:  <none>

oc get deployment --namespace=kube-system
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
vsphere-csi-controller   1/1     1            1           47h

oc get daemonsets vsphere-csi-node --namespace=kube-system
NAME               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
vsphere-csi-node   4         4         4       4            4           <none>          47h

oc get nodes
NAME                                 STATUS   ROLES           AGE   VERSION
k8s-master-1-01.***.***   Ready    master,worker   47h   v1.20.0+87cc9a4-1079
k8s-master-2-01.***.***   Ready    master,worker   47h   v1.20.0+87cc9a4-1079
k8s-master-3-01.***.***   Ready    master,worker   47h   v1.20.0+87cc9a4-1079
k8s-worker-1-01.***.***   Ready    worker          42h   v1.20.0+87cc9a4-1079

[root@k8s-master-1-01 core]# cat /sys/class/dmi/id/product_uuid
7b760a42-cecc-951e-3b31-b37a3d279639

[core@k8s-master-1-01 ~]$ sudo cat /sys/class/dmi/id/product_serial 
VMware-42 0a 76 7b cc ce 1e 95-3b 31 b3 7a 3d 27 96 39

oc describe nodes | grep "System UUID"
  System UUID:                            7b760a42-cecc-951e-3b31-b37a3d279639
  System UUID:                            19970a42-4279-8530-7b90-e4ffa0916a27
  System UUID:                            30ed0a42-a0bf-60cc-7c69-fab58ad67423
  System UUID:                            e5a10a42-316f-07cf-cf7d-2b7e4ef42506

Environment:

  • csi-vsphere version: 2.2.1
  • Kubernetes version: 1.20.0
  • vSphere version: 7.0.2A
  • OS (e.g. from /etc/os-release): Fedora CoreOS 34 (34.20210529.3.0)
  • Kernel (e.g. uname -a): Linux 5.12.7-300.fc34.x86_64

created time in 10 hours

pull request commentkubernetes-sigs/vsphere-csi-driver

Cherry-pick PR (#1013) to rel 2.2 - Add check for CSI volume spec in PVUpdated method

@chethanv28: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
pull-vsphere-csi-driver-verify-staticcheck 80f62b310c878fafcc3bbf29cc813ec4fbc65422 link /test pull-vsphere-csi-driver-verify-staticcheck

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. </details> <!-- test report -->

chethanv28

comment created time in 17 hours

pull request commentkubernetes-sigs/vsphere-csi-driver

Cherry pick PR 1013 - Add volume spec check for CSI volumes

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1016#" title="Author self-approved">chethanv28</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

chethanv28

comment created time in 17 hours

PR opened kubernetes-sigs/vsphere-csi-driver

Cherry pick PR 1013 - Add volume spec check for CSI volumes

<!-- Thanks for sending a pull request! -->

What this PR does / why we need it: This PR is cherry-picking the fix for adding volume spec check to release-2.2 branch

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:

Release note: <!-- Steps to write your release note:

  1. Use the release-note-* labels to set the release note state (if you have access)
  2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write NONE. -->
Cherry pick PR 1013 - Add volume spec check for CSI volumes
+5 -2

0 comment

1 changed file

pr created time in 17 hours

pull request commentkubernetes-sigs/vsphere-csi-driver

Cherry-pick PR (#1013) - Add check for CSI volume spec in PVUpdated method

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1015#" title="Author self-approved">chethanv28</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

chethanv28

comment created time in 17 hours

PR opened kubernetes-sigs/vsphere-csi-driver

Cherry-pick PR (#1013) - Add check for CSI volume spec in PVUpdated method
  • Add check for volume spec to avoid NPE

<!-- Thanks for sending a pull request! -->

What this PR does / why we need it: This PR is cherry-picking PR 1013 from master

Release note: <!-- Steps to write your release note:

  1. Use the release-note-* labels to set the release note state (if you have access)
  2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write NONE. -->
Cherry-pick PR (#1013) - Add check for CSI volume spec in PVUpdated method 
+5 -2

0 comment

1 changed file

pr created time in 17 hours

push eventkubernetes-sigs/vsphere-csi-driver

Chethan Venkatesh

commit sha 9e01c676823bc1d2e85dfe838a38c11924e2bf09

Add check for CSI volume spec in PVUpdated method (#1013) * Add check for volume spec to avoid NPE * Rename the variable used for checking if it is a dynamic PV * Add an extra check to ensure code block is for static CSI volume

view details

push time in 17 hours

PR merged kubernetes-sigs/vsphere-csi-driver

Reviewers
Add check for CSI volume spec in PVUpdated method approved cncf-cla: yes lgtm ok-to-test release-2.3.0-candidate size/XS

<!-- Thanks for sending a pull request! -->

What this PR does / why we need it: This PR is adding CSI check for volume spec in metadatasyncer. Currently we are assuming that the volumes are always CSI volumes, however for migrated volumes the spec does not contain CSI field and hence a crash was observed during CSI migration testing.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Testing done: Before:

1. Create in-tree volumes
2. Deploy CSI driver
3. Enable CSI migration FSS
4. Enable Migration flags on kube-controller-manager and kubelet

k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.1/pkg/util/runtime/runtime.go:55 +0x109
panic(0x203bdc0, 0x385fbb0)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
sigs.k8s.io/vsphere-csi-driver/pkg/syncer.csiPVUpdated(0x27e8c78, 0xc0010a8090, 0xc0008ff900, 0xc00024b900, 0xc0000ec000)
	/build/pkg/syncer/metadatasyncer.go:989 +0x304

Performed same steps after the fix and no crash was observed

& Running e2e pipelines

Special notes for your reviewer:

Release note: <!-- Steps to write your release note:

  1. Use the release-note-* labels to set the release note state (if you have access)
  2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write NONE. -->
Add check for volume spec in PVUpdated
+5 -2

7 comments

1 changed file

chethanv28

pr closed time in 17 hours

pull request commentkubernetes-sigs/vsphere-csi-driver

Add check for CSI volume spec in PVUpdated method

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1013#" title="Author self-approved">chethanv28</a>, <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1013#pullrequestreview-691260872" title="Approved">divyenpatel</a>, <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1013#issuecomment-867206258" title="Approved">gohilankit</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

  • OWNERS [chethanv28,divyenpatel]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

chethanv28

comment created time in 17 hours

pull request commentkubernetes-sigs/vsphere-csi-driver

Add check for CSI volume spec in PVUpdated method

Build ID: 105
Block vanilla build status: FAILURE 
Stage before exit: e2e-tests 
Jenkins E2E Test Results: 
Ran 45 of 195 Specs in 11417.785 seconds
FAIL! -- 44 Passed | 1 Failed | 0 Pending | 150 Skipped
--- FAIL: TestE2E (11417.87s)
FAIL

Ginkgo ran 1 suite in 3h10m57.042729493s
Test Suite Failed
make: Leaving directory `/home/worker/workspace/csi-block-vanilla-pre-check-in/105/vsphere-csi-driver`
[csi-block-vanilla] [csi-supervisor] statefulset
/home/worker/workspace/csi-block-vanilla-pre-check-in/105/vsphere-csi-driver/tests/e2e/statefulsets.go:56
  Statefulset testing with parallel podManagementPolicy [It]
  /home/worker/workspace/csi-block-vanilla-pre-check-in/105/vsphere-csi-driver/tests/e2e/statefulsets.go:266

  Unexpected error:
      <*errors.StatusError | 0xc0004dec80>: {
          ErrStatus: {
              TypeMeta: {Kind: "", APIVersion: ""},
              ListMeta: {
                  SelfLink: "",
                  ResourceVersion: "",
                  Continue: "",
                  RemainingItemCount: nil,
              },
              Status: "Failure",
              Message: "etcdserver: request timed out",
              Reason: "",
              Details: nil,
              Code: 500,
          },
      }
      etcdserver: request timed out
  occurred

The 1 failure is a known intermittent failure which is tracked internally.

cc: @divyenpatel Can you help with approvals & merge the PR

chethanv28

comment created time in 17 hours

pull request commentkubernetes-sigs/vsphere-csi-driver

Add check for CSI volume spec in PVUpdated method

Build ID: 105
Block vanilla build status: FAILURE 
Stage before exit: e2e-tests 
Jenkins E2E Test Results: 
Ran 45 of 195 Specs in 11417.785 seconds
FAIL! -- 44 Passed | 1 Failed | 0 Pending | 150 Skipped
--- FAIL: TestE2E (11417.87s)
FAIL

Ginkgo ran 1 suite in 3h10m57.042729493s
Test Suite Failed
make: Leaving directory `/home/worker/workspace/csi-block-vanilla-pre-check-in/105/vsphere-csi-driver`

chethanv28

comment created time in 17 hours

pull request commentkubernetes-sigs/vsphere-csi-driver

Add benchmarks for LogNewError methods

Hi @zhelongp. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

<details>

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. </details>

zhelongp

comment created time in 19 hours

pull request commentkubernetes-sigs/vsphere-csi-driver

Add benchmarks for LogNewError methods

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1014#" title="Author self-approved">zhelongp</a> To complete the pull request process, please assign chethanv28 after the PR has been reviewed. You can assign the PR to them by writing /assign @chethanv28 in a comment when ready.

The full list of commands accepted by this bot can be found here.

<details open> Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":["chethanv28"]} -->

zhelongp

comment created time in 19 hours

PR opened kubernetes-sigs/vsphere-csi-driver

Add benchmarks for LogNewError methods

What this PR does / why we need it: Add four benchmarks to measure the overhead of LogNewError methods.

Testing done: Local build and test.

BenchmarkLogNewError-2 2715090 439 ns/op BenchmarkLogNewErrorf-2 2144805 528 ns/op BenchmarkLogNewErrorCode-2 2473299 457 ns/op BenchmarkLogNewErrorCodef-2 2064558 589 ns/op

+44 -0

0 comment

1 changed file

pr created time in 19 hours