Started WCP block pre-checkin pipeline... Build Number: 67

• Add check for volume spec to avoid NPE

<!-- Thanks for sending a pull request! -->

What this PR does / why we need it: This PR is cherry-picking https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1013 from master

Release note: <!-- Steps to write your release note:

1. Use the release-note-* labels to set the release note state (if you have access)
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write NONE. -->

Cherry-pick PR (#1013) - Add check for CSI volume spec in PVUpdated method 

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1014#pullrequestreview-692063365" title="Approved">divyenpatel</a>, <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1014#" title="Author self-approved">zhelongp</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

• OWNERS [divyenpatel]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

@zhelongp do you intend to run these tests every time as part of PR validation?

I see tests are not executed for this PR - https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_vsphere-csi-driver/1014/pull-vsphere-csi-driver-unit-test/1408113957868146688/build-log.txt

Thanks for the review, @divyenpatel . These tests are benchmarks, which do not run by default (e.g. via make unit). I do not mean to trigger those tests automatically for any PR, which would be a waste of resource. It is good for now.

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1015#" title="Author self-approved">chethanv28</a>, <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1015#pullrequestreview-692038956" title="Approved">divyenpatel</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

• OWNERS [chethanv28,divyenpatel]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1009#issuecomment-867818520" title="Approved">divyenpatel</a>, <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1009#" title="Author self-approved">zhelongp</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

• OWNERS [divyenpatel]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1007#issuecomment-867818379" title="Approved">divyenpatel</a>, <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1007#" title="Author self-approved">zhelongp</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

• OWNERS [divyenpatel]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1006#issuecomment-867817981" title="Approved">divyenpatel</a>, <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1006#" title="Author self-approved">zhelongp</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

• OWNERS [divyenpatel]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1002#issuecomment-867817438" title="Approved">divyenpatel</a>, <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1002#" title="Author self-approved">zhelongp</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

• OWNERS [divyenpatel]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

Build ID: 12
WCP build status: FAILURE 
Stage before exit: null 
Jenkins E2E Test Results: 
Ran 14 of 195 Specs in 2822.795 seconds
FAIL! -- 9 Passed | 5 Failed | 0 Pending | 181 Skipped
--- FAIL: TestE2E (2822.91s)
FAIL

Ginkgo ran 1 suite in 47m42.798649647s
Test Suite Failed
make: Leaving directory `/home/worker/workspace/csi-wcp-pre-check-in-trials/Results/12/vsphere-csi-driver`

Started WCP block pre-checkin pipeline... Build Number: 12

<!-- Thanks for sending a pull request! -->

What this PR does / why we need it: This PR is updating CSI Spec to v1.4.0 The latest CSI spec has few limitations relaxed, such as volume name length and node id. The list of improvements can be found here: https://github.com/container-storage-interface/spec/releases/tag/v1.4.0

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:

Release note: <!-- Steps to write your release note:

1. Use the release-note-* labels to set the release note state (if you have access)
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write NONE. -->

Update CSI Spec to v1.4.0

Not needed since CSI spec has been updated along with go-csi removal work items

cc: @divyenpatel Can you help merge this cherry-pick PR

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened: If I try to create a pvc I receive the error "no datastores found to create file volume" or sometimes I get "failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager". I think the main reason is that the node UUIDs are empty for some reason.

In the logs of the vsphere-csi-controller container I found the following entries:

{"level":"info","time":"2021-06-23T17:14:39.377247663Z","caller":"logger/logger.go:37","msg":"Setting default log level to :\"PRODUCTION\""}
{"level":"info","time":"2021-06-23T17:14:39.377457762Z","caller":"vsphere-csi/main.go:54","msg":"Version : v2.2.1","TraceId":"de2d5508-f9e7-4659-b426-b4dad397629b"}
{"level":"info","time":"2021-06-23T17:14:39.379655703Z","caller":"logger/logger.go:37","msg":"Setting default log level to :\"PRODUCTION\""}
{"level":"info","time":"2021-06-23T17:14:39.384817331Z","caller":"k8sorchestrator/k8sorchestrator.go:122","msg":"Initializing k8sOrchestratorInstance","TraceId":"29b35dba-cd21-4c6c-8e85-33c96158773c"}
{"level":"info","time":"2021-06-23T17:14:39.385285147Z","caller":"kubernetes/kubernetes.go:91","msg":"k8s client using in-cluster config","TraceId":"29b35dba-cd21-4c6c-8e85-33c96158773c"}
{"level":"info","time":"2021-06-23T17:14:39.388263712Z","caller":"kubernetes/kubernetes.go:304","msg":"Setting client QPS to 100.000000 and Burst to 100.","TraceId":"29b35dba-cd21-4c6c-8e85-33c96158773c"}
{"level":"info","time":"2021-06-23T17:14:39.42948329Z","caller":"k8sorchestrator/k8sorchestrator.go:234","msg":"New internal feature states values stored successfully: map[csi-auth-check:true csi-migration:false online-volume-extend:true]","TraceId":"29b35dba-cd21-4c6c-8e85-33c96158773c"}
{"level":"info","time":"2021-06-23T17:14:39.429767408Z","caller":"k8sorchestrator/k8sorchestrator.go:148","msg":"k8sOrchestratorInstance initialized","TraceId":"29b35dba-cd21-4c6c-8e85-33c96158773c"}
{"level":"info","time":"2021-06-23T17:14:39.431977646Z","caller":"config/config.go:302","msg":"No Net Permissions given in Config. Using default permissions.","TraceId":"29b35dba-cd21-4c6c-8e85-33c96158773c"}
{"level":"info","time":"2021-06-23T17:14:39.4326364Z","caller":"vanilla/controller.go:79","msg":"Initializing CNS controller","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.432688233Z","caller":"vsphere/utils.go:132","msg":"Defaulting timeout for vCenter Client to 5 minutes","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.432712115Z","caller":"vsphere/virtualcentermanager.go:72","msg":"Initializing defaultVirtualCenterManager...","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.43272412Z","caller":"vsphere/virtualcentermanager.go:74","msg":"Successfully initialized defaultVirtualCenterManager","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.432742534Z","caller":"vsphere/virtualcentermanager.go:118","msg":"Successfully registered VC \"****.vcsa.01.***\"","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.432755507Z","caller":"volume/manager.go:111","msg":"Initializing new volume.defaultManager...","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.44806408Z","caller":"k8sorchestrator/k8sorchestrator.go:273","msg":"New feature states values from \"internal-feature-states.csi.vsphere.vmware.com\" stored successfully: map[csi-auth-check:true csi-migration:false online-volume-extend:true]","TraceId":"0b7eac10-7e14-4bb8-984d-f5939891039b"}
{"level":"info","time":"2021-06-23T17:14:39.550063804Z","caller":"vsphere/virtualcenter.go:169","msg":"New session ID for '***\\***' = 5230da53-6841-4bfe-8bb1-f72e8a4b4c19","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.550125363Z","caller":"node/manager.go:75","msg":"Initializing node.defaultManager...","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.550137425Z","caller":"node/manager.go:79","msg":"node.defaultManager initialized","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.550153808Z","caller":"kubernetes/kubernetes.go:91","msg":"k8s client using in-cluster config","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.550717029Z","caller":"kubernetes/kubernetes.go:304","msg":"Setting client QPS to 100.000000 and Burst to 100.","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.558542125Z","caller":"vanilla/controller.go:142","msg":"CSIAuthCheck feature is enabled, loading AuthorizationService","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.558581435Z","caller":"common/authmanager.go:76","msg":"Initializing authorization service...","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.558592813Z","caller":"common/authmanager.go:86","msg":"authorization service initialized","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.55863975Z","caller":"vanilla/controller.go:197","msg":"Adding watch on path: \"/etc/cloud\"","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.558674841Z","caller":"service/service.go:106","msg":"configured: \"csi.vsphere.vmware.com\" with clusterFlavor: \"VANILLA\" and mode: \"controller\"","TraceId":"29b35dba-cd21-4c6c-8e85-33c96158773c"}
time="2021-06-23T17:14:39Z" level=info msg="identity service registered"
time="2021-06-23T17:14:39Z" level=info msg="controller service registered"
time="2021-06-23T17:14:39Z" level=info msg=serving endpoint="unix:///csi/csi.sock"
{"level":"info","time":"2021-06-23T17:14:39.559394784Z","caller":"common/authmanager.go:156","msg":"auth manager: ComputeDatastoreMapForBlockVolumes enter"}
{"level":"info","time":"2021-06-23T17:14:39.562355896Z","caller":"common/authmanager.go:166","msg":"auth manager: ComputeDatastoreMapForFileVolumes enter"}
{"level":"info","time":"2021-06-23T17:14:39.572984454Z","caller":"vanilla/controller.go:215","msg":"Starting the http server to expose Prometheus metrics..","TraceId":"f002819a-2d89-4214-8fb4-c041826f40f8"}
{"level":"info","time":"2021-06-23T17:14:39.62088948Z","caller":"vanilla/controller.go:1069","msg":"ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"79725ae3-386e-4f14-96bb-ea19f3e686f0"}
{"level":"info","time":"2021-06-23T17:14:39.624403738Z","caller":"node/manager.go:103","msg":"Successfully registered node: \"k8s-worker-1-01.***.***\" with nodeUUID \"\"","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110"}
{"level":"info","time":"2021-06-23T17:14:39.62478924Z","caller":"vsphere/virtualmachine.go:123","msg":"Initiating asynchronous datacenter listing with uuid ","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110"}
{"level":"info","time":"2021-06-23T17:14:39.631695568Z","caller":"vanilla/controller.go:1069","msg":"ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"92dee706-80d2-4856-b752-e3b17e8cfb2a"}
{"level":"info","time":"2021-06-23T17:14:39.646434025Z","caller":"vsphere/datacenter.go:145","msg":"Publishing datacenter Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***]","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110"}
{"level":"info","time":"2021-06-23T17:14:39.646610493Z","caller":"vsphere/virtualmachine.go:160","msg":"AsyncGetAllDatacenters with uuid  sent a dc Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***]","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110"}
{"level":"error","time":"2021-06-23T17:14:39.652400548Z","caller":"vsphere/datacenter.go:98","msg":"Couldn't find VM given uuid ","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.(*Datacenter).GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/datacenter.go:98\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID.func1\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:161"}
{"level":"warn","time":"2021-06-23T17:14:39.65263142Z","caller":"vsphere/virtualmachine.go:164","msg":"Couldn't find VM given uuid  on DC Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***] with err: virtual machine wasn't found, continuing search","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110"}
{"level":"error","time":"2021-06-23T17:14:39.652723572Z","caller":"vsphere/virtualmachine.go:191","msg":"Returning VM not found err for UUID ","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:191\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:117\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73"}
{"level":"error","time":"2021-06-23T17:14:39.652877692Z","caller":"node/manager.go:119","msg":"Couldn't find VM instance with nodeUUID , failed to discover with err: virtual machine wasn't found","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:119\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73"}
{"level":"error","time":"2021-06-23T17:14:39.653048955Z","caller":"node/manager.go:106","msg":"failed to discover VM with uuid: \"\" for node: \"k8s-worker-1-01.***.***\"","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:106\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73"}
{"level":"warn","time":"2021-06-23T17:14:39.653167976Z","caller":"vanilla/nodes.go:67","msg":"failed to register node:\"k8s-worker-1-01.***.***\". err=virtual machine wasn't found","TraceId":"7b74209f-e671-43f4-ae41-2e85c0460110"}
{"level":"info","time":"2021-06-23T17:14:39.653677111Z","caller":"node/manager.go:103","msg":"Successfully registered node: \"k8s-master-1-01.***.***\" with nodeUUID \"\"","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15"}
{"level":"info","time":"2021-06-23T17:14:39.655341817Z","caller":"vsphere/virtualmachine.go:123","msg":"Initiating asynchronous datacenter listing with uuid ","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15"}
{"level":"info","time":"2021-06-23T17:14:39.680181413Z","caller":"vsphere/datacenter.go:145","msg":"Publishing datacenter Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***]","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15"}
{"level":"info","time":"2021-06-23T17:14:39.680636976Z","caller":"vsphere/virtualmachine.go:160","msg":"AsyncGetAllDatacenters with uuid  sent a dc Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***]","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15"}
{"level":"error","time":"2021-06-23T17:14:39.687238379Z","caller":"vsphere/datacenter.go:98","msg":"Couldn't find VM given uuid ","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.(*Datacenter).GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/datacenter.go:98\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID.func1\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:161"}
{"level":"warn","time":"2021-06-23T17:14:39.687338753Z","caller":"vsphere/virtualmachine.go:164","msg":"Couldn't find VM given uuid  on DC Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***] with err: virtual machine wasn't found, continuing search","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15"}
{"level":"error","time":"2021-06-23T17:14:39.687361172Z","caller":"vsphere/virtualmachine.go:191","msg":"Returning VM not found err for UUID ","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:191\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:117\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73"}
{"level":"error","time":"2021-06-23T17:14:39.687416376Z","caller":"node/manager.go:119","msg":"Couldn't find VM instance with nodeUUID , failed to discover with err: virtual machine wasn't found","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:119\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73"}
{"level":"error","time":"2021-06-23T17:14:39.687474217Z","caller":"node/manager.go:106","msg":"failed to discover VM with uuid: \"\" for node: \"k8s-master-1-01.***.***\"","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:106\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73"}
{"level":"warn","time":"2021-06-23T17:14:39.687530348Z","caller":"vanilla/nodes.go:67","msg":"failed to register node:\"k8s-master-1-01.***.***\". err=virtual machine wasn't found","TraceId":"751cc3e1-5d4e-4054-9993-64bf470f8d15"}
{"level":"info","time":"2021-06-23T17:14:39.687764295Z","caller":"node/manager.go:103","msg":"Successfully registered node: \"k8s-master-2-01.***.***\" with nodeUUID \"\"","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812"}
{"level":"info","time":"2021-06-23T17:14:39.687792615Z","caller":"vsphere/virtualmachine.go:123","msg":"Initiating asynchronous datacenter listing with uuid ","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812"}
{"level":"info","time":"2021-06-23T17:14:39.697016254Z","caller":"vsphere/datacenter.go:145","msg":"Publishing datacenter Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***]","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812"}
{"level":"info","time":"2021-06-23T17:14:39.697270216Z","caller":"vsphere/virtualmachine.go:160","msg":"AsyncGetAllDatacenters with uuid  sent a dc Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***]","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812"}
{"level":"error","time":"2021-06-23T17:14:39.706040552Z","caller":"vsphere/datacenter.go:98","msg":"Couldn't find VM given uuid ","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.(*Datacenter).GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/datacenter.go:98\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID.func1\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:161"}
{"level":"warn","time":"2021-06-23T17:14:39.706358026Z","caller":"vsphere/virtualmachine.go:164","msg":"Couldn't find VM given uuid  on DC Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***] with err: virtual machine wasn't found, continuing search","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812"}
{"level":"error","time":"2021-06-23T17:14:39.706466123Z","caller":"vsphere/virtualmachine.go:191","msg":"Returning VM not found err for UUID ","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:191\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:117\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73"}
{"level":"error","time":"2021-06-23T17:14:39.706588692Z","caller":"node/manager.go:119","msg":"Couldn't find VM instance with nodeUUID , failed to discover with err: virtual machine wasn't found","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:119\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73"}
{"level":"error","time":"2021-06-23T17:14:39.706664133Z","caller":"node/manager.go:106","msg":"failed to discover VM with uuid: \"\" for node: \"k8s-master-2-01.***.***\"","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:106\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73"}
{"level":"warn","time":"2021-06-23T17:14:39.706738905Z","caller":"vanilla/nodes.go:67","msg":"failed to register node:\"k8s-master-2-01.***.***\". err=virtual machine wasn't found","TraceId":"c787c7f7-3bb4-4d9e-9a19-f2b90d125812"}
{"level":"info","time":"2021-06-23T17:14:39.706966564Z","caller":"node/manager.go:103","msg":"Successfully registered node: \"k8s-master-3-01.***.***\" with nodeUUID \"\"","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163"}
{"level":"info","time":"2021-06-23T17:14:39.707034283Z","caller":"vsphere/virtualmachine.go:123","msg":"Initiating asynchronous datacenter listing with uuid ","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163"}
{"level":"info","time":"2021-06-23T17:14:39.716086532Z","caller":"vsphere/datacenter.go:145","msg":"Publishing datacenter Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***]","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163"}
{"level":"info","time":"2021-06-23T17:14:39.716200395Z","caller":"vsphere/virtualmachine.go:160","msg":"AsyncGetAllDatacenters with uuid  sent a dc Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***]","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163"}
{"level":"error","time":"2021-06-23T17:14:39.721503507Z","caller":"vsphere/datacenter.go:98","msg":"Couldn't find VM given uuid ","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.(*Datacenter).GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/datacenter.go:98\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID.func1\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:161"}
{"level":"warn","time":"2021-06-23T17:14:39.721577793Z","caller":"vsphere/virtualmachine.go:164","msg":"Couldn't find VM given uuid  on DC Datacenter [Datacenter: Datacenter:datacenter-28 @ /W***, VirtualCenterHost: ****.vcsa.01.***] with err: virtual machine wasn't found, continuing search","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163"}
{"level":"error","time":"2021-06-23T17:14:39.721597247Z","caller":"vsphere/virtualmachine.go:191","msg":"Returning VM not found err for UUID ","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:191\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:117\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73"}
{"level":"error","time":"2021-06-23T17:14:39.721664846Z","caller":"node/manager.go:119","msg":"Couldn't find VM instance with nodeUUID , failed to discover with err: virtual machine wasn't found","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:119\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73"}
{"level":"error","time":"2021-06-23T17:14:39.721712072Z","caller":"node/manager.go:106","msg":"failed to discover VM with uuid: \"\" for node: \"k8s-master-3-01.***.***\"","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:106\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:65\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:218\nk8s.io/client-go/tools/cache.(*processorListener).run.func1\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:744\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90\nk8s.io/client-go/tools/cache.(*processorListener).run\n\t/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:738\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73"}
{"level":"warn","time":"2021-06-23T17:14:39.721755623Z","caller":"vanilla/nodes.go:67","msg":"failed to register node:\"k8s-master-3-01.***.***\". err=virtual machine wasn't found","TraceId":"2ed950ea-358f-4bb2-90d9-432f73421163"}
{"level":"info","time":"2021-06-23T17:14:39.809397819Z","caller":"vanilla/controller.go:1069","msg":"ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"743deeba-68ef-4637-8feb-41f77bcf4aa2"}
{"level":"info","time":"2021-06-23T17:14:41.158843676Z","caller":"vanilla/controller.go:1069","msg":"ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"f87052c2-17de-4f90-a15c-81bb579b30ea"}
{"level":"info","time":"2021-06-23T17:44:39.44905192Z","caller":"k8sorchestrator/k8sorchestrator.go:293","msg":"New feature states values from \"internal-feature-states.csi.vsphere.vmware.com\" stored successfully: map[csi-auth-check:true csi-migration:false online-volume-extend:true]","TraceId":"766f18fd-94fd-4f4e-9322-49cc9a0a6931"}
{"level":"info","time":"2021-06-24T03:44:39.506774985Z","caller":"k8sorchestrator/k8sorchestrator.go:293","msg":"New feature states values from \"internal-feature-states.csi.vsphere.vmware.com\" stored successfully: map[csi-auth-check:true csi-migration:false online-volume-extend:true]","TraceId":"7ce83dbf-ab35-4619-bf74-20db0134dfa7"}
{"level":"info","time":"2021-06-24T03:55:52.000394761Z","caller":"vanilla/controller.go:1069","msg":"ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"d3acbc20-1e5f-482c-9418-7ab6ed5a3586"}
{"level":"info","time":"2021-06-24T04:00:22.916524464Z","caller":"vanilla/controller.go:1069","msg":"ControllerGetCapabilities: called with args {XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"0929c14c-9f0b-497b-ba64-6cc94766d227"}
{"level":"info","time":"2021-06-24T04:14:39.503498331Z","caller":"k8sorchestrator/k8sorchestrator.go:293","msg":"New feature states values from \"internal-feature-states.csi.vsphere.vmware.com\" stored successfully: map[csi-auth-check:true csi-migration:false online-volume-extend:true]","TraceId":"1c0113f2-8ee5-4c21-af09-d139959ee159"}
{"level":"info","time":"2021-06-24T07:14:39.49835871Z","caller":"k8sorchestrator/k8sorchestrator.go:293","msg":"New feature states values from \"internal-feature-states.csi.vsphere.vmware.com\" stored successfully: map[csi-auth-check:true csi-migration:false online-volume-extend:true]","TraceId":"dcb58eb4-399b-4dba-88ec-a045ae7ddcd8"}
{"level":"info","time":"2021-06-24T07:35:39.517171263Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-6ef59515-e2ae-41e7-b506-1d729b1d7a9d CapacityRange:required_bytes:1073741824  VolumeCapabilities:[block:<> access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.517223011Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.517234728Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.531998917Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"error","time":"2021-06-24T07:35:39.532040552Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:39.53213164Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.532144002Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.546474831Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"error","time":"2021-06-24T07:35:39.546523548Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:39.546625075Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.546638392Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.554171081Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"error","time":"2021-06-24T07:35:39.554211076Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:39.554299506Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.554312086Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"info","time":"2021-06-24T07:35:39.560668452Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e"}
{"level":"error","time":"2021-06-24T07:35:39.560713428Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:35:39.560837658Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:35:39.560890413Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"c9f84ef3-ca41-4ac6-b5a2-047a39f9a57e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:40.562424896Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-6ef59515-e2ae-41e7-b506-1d729b1d7a9d CapacityRange:required_bytes:1073741824  VolumeCapabilities:[block:<> access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.562636273Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.562677201Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.569787296Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"error","time":"2021-06-24T07:35:40.569949985Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:40.570085971Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.570182137Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.576102603Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"error","time":"2021-06-24T07:35:40.576217073Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:40.576357269Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.576398489Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.582433874Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"error","time":"2021-06-24T07:35:40.582515763Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:40.582601509Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.58261388Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"info","time":"2021-06-24T07:35:40.597313293Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d"}
{"level":"error","time":"2021-06-24T07:35:40.59737204Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:35:40.597450419Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:35:40.598421879Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"f23e1db1-2178-4ab4-8822-59848e6ec63d","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:42.600379178Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-6ef59515-e2ae-41e7-b506-1d729b1d7a9d CapacityRange:required_bytes:1073741824  VolumeCapabilities:[block:<> access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.600427404Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.600452334Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.616491396Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"error","time":"2021-06-24T07:35:42.616531646Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"da350121-741b-4af5-8306-e67e5585228e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:42.616640356Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.616652322Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.642052498Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"error","time":"2021-06-24T07:35:42.642100282Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"da350121-741b-4af5-8306-e67e5585228e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:42.642198273Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.642210124Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.663517728Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"error","time":"2021-06-24T07:35:42.663647082Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"da350121-741b-4af5-8306-e67e5585228e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:42.663780904Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.663831735Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"info","time":"2021-06-24T07:35:42.676333554Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"da350121-741b-4af5-8306-e67e5585228e"}
{"level":"error","time":"2021-06-24T07:35:42.676517963Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"da350121-741b-4af5-8306-e67e5585228e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:35:42.676756044Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"da350121-741b-4af5-8306-e67e5585228e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:35:42.676845696Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"da350121-741b-4af5-8306-e67e5585228e","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:46.680692707Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-6ef59515-e2ae-41e7-b506-1d729b1d7a9d CapacityRange:required_bytes:1073741824  VolumeCapabilities:[block:<> access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.680752247Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.680768598Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.696105192Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"error","time":"2021-06-24T07:35:46.696263941Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:46.69645028Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.696494523Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.718759556Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"error","time":"2021-06-24T07:35:46.719198657Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:46.71949466Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.719514382Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.732399548Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"error","time":"2021-06-24T07:35:46.732440905Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:35:46.732517863Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.732529239Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"info","time":"2021-06-24T07:35:46.744510827Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6"}
{"level":"error","time":"2021-06-24T07:35:46.744584083Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:35:46.744693871Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:35:46.744809166Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"901a5e80-04f2-4547-b016-991a855b9ee6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:02.263369522Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-650ee0f9-6939-4599-a6c7-8104563cd094 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.263475695Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.267481858Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.274630112Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"error","time":"2021-06-24T07:36:02.274781601Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:02.274953389Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.27501909Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.285935614Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"error","time":"2021-06-24T07:36:02.28597211Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:02.286063464Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.286075098Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.291910385Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"error","time":"2021-06-24T07:36:02.29194672Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:02.292022881Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.292033973Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"info","time":"2021-06-24T07:36:02.297246316Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9"}
{"level":"error","time":"2021-06-24T07:36:02.297358462Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:02.297466229Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:02.297520002Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"cae2148d-4056-489e-80ac-3860f56f9db9","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:03.299757413Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-650ee0f9-6939-4599-a6c7-8104563cd094 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.299876238Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.299933615Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.312288413Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"error","time":"2021-06-24T07:36:03.312342912Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:03.312449124Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.312460949Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.319093193Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"error","time":"2021-06-24T07:36:03.31914127Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:03.319241097Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.31925331Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.324215441Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"error","time":"2021-06-24T07:36:03.324250327Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:03.324335731Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.324346784Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"info","time":"2021-06-24T07:36:03.338195221Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667"}
{"level":"error","time":"2021-06-24T07:36:03.338236454Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:03.338311386Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:03.338383805Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"dfbf06f3-bc34-44f1-aec7-3b2d858f5667","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:05.34414704Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-650ee0f9-6939-4599-a6c7-8104563cd094 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.34438201Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.3444321Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.358437025Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"error","time":"2021-06-24T07:36:05.358589374Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:05.358739173Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.358794526Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.368311311Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"error","time":"2021-06-24T07:36:05.368373198Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:05.368458019Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.368470212Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.373226807Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"error","time":"2021-06-24T07:36:05.373386349Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:05.373562725Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.373613147Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"info","time":"2021-06-24T07:36:05.379161465Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa"}
{"level":"error","time":"2021-06-24T07:36:05.379293022Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:05.379469257Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:05.379603273Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"597db599-8a7a-4a55-b3a2-19819f7b9caa","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:09.383009985Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-650ee0f9-6939-4599-a6c7-8104563cd094 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.383075308Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.383449201Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.392964784Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"error","time":"2021-06-24T07:36:09.393004611Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:09.393077416Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.393088711Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.398475254Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"error","time":"2021-06-24T07:36:09.398604675Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:09.398807996Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.398864397Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.403210165Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"error","time":"2021-06-24T07:36:09.403332577Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:09.403453893Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.403491092Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"info","time":"2021-06-24T07:36:09.408316534Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6"}
{"level":"error","time":"2021-06-24T07:36:09.408441619Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:09.408525195Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:09.408598948Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"97341a9d-4205-44aa-b947-eef76104c0a6","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:17.411103961Z","caller":"vanilla/controller.go:624","msg":"CreateVolume: called with args {Name:pvc-650ee0f9-6939-4599-a6c7-8104563cd094 CapacityRange:required_bytes:1073741824  VolumeCapabilities:[mount:<fs_type:\"ext4\" > access_mode:<mode:SINGLE_NODE_WRITER > ] Parameters:map[datastoreurl:ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.412257567Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-2-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.413229955Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-2-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.420589296Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-2-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"error","time":"2021-06-24T07:36:17.420710833Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-2-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:17.420940961Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-3-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.420990529Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-3-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.429127075Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-3-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"error","time":"2021-06-24T07:36:17.42916452Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-3-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:17.429462977Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-worker-1-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.429502926Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-worker-1-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.4351766Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-worker-1-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"error","time":"2021-06-24T07:36:17.435311744Z","caller":"node/manager.go:207","msg":"Node: \"k8s-worker-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"info","time":"2021-06-24T07:36:17.435519221Z","caller":"node/manager.go:200","msg":"Empty node UUID observed for the node: \"k8s-master-1-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.435530836Z","caller":"kubernetes/kubernetes.go:261","msg":"GetNodeVMUUID called for the node: \"k8s-master-1-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"info","time":"2021-06-24T07:36:17.441886822Z","caller":"kubernetes/kubernetes.go:268","msg":"Retrieved node UUID: \"\" for the node: \"k8s-master-1-01.***.***\"","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca"}
{"level":"error","time":"2021-06-24T07:36:17.441930122Z","caller":"node/manager.go:207","msg":"Node: \"k8s-master-1-01.***.***\" with empty providerId found in the cluster. aborting get all nodes","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes.func1\n\t/build/pkg/common/cns-lib/node/manager.go:207\nsync.(*Map).Range\n\t/usr/local/go/src/sync/map.go:345\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).GetAllNodes\n\t/build/pkg/common/cns-lib/node/manager.go:198\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:232\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:17.442006352Z","caller":"vanilla/nodes.go:239","msg":"Empty List of Node VMs returned from nodeManager","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).GetSharedDatastoresInK8SCluster\n\t/build/pkg/csi/service/vanilla/nodes.go:239\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:442\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}
{"level":"error","time":"2021-06-24T07:36:17.442062864Z","caller":"vanilla/controller.go:445","msg":"failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager","TraceId":"b8014ef1-816f-492a-87b1-5f19487dbfca","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).createBlockVolume\n\t/build/pkg/csi/service/vanilla/controller.go:445\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume.func1\n\t/build/pkg/csi/service/vanilla/controller.go:644\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*controller).CreateVolume\n\t/build/pkg/csi/service/vanilla/controller.go:646\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler.func1\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5146\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).createVolume\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:162\ngithub.com/rexray/gocsi/middleware/serialvolume.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/serialvolume/serial_volume_locker.go:90\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer.func1\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:178\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handle\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:218\ngithub.com/rexray/gocsi/middleware/specvalidator.(*interceptor).handleServer\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware/specvalidator/spec_validator.go:177\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi.(*StoragePlugin).injectContext\n\t/go/pkg/mod/github.com/rexray/[email protected]/middleware.go:231\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2.1.1\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:99\ngithub.com/rexray/gocsi/utils.ChainUnaryServer.func2\n\t/go/pkg/mod/github.com/rexray/[email protected]/utils/utils_middleware.go:106\ngithub.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler\n\t/go/pkg/mod/github.com/container-storage-interface/[email protected]/lib/go/csi/csi.pb.go:5148\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1024\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:1313\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.1\n\t/go/pkg/mod/google.golang.org/[email protected]/server.go:722"}

What you expected to happen: I want to create successfully create a pvc.

How to reproduce it (as minimally and precisely as possible):

create csi-vsphere.conf

cat << EOF > csi-vsphere.conf
[Global]
cluster-id = "csi-vsphere-cluster"
cluster-distribution = "Openshift"

[VirtualCenter "****.vcsa.01.***"]
insecure-flag = "true"
user = "***"
password = "***"
port = "443"
datacenters = "W***"
EOF

Create csi-vsphere secret oc create secret generic vsphere-config-secret --from-file=csi-vsphere.conf --namespace=kube-system

Apply required vsphere-csi-driver yamls

oc apply -f https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v2.2.1/manifests/v2.2.1/rbac/vsphere-csi-controller-rbac.yaml
oc apply -f https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v2.2.1/manifests/v2.2.1/rbac/vsphere-csi-node-rbac.yaml
oc apply -f https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v2.2.1/manifests/v2.2.1/deploy/vsphere-csi-controller-deployment.yaml
oc apply -f https://raw.githubusercontent.com/kubernetes-sigs/vsphere-csi-driver/v2.2.1/manifests/v2.2.1/deploy/vsphere-csi-node-ds.yaml

EnableUUID for all nodes

govc vm.change -e="disk.enableUUID=TRUE" -vm k8s-master-1-01.***.***
govc vm.change -e="disk.enableUUID=TRUE" -vm k8s-master-2-01.***.***
govc vm.change -e="disk.enableUUID=TRUE" -vm k8s-master-3-01.***.***
govc vm.change -e="disk.enableUUID=TRUE" -vm k8s-worker-1-01.***.***

I restarted all nodes after this change.

Create some test namespace oc create namespace test

Add storageclass

(
cat <<EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: csi-vsphere-vmware
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
provisioner: csi.vsphere.vmware.com
parameters:
  storagepolicyname: "OKD-Test Storage Policy"
  datastoreurl: "ds:///vmfs/volumes/5df8d0b5-ec8df8e2-c01e-da3697500067/"
  csi.storage.k8s.io/fstype: "nfs4"
EOF
 ) | oc create -f -

Create PVC

(
cat <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc1
  namespace: test
  annotations:
    volume.beta.kubernetes.io/storage-class: csi-vsphere-vmware
  labels:
    appname: testapp
spec:
  accessModes:
    - ReadWriteMany 
  resources:
   requests:
     storage: "1Gi"
  storageClassName: csi-vsphere-vmware
EOF
 ) | oc create -f -

Anything else we need to know?:

• We do not have a vSAN-Cluster, we just want to use our storage from our HPE Primera-Cluster.
• All nodes are in compatibility mode "ESXi 7.0 U2 and later (VM version 19)" (I also tried with ESXi 6.7 U2 and later (VM version 15), but no diference at all)

oc get CSINode
NAME                                 DRIVERS   AGE
k8s-master-1-01.***.***   1         47h
k8s-master-2-01.***.***   1         47h
k8s-master-3-01.***.***   1         47h
k8s-worker-1-01.***.***   1         42h

oc describe csidrivers
Name:         csi.vsphere.vmware.com
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  storage.k8s.io/v1
Kind:         CSIDriver
Metadata:
  Creation Timestamp:  2021-06-22T08:39:12Z
  Managed Fields:
    API Version:  storage.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        f:attachRequired:
        f:fsGroupPolicy:
        f:podInfoOnMount:
        f:volumeLifecycleModes:
          .:
          v:"Persistent":
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-06-22T08:39:12Z
  Resource Version:  25040
  Self Link:         /apis/storage.k8s.io/v1/csidrivers/csi.vsphere.vmware.com
  UID:               4ccbe69f-4693-4092-b8a6-6c5bbb2d71ba
Spec:
  Attach Required:    true
  Fs Group Policy:    ReadWriteOnceWithFSType
  Pod Info On Mount:  false
  Volume Lifecycle Modes:
    Persistent
Events:  <none>

oc get deployment --namespace=kube-system
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
vsphere-csi-controller   1/1     1            1           47h

oc get daemonsets vsphere-csi-node --namespace=kube-system
NAME               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
vsphere-csi-node   4         4         4       4            4           <none>          47h

oc get nodes
NAME                                 STATUS   ROLES           AGE   VERSION
k8s-master-1-01.***.***   Ready    master,worker   47h   v1.20.0+87cc9a4-1079
k8s-master-2-01.***.***   Ready    master,worker   47h   v1.20.0+87cc9a4-1079
k8s-master-3-01.***.***   Ready    master,worker   47h   v1.20.0+87cc9a4-1079
k8s-worker-1-01.***.***   Ready    worker          42h   v1.20.0+87cc9a4-1079

[root@k8s-master-1-01 core]# cat /sys/class/dmi/id/product_uuid
7b760a42-cecc-951e-3b31-b37a3d279639

[core@k8s-master-1-01 ~]$ sudo cat /sys/class/dmi/id/product_serial 
VMware-42 0a 76 7b cc ce 1e 95-3b 31 b3 7a 3d 27 96 39

oc describe nodes | grep "System UUID"
  System UUID:                            7b760a42-cecc-951e-3b31-b37a3d279639
  System UUID:                            19970a42-4279-8530-7b90-e4ffa0916a27
  System UUID:                            30ed0a42-a0bf-60cc-7c69-fab58ad67423
  System UUID:                            e5a10a42-316f-07cf-cf7d-2b7e4ef42506

Environment:

• csi-vsphere version: 2.2.1
• Kubernetes version: 1.20.0
• vSphere version: 7.0.2A
• OS (e.g. from /etc/os-release): Fedora CoreOS 34 (34.20210529.3.0)
• Kernel (e.g. uname -a): Linux 5.12.7-300.fc34.x86_64

@chethanv28: The following test failed, say /retest to rerun all failed tests:

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. </details> <!-- test report -->

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1016#" title="Author self-approved">chethanv28</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

• OWNERS [chethanv28]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

<!-- Thanks for sending a pull request! -->

What this PR does / why we need it: This PR is cherry-picking the fix for adding volume spec check to release-2.2 branch

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:

Release note: <!-- Steps to write your release note:

1. Use the release-note-* labels to set the release note state (if you have access)
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write NONE. -->

Cherry pick PR 1013 - Add volume spec check for CSI volumes

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1015#" title="Author self-approved">chethanv28</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

• OWNERS [chethanv28]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

• Add check for volume spec to avoid NPE

<!-- Thanks for sending a pull request! -->

What this PR does / why we need it: This PR is cherry-picking PR 1013 from master

Release note: <!-- Steps to write your release note:

1. Use the release-note-* labels to set the release note state (if you have access)
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write NONE. -->

Cherry-pick PR (#1013) - Add check for CSI volume spec in PVUpdated method 

<!-- Thanks for sending a pull request! -->

What this PR does / why we need it: This PR is adding CSI check for volume spec in metadatasyncer. Currently we are assuming that the volumes are always CSI volumes, however for migrated volumes the spec does not contain CSI field and hence a crash was observed during CSI migration testing.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Testing done: Before:

1. Create in-tree volumes
2. Deploy CSI driver
3. Enable CSI migration FSS
4. Enable Migration flags on kube-controller-manager and kubelet

k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x109
panic(0x203bdc0, 0x385fbb0)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
sigs.k8s.io/vsphere-csi-driver/pkg/syncer.csiPVUpdated(0x27e8c78, 0xc0010a8090, 0xc0008ff900, 0xc00024b900, 0xc0000ec000)
	/build/pkg/syncer/metadatasyncer.go:989 +0x304

Performed same steps after the fix and no crash was observed

& Running e2e pipelines

Special notes for your reviewer:

Release note: <!-- Steps to write your release note:

1. Use the release-note-* labels to set the release note state (if you have access)
2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write NONE. -->

Add check for volume spec in PVUpdated

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1013#" title="Author self-approved">chethanv28</a>, <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1013#pullrequestreview-691260872" title="Approved">divyenpatel</a>, <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1013#issuecomment-867206258" title="Approved">gohilankit</a>

The full list of commands accepted by this bot can be found here.

The pull request process is described here

<details > Needs approval from an approver in each of these files:

• OWNERS [chethanv28,divyenpatel]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":[]} -->

Build ID: 105
Block vanilla build status: FAILURE 
Stage before exit: e2e-tests 
Jenkins E2E Test Results: 
Ran 45 of 195 Specs in 11417.785 seconds
FAIL! -- 44 Passed | 1 Failed | 0 Pending | 150 Skipped
--- FAIL: TestE2E (11417.87s)
FAIL

Ginkgo ran 1 suite in 3h10m57.042729493s
Test Suite Failed
make: Leaving directory `/home/worker/workspace/csi-block-vanilla-pre-check-in/105/vsphere-csi-driver`

[csi-block-vanilla] [csi-supervisor] statefulset
/home/worker/workspace/csi-block-vanilla-pre-check-in/105/vsphere-csi-driver/tests/e2e/statefulsets.go:56
  Statefulset testing with parallel podManagementPolicy [It]
  /home/worker/workspace/csi-block-vanilla-pre-check-in/105/vsphere-csi-driver/tests/e2e/statefulsets.go:266

  Unexpected error:
      <*errors.StatusError | 0xc0004dec80>: {
          ErrStatus: {
              TypeMeta: {Kind: "", APIVersion: ""},
              ListMeta: {
                  SelfLink: "",
                  ResourceVersion: "",
                  Continue: "",
                  RemainingItemCount: nil,
              },
              Status: "Failure",
              Message: "etcdserver: request timed out",
              Reason: "",
              Details: nil,
              Code: 500,
          },
      }
      etcdserver: request timed out
  occurred

The 1 failure is a known intermittent failure which is tracked internally.

cc: @divyenpatel Can you help with approvals & merge the PR

Build ID: 105
Block vanilla build status: FAILURE 
Stage before exit: e2e-tests 
Jenkins E2E Test Results: 
Ran 45 of 195 Specs in 11417.785 seconds
FAIL! -- 44 Passed | 1 Failed | 0 Pending | 150 Skipped
--- FAIL: TestE2E (11417.87s)
FAIL

Ginkgo ran 1 suite in 3h10m57.042729493s
Test Suite Failed
make: Leaving directory `/home/worker/workspace/csi-block-vanilla-pre-check-in/105/vsphere-csi-driver`

Hi @zhelongp. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

<details>

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. </details>

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: <a href="https://github.com/kubernetes-sigs/vsphere-csi-driver/pull/1014#" title="Author self-approved">zhelongp</a> To complete the pull request process, please assign chethanv28 after the PR has been reviewed. You can assign the PR to them by writing /assign @chethanv28 in a comment when ready.

The full list of commands accepted by this bot can be found here.

<details open> Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":["chethanv28"]} -->

What this PR does / why we need it: Add four benchmarks to measure the overhead of LogNewError methods.

Testing done: Local build and test.

BenchmarkLogNewError-2 2715090 439 ns/op BenchmarkLogNewErrorf-2 2144805 528 ns/op BenchmarkLogNewErrorCode-2 2473299 457 ns/op BenchmarkLogNewErrorCodef-2 2064558 589 ns/op