Thanks!

Describe the bug

Restore stuck in phase "InProgress" for Vanilla cluster.

To Reproduce

1. succeeds to create backup with snapshot by command velero -n wangc backup create bk --snapshot-volumes=true --include-namespaces nginx-example --include-cluster-resources=false --storage-location=bsl-aws --volume-snapshot-locations=vsl-vsphere
2. fails to restore the backup with snapshot by command velero -n wangc restore create --from-backup bk

❯ velero -n wangc restore get
NAME                BACKUP   STATUS       STARTED                         COMPLETED                       ERRORS   WARNINGS   CREATED                         SELECTOR
bk-20210615202001   bk       InProgress   2021-06-15 20:20:04 +0800 CST   <nil>                           0        0          2021-06-15 20:20:05 +0800 CST   <none>

Expected behavior Restore should success if creating backup works.

Troubleshooting Information

[Please refer to the Troubleshooting page and collect the required information]

Velero version: 1.5.3

Velero features (use velero client config get features): features: <NOT SET>

velero-plugin-for-vsphere version: v1.1.1

velero/velero-plugin-for-aws: v1.1.0

Kubernetes cluster flavor: Vanilla

vSphere CSI driver version: harbor-repo.vmware.com/velero/backup-driver:1.1.0-rc6 all images in vsphere-csi-controller deployment and vsphere-csi-node daemonset: harbor-repo.vmware.com/velero/data-manager-for-plugin:1.1.0-rc6

Kubernetes version (use kubectl version) Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"clean", BuildDate:"2021-05-12T14:18:45Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.3", GitCommit:"b3cbbae08ec52a7fc73d334838e18d17e8512749", GitTreeState:"clean", BuildDate:"2019-11-13T11:13:49Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

vCenter version: 7.0.1.10000 ESXi version: 7.0.1

kubectl -n wangc logs deploy/velero
❯ velero -n wangc backup describe bk
Name:         bk
Namespace:    wangc
Labels:       velero.io/storage-location=bsl-aws
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.16.3
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=16

Phase:  Completed

Errors:    0
Warnings:  0

Namespaces:
  Included:  nginx-example
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  excluded

Label selector:  <none>

Storage Location:  bsl-aws

Velero-Native Snapshot PVs:  true

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2021-06-04 18:39:16 +0800 CST
Completed:  2021-06-04 18:39:30 +0800 CST

Expiration:  2021-07-04 18:39:16 +0800 CST

Total items to be backed up:  9
Items backed up:              9

Velero-Native Snapshots: <none included>

❯ velero -n wangc restore describe bk-20210615202001
Name:         bk-20210615202001
Namespace:    wangc
Labels:       <none>
Annotations:  <none>

Phase:  InProgress

Started:    2021-06-15 20:20:04 +0800 CST
Completed:  <n/a>

Backup:  bk

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

❯ kubectl get crd -A
NAME                                                   CREATED AT
backuprepositories.backupdriver.cnsdp.vmware.com       2021-06-04T07:23:24Z
backuprepositoryclaims.backupdriver.cnsdp.vmware.com   2021-06-04T07:23:24Z
backups.velero.io                                      2021-06-07T01:21:22Z
backupstoragelocations.velero.io                       2021-06-07T01:21:22Z
clonefromsnapshots.backupdriver.cnsdp.vmware.com       2021-06-04T07:23:24Z
deletebackuprequests.velero.io                         2021-06-07T01:21:22Z
deletesnapshots.backupdriver.cnsdp.vmware.com          2021-06-04T07:23:24Z
downloadrequests.velero.io                             2021-06-07T01:21:23Z
downloads.datamover.cnsdp.vmware.com                   2021-06-04T07:23:24Z
podvolumebackups.velero.io                             2021-06-07T01:21:23Z
podvolumerestores.velero.io                            2021-06-07T01:21:23Z
resticrepositories.velero.io                           2021-06-07T01:21:23Z
restores.velero.io                                     2021-06-07T01:21:24Z
schedules.velero.io                                    2021-06-07T01:21:24Z
serverstatusrequests.velero.io                         2021-06-07T01:21:24Z
snapshots.backupdriver.cnsdp.vmware.com                2021-06-04T07:23:24Z
uploads.datamover.cnsdp.vmware.com                     2021-06-04T07:23:25Z
volumesnapshotlocations.velero.io                      2021-06-07T01:21:24Z

Screenshots

[If applicable, add screenshots to help explain your problem.]

Anything else you would like to add: backup-driver.log.zip data-mgt.log.zip velero.log.zip

[Miscellaneous information that will assist in solving the issue]

QE is testing 1.2 RC1 now. Should be released soon.

Thanks a lot! It works if both application and its PV are removed.

One short question. When will #341 be in the official release?

What this PR does / why we need it: Currently we hardcoding the default log level by explicitly calling VixDiskLib_Init in VDDK side.

VDDK has a function VixDiskLib_InitEx which allows user to pass a config file which contains customized log level configuration. We have already added the gvddk wrapper function for VixDiskLib_InitEx, we need to update our vsphere plugin to allow user to configure log level during run time.

Test steps:

1. Backup/Restore without config map, default vddk log level is 2
2. Create a configMap to set vddk log level

apiVersion: v1
kind: ConfigMap
metadata:
  # any name can be used; Velero uses the labels (below)
  # to identify it rather than the name
  name: vddk-config
  # must be in the velero namespace
  namespace: velero
  labels:
    # this value-less label identifies the ConfigMap as
    # config for vddk
    velero.io/vddk-config: vix-disk-lib
data:
  # NFC LogLevel (0 = Quiet, 1 = Error, 2 = Warning, 3 = Info, 4 = Debug)
  vixDiskLib.nfc.LogLevel: "4"
  vixDiskLib.transport.LogLevel: "5"

3. Manually restart data manager pod

kubectl -n velero delete deployment.apps/backup-driver
kubectl delete crds backuprepositories.backupdriver.cnsdp.vmware.com \
                    backuprepositoryclaims.backupdriver.cnsdp.vmware.com \
                    clonefromsnapshots.backupdriver.cnsdp.vmware.com \
                    deletesnapshots.backupdriver.cnsdp.vmware.com \
                    snapshots.backupdriver.cnsdp.vmware.com
kubectl -n velero delete daemonset.apps/datamgr-for-vsphere-plugin
kubectl delete crds uploads.datamover.cnsdp.vmware.com downloads.datamover.cnsdp.vmware.com
kubectl -n velero scale deploy/velero --replicas=0
kubectl -n velero scale deploy/velero --replicas=1

4. Backup/Restore, check logs

kubectl exec -n velero -it datamgr-for-vsphere-plugin-XXXXX -- /bin/bash
cd tmp/vmware-root/
cat vixDiskLib-1.log 

...
2021-06-22T10:07:12.914Z| host-33| I005: Setting NFC log level to 1
2021-06-22T10:07:12.914Z| host-33| I005: Setting NFC log level to 2
2021-06-22T10:07:12.919Z| host-33| I005: NFC Async IO session is established for '[vsanDatastore] 693cd160-ba9d-0371-f65b-020074a6838e/2ff18e95ad4f45ac83eade2d3e90bc4b.vmdk' with log level 2.
...

Precheck: https://container-dp.svc.eng.vmware.com/job/Container_Precheck_Velero/1102/ Supervisor/Guest cluster: https://container-dp.svc.eng.vmware.com/view/CNS-DP/job/Velero-Pipeline-WCP/637/ https://container-dp.svc.eng.vmware.com/view/CNS-DP/job/Velero-Pipeline-WCP/639/

Which issue(s) this PR fixes: <!-- Usage: Fixes #<issue number>, or Fixes (paste link of issue). --> Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?: <!-- If no, just write "NONE" in the release-note block below. If yes, a release note is required: Enter your extended release note in the block below. -->

Make vddk log level configurable

Sorry, some of the pre-check tests are failing for this PR !

Please check the Jenkins job for more details https://container-dp.svc.eng.vmware.com/job/CNSDP-CI/555/

Remove this piece of code. Only the first call for vddk initial works as expected, which is called when starting server.

Sorry, some of the pre-check tests are failing for this PR !

Please check the Jenkins job for more details https://container-dp.svc.eng.vmware.com/job/CNSDP-CI/554/

We don't plan to support reloading VDDK log level in the run time. As far as we observe, it doesn't work somehow. Do we really need this piece of code?

FYI, we had a fix at #341 to improve the user experience. Basically, plugin would skip PVC restoration if it already exists. It would be available in the upcoming release.

Currently we hardcoding the default log level by explicitly calling VixDiskLib_Init in VDDK side.

VDDK has a function VixDiskLib_InitEx which allows user to pass a config file which contains customized log level configuration. We have already added the gvddk wrapper function for VixDiskLib_InitEx, we need to update our vsphere plugin to allow user to configure log level during run time.

This change includes:

1. Pick up virtual-disks change.
2. Change to use disklib.InitEx and pass a config file path when initializing vddk lib.

Manually tested. Detail test steps refer to https://github.com/vmware-tanzu/velero-plugin-for-vsphere/pull/354.

VDDK is not used in backup-driver. So it doesn't matter whether we retrieve configured VDDK log level in backup-driver.

Congratulations !!

All the pre-check tests passed https://container-dp.svc.eng.vmware.com/job/CNSDP-CI/553/

Infof

Would you please move this constant definition to the top of this file? Eventually, we might want to move them to a separate file.

this line is unnecessary

Here using labels just to allow user to use any name for vddk config map. Vddk configMap is identified by the specific label, and in code I restrict that only one vddk configMap can exist.

It is better not to retrieve VDDK log level here in snapshot manager, since snapshot manager is shared by both backup-driver server and data-manager server while backup-driver server care nothing about vddk log level. let's do in data-manager server instead

do we really need this line of log?

Errorf -> Error

Info -> Debug

The same as the comment in snapshot manager.

What's the benefit of using label selector here? Do you plan to support retrieving VDDK config information from any ConfigMap with the specific label, rather than the specific ConfigMap name?

the code snippet L387-L390 is used twice. Would you please re-organize your code to get rid of the repeated code snippet?

Infof -> Debugf?

Please do not hardcode anything in func. Make a constant and use it here.

Please double check all logging statement in your PR and remove the end period in each of them.

this.logger.Info("No customized vddk log level provided, set vddk log level as default")

From VDDK log,

2021-06-23T06:31:31.704Z| host-30| I125: Opening file [datastore122b] fcd/b210430c7f774b0280dadee2141beded-000022.vmdk (ha-nfc://[datastore122b] fcd/[email protected]:902)
2021-06-23T06:31:33.214Z| host-87| W115: [NFC ERROR]NfcAioRcvErrorMsg: Error from server: NfcAioProcessOpenFileMsg: Failed to open '[datastore122b] fcd/b210430c7f774b0280dadee2141beded-000022.vmdk': DiskLib error 16392: Failed to lock the file

The root cause is Failed to lock the specific VMDK file. I guess you didn't remove the application before the restore. Basically, you are supposed to remove the backed-up application from the cluster before the restore. velero-plugin-for-vsphere doesn't support in-place restore. I don't think velero does, either.

From backup-driver.log

time="2021-06-23T06:10:44Z" level=info msg="PVC nginx-example/nginx-logs already exists, reusing" logSource="/go/pkg/mod/github.com/vmware-tanzu/[email protected]/pkg/pvc/pvc_protected_entity_type_manager.go:175"

We can confirm the application or PVC of the application is still available in the k8s cluster. If running restore in such case, the observed error Failed to download snapshot, ..., Open virtual disk file failed. The error code is 1. with error code: 1 is expected.