Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PodVolumeRestore remains InProgress after status reports that all data has been processed #7798

Open
cbodonnell opened this issue May 15, 2024 · 4 comments
Open

PodVolumeRestore remains InProgress after status reports that all data has been processed #7798

cbodonnell opened this issue May 15, 2024 · 4 comments
Assignees
@Lyndon-Li
Labels
area/fs-uploader depends-upstream The issue/PR that has dependency on external or upstream projects downstream-integration The issue originated from downstream integration

Comments

@cbodonnell
Copy link

What steps did you take and what happened:

Hello, my goal is to be able to present the progress of a restore to a user. One piece of this is the PodVolumeRestores that are executed during the restore process. I've noticed that the TotalBytes and BytesDone status fields don't seem to represent the progress as I would expect and am wondering if this is a bug or if there is some other details that I should be looking at during the volume restore?

For example, I have a roughly 10G volume that I am restoring and the progress shows 100% after a rather short time, but the PodVolumeRestore remains with an InProgress status for several more minutes before becoming Completed.

$ kubectl get podvolumerestores -n velero -w
NAME                            NAMESPACE   POD                         UPLOADER TYPE   VOLUME   STATUS       TOTALBYTES    BYTESDONE  AGE
instance-cfzp8.registry-8brkh   registry    registry-744bcbf579-4w4kw   kopia           data                                           0s
instance-cfzp8.registry-8brkh   registry    registry-744bcbf579-4w4kw   kopia           data     InProgress                            3s
instance-cfzp8.registry-8brkh   registry    registry-744bcbf579-4w4kw   kopia           data     InProgress   10421465232   10421465232   58s
instance-cfzp8.registry-8brkh   registry    registry-744bcbf579-4w4kw   kopia           data     Completed    10421465232   10421465232   8m11s

In the node-agent logs I see the following that seems to indicate the actual Kopia restore for this volume took ~8 minutes:

time="2024-05-15T17:36:02Z" level=info msg="Starting restore" PodVolumeRestore=velero/instance-cfzp8.registry-8brkh controller=PodVolumeRestore logSource="pkg/uploader/provider/kopia.go:221" pod=registry/registry-744bcbf579-4w4kw restore=velero/instance-cfzp8.registry snapshotID=e001b91afc6a8b5223e4c270c90d398b volumePath="/host_pods/78fb42da-a549-470c-9b88-443d58af4191/volumes/kubernetes.io~local-volume/pvc-1d0a7b65-51ca-4a90-9739-6b69c11d94d3"
time="2024-05-15T17:36:02Z" level=info msg="Start to restore..." PodVolumeRestore=velero/instance-cfzp8.registry-8brkh controller=PodVolumeRestore logSource="pkg/uploader/kopia/snapshot.go:379" pod=registry/registry-744bcbf579-4w4kw restore=velero/instance-cfzp8.registry snapshotID=e001b91afc6a8b5223e4c270c90d398b volumePath="/host_pods/78fb42da-a549-470c-9b88-443d58af4191/volumes/kubernetes.io~local-volume/pvc-1d0a7b65-51ca-4a90-9739-6b69c11d94d3"
time="2024-05-15T17:36:02Z" level=info msg="Restore from snapshot e001b91afc6a8b5223e4c270c90d398b, description Kopia Uploader, created time 2024-05-14 21:22:55.12187834 +0000 UTC, tags map[backup:instance-cfzp8 backup-uid:b04e7149-1b65-465e-a45d-b84247747244 ns:registry pod:registry-744bcbf579-4w4kw pod-uid:6419fa43-77e1-4bf5-95b7-ee02d7cebc14 pvc-uid:dd711408-8e30-4209-9134-dee3489cb00b snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia volume:data]" PodVolumeRestore=velero/instance-cfzp8.registry-8brkh controller=PodVolumeRestore logSource="pkg/uploader/kopia/snapshot.go:388" pod=registry/registry-744bcbf579-4w4kw restore=velero/instance-cfzp8.registry snapshotID=e001b91afc6a8b5223e4c270c90d398b volumePath="/host_pods/78fb42da-a549-470c-9b88-443d58af4191/volumes/kubernetes.io~local-volume/pvc-1d0a7b65-51ca-4a90-9739-6b69c11d94d3"
time="2024-05-15T17:44:08Z" level=info msg="Kopia restore finished, restore size 10421465232, file count 61" PodVolumeRestore=velero/instance-cfzp8.registry-8brkh controller=PodVolumeRestore logSource="pkg/uploader/provider/kopia.go:250" pod=registry/registry-744bcbf579-4w4kw restore=velero/instance-cfzp8.registry snapshotID=e001b91afc6a8b5223e4c270c90d398b volumePath="/host_pods/78fb42da-a549-470c-9b88-443d58af4191/volumes/kubernetes.io~local-volume/pvc-1d0a7b65-51ca-4a90-9739-6b69c11d94d3"

Is this expected that the BytesDone would be 100% of the TotalBytes while the restore still appears to be InProgress for a while longer?

What did you expect to happen:

I would expect that the PodVolumeRestore would reach a Completed state shortly after all bytes have been processed and that the BytesDone indicates the relative progress of the volume restore.

The following information will help us better understand what's going on:

bundle-2024-05-15-18-04-26.tar.gz

Anything else you would like to add:

As mentioned, my goal is to display some sort of progress indicator to the user during a volume restore. Are thare any other resources outside of PodVolumeRestore that I should be considering?

Environment:

  • Velero version (use velero version):
Client:
        Version: v1.13.2
        Git commit: 4d961fb6fec384ed7f3c1b7c65c818106107f5a6
Server:
        Version: v1.13.2
  • Velero features (use velero client config get features):
  • Kubernetes version (use kubectl version):
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.4+k0s
  • Kubernetes installer & version:
  • Cloud provider or hardware configuration: GCP
  • OS (e.g. from /etc/os-release):
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

  • 👍 for "I would like to see this bug fixed as soon as possible"
  • 👎 for "There are more important bugs to focus on right now"
@cbodonnell
Copy link
Author

cbodonnell commented May 15, 2024 •
edited

I'll add that this is with Kopia, but I observed similar behavior when using Restic where it remained InProgress for several minutes after reaching 100% of data processed.

@Lyndon-Li
Copy link
Contributor

This may be related to your file system data, e.g., are there many actually-zero-byte items (empty dirs, symbol links, sparse files)?

@cbodonnell
Copy link
Author

cbodonnell commented May 16, 2024 •
edited

@Lyndon-Li For some context, this particular volume is the backend for a Distribution OCI registry. I've done a more recent test today with some different images and still see similar behavior with regards to the progress status:

$ kubectl get podvolumerestores -n velero -w
NAME                            NAMESPACE   POD                         UPLOADER TYPE   VOLUME   STATUS       TOTALBYTES    BYTESDONE  AGE
instance-djdsk.registry-hkrfk   registry    registry-5bcdf4b5d7-bhpjq   kopia           data                                           0s
instance-djdsk.registry-hkrfk   registry    registry-5bcdf4b5d7-bhpjq   kopia           data     InProgress                            3s
instance-djdsk.registry-hkrfk   registry    registry-5bcdf4b5d7-bhpjq   kopia           data     InProgress   12729478850   12729478850   18s
instance-djdsk.registry-hkrfk   registry    registry-5bcdf4b5d7-bhpjq   kopia           data     Completed    12729478850   12729478850   3m14s

time="2024-05-16T13:59:35Z" level=info msg="Starting restore" PodVolumeRestore=velero/instance-djdsk.registry-hkrfk controller=PodVolumeRestore logSource="pkg/uploader/provider/kopia.go:221" pod=registry/registry-5bcdf4b5d7-bhpjq restore=velero/instance-djdsk.registry snapshotID=790f5beda880ff4ee5a0dee4969209fc volumePath="/host_pods/1c99d44a-70b4-48dd-b75c-db80b916aff8/volumes/kubernetes.io~local-volume/pvc-6cf42a4a-839c-43ce-9417-b5885a8f230b"
time="2024-05-16T13:59:35Z" level=info msg="Start to restore..." PodVolumeRestore=velero/instance-djdsk.registry-hkrfk controller=PodVolumeRestore logSource="pkg/uploader/kopia/snapshot.go:379" pod=registry/registry-5bcdf4b5d7-bhpjq restore=velero/instance-djdsk.registry snapshotID=790f5beda880ff4ee5a0dee4969209fc volumePath="/host_pods/1c99d44a-70b4-48dd-b75c-db80b916aff8/volumes/kubernetes.io~local-volume/pvc-6cf42a4a-839c-43ce-9417-b5885a8f230b"
time="2024-05-16T13:59:35Z" level=info msg="Restore from snapshot 790f5beda880ff4ee5a0dee4969209fc, description Kopia Uploader, created time 2024-05-16 13:12:27.785796608 +0000 UTC, tags map[backup:instance-djdsk backup-uid:78226ecd-488d-4ed6-b70d-cccfe29b24f3 ns:registry pod:registry-5bcdf4b5d7-bhpjq pod-uid:caf302bf-18aa-4e28-81e8-a7d74166f1a3 pvc-uid:7f55fb82-1882-4341-91e7-d862da323c48 snapshot-requester:pod-volume-backup-restore snapshot-uploader:kopia volume:data]" PodVolumeRestore=velero/instance-djdsk.registry-hkrfk controller=PodVolumeRestore logSource="pkg/uploader/kopia/snapshot.go:388" pod=registry/registry-5bcdf4b5d7-bhpjq restore=velero/instance-djdsk.registry snapshotID=790f5beda880ff4ee5a0dee4969209fc volumePath="/host_pods/1c99d44a-70b4-48dd-b75c-db80b916aff8/volumes/kubernetes.io~local-volume/pvc-6cf42a4a-839c-43ce-9417-b5885a8f230b"
time="2024-05-16T14:02:45Z" level=info msg="Kopia restore finished, restore size 12729478850, file count 87" PodVolumeRestore=velero/instance-djdsk.registry-hkrfk controller=PodVolumeRestore logSource="pkg/uploader/provider/kopia.go:250" pod=registry/registry-5bcdf4b5d7-bhpjq restore=velero/instance-djdsk.registry snapshotID=790f5beda880ff4ee5a0dee4969209fc volumePath="/host_pods/1c99d44a-70b4-48dd-b75c-db80b916aff8/volumes/kubernetes.io~local-volume/pvc-6cf42a4a-839c-43ce-9417-b5885a8f230b"

bundle-2024-05-16-14-19-20.tar.gz

Here's some details on the filesystem - a few empty dirs, several small (non-empty) files, and some larger files 
$ tree . -s
[       4096]  .
└── [       4096]  docker
  └── [       4096]  registry
      └── [       4096]  v2
          ├── [       4096]  blobs
          │   └── [       4096]  sha256
          │       ├── [       4096]  04
          │       │   └── [       4096]  04f4a67bf46c032f001cfa9b922656dea915018ecb603d45250548ae17e5452b
          │       │       └── [ 1091975680]  data
          │       ├── [       4096]  10
          │       │   └── [       4096]  10fb62d84c33d2ce4babf3676fe51de556bae682bc9cb6e3cdd6f10301c14830
          │       │       └── [        397]  data
          │       ├── [       4096]  12
          │       │   └── [       4096]  12e18843c097ded074be03559fa8c452e490ba9b505a8a6566e44f38c9ef3069
          │       │       └── [       8966]  data
          │       ├── [       4096]  16
          │       │   └── [       4096]  16eaaaf5f1c0db05389b2c1c2d90db8c9154289520fad38d580acdf2390e846e
          │       │       └── [       1398]  data
          │       ├── [       4096]  24
          │       │   ├── [       4096]  249f59e1dec7f7eacbeba4bb9215b8000e4bdbb672af523b3dacc89915b026ae
          │       │   │   └── [      11417]  data
          │       │   └── [       4096]  24d9866c72b8bf308ad20399a0b8ae2cfde1ee09b31408e0eefc640088b1e00e
          │       │       └── [       3685]  data
          │       ├── [       4096]  2a
          │       │   └── [       4096]  2a2b68ac8c8a0b1c7ad026ed30789509cb9ad9f235c213d6030edadf55372f39
          │       │       └── [        611]  data
          │       ├── [       4096]  2b
          │       │   └── [       4096]  2b90d2b26936bcb151445decd7b3090669e94e7c442782ef4c23b63a6b72ecd3
          │       │       └── [     264454]  data
          │       ├── [       4096]  2e
          │       │   └── [       4096]  2e9f6d8a64372a8b9e4d13b53d72daed57341a65a6683b630345893af4bcd92c
          │       │       └── [     711259]  data
          │       ├── [       4096]  3c
          │       │   └── [       4096]  3c854c8cbf469fda815b8f6183300c07cfa2fbb5703859ca79aff93ae934961b
          │       │       └── [    3379404]  data
          │       ├── [       4096]  44
          │       │   └── [       4096]  44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
          │       │       └── [          2]  data
          │       ├── [       4096]  56
          │       │   ├── [       4096]  5604a14ddd886033b2daf58fcc8b739cf85c1898b3a48572673bf82660373804
          │       │   │   └── [      12652]  data
          │       │   └── [       4096]  56df1c47fb3430195a5ec8e46b6d5945d58d178546f6b5471792b868e14fe728
          │       │       └── [ 3690987039]  data
          │       ├── [       4096]  5e
          │       │   └── [       4096]  5e845cc16269ce694cab9e13e1bdc03d82091f380f1b2975e6985767db717722
          │       │       └── [   11668396]  data
          │       ├── [       4096]  62
          │       │   └── [       4096]  62a036a4d564d18ee8de3ebb3803ea84ac5d97fe8839c76273993936f1cb5584
          │       │       └── [ 3256341243]  data
          │       ├── [       4096]  66
          │       │   └── [       4096]  663fcd6603bbecda62424981d1d9c90f3fba42a21329e5b2bff15aaa13f53a03
          │       │       └── [  668968056]  data
          │       ├── [       4096]  68
          │       │   └── [       4096]  68e7bb398b9ff421236990bfeaf5c1feab26c590eed93489e245375c23551e2a
          │       │       └── [   26708066]  data
          │       ├── [       4096]  7a
          │       │   └── [       4096]  7afc713436b06101731111be08d107fd8339f5d36d10f3638d5f0d7bc050a2fc
          │       │       └── [       5035]  data
          │       ├── [       4096]  81
          │       │   └── [       4096]  81e20d5bf114e8e408cc465d9b67f6cd134d2c688158565a936cf6ce25d927c1
          │       │       └── [ 3620811120]  data
          │       ├── [       4096]  86
          │       │   └── [       4096]  86ff50a23c5951ba1a297070dc6627beb153c268b26996937c28ca38c0662a59
          │       │       └── [  158006846]  data
          │       ├── [       4096]  92
          │       │   └── [       4096]  928e4a13084bcccb34db9932019598d7fa5b939f377a19ee0a5efba5e884aeaa
          │       │       └── [        599]  data
          │       ├── [       4096]  97
          │       │   └── [       4096]  97daab82b539cf6f6a4dd48ee0a21d4d688f41bd0ce9dcb029026b14831b2e64
          │       │       └── [   39945749]  data
          │       ├── [       4096]  9e
          │       │   └── [       4096]  9e9d525a370ddae32826a814ecc01553847c8543a3024cc7973aaa939a296ea1
          │       │       └── [        262]  data
          │       ├── [       4096]  a0
          │       │   └── [       4096]  a0fbd691d7c1a07fbdeb8b338578f4e199a49e2491eff171105eb0dc7cf61628
          │       │       └── [        768]  data
          │       ├── [       4096]  a3
          │       │   └── [       4096]  a3efca40a7c9ecc35f7afe4e8b40379cead43e480b38dca3317035b27c214d3a
          │       │       └── [  123794939]  data
          │       ├── [       4096]  a6
          │       │   └── [       4096]  a67ef490b2756d033dd7b29a7cfb8da284b0b6d735d42482a7fd033b21e07728
          │       │       └── [        166]  data
          │       ├── [       4096]  af
          │       │   └── [       4096]  af5d3e56b453711084ad3d38ebbdeaf8b6d2487d8e37a233a0113196cbda1c56
          │       │       └── [        181]  data
          │       ├── [       4096]  b4
          │       │   └── [       4096]  b407bcc8063852cf7b980fa6d83d6caa2c17b2fa4c10e87835d72f21ed40c41a
          │       │       └── [        625]  data
          │       ├── [       4096]  c6
          │       │   └── [       4096]  c624f9bfc2ec04e9e8b7668766b61aada0d7b57eed22ae3aaf1021cd677901e2
          │       │       └── [        605]  data
          │       ├── [       4096]  d1
          │       │   └── [       4096]  d15acaad07605937ab40a0b64c7c53c3bbfa0eca0070eaee4f4f5ea294fd2680
          │       │       └── [        222]  data
          │       ├── [       4096]  d7
          │       │   └── [       4096]  d764ae9b7836de16d2e176930e00d3ec85102fdf9629b5eeb17215fcbf4b06b9
          │       │       └── [   33623907]  data
          │       ├── [       4096]  da
          │       │   └── [       4096]  da33b1ad0ac4b49641e40469216939f6488c1d8116b2513ba2caa561d4898067
          │       │       └── [        954]  data
          │       ├── [       4096]  de
          │       │   └── [       4096]  de5d475193dd13b444c2e58fc772d8a3297e370eb90e67e483095bb25f1861a6
          │       │       └── [    1797087]  data
          │       ├── [       4096]  e7
          │       │   └── [       4096]  e7ece7ddc2e44b2af69a1aa282047794bb31287a960f6c9f53e089c269bee1c7
          │       │       └── [       2307]  data
          │       ├── [       4096]  ea
          │       │   └── [       4096]  eaf6678c3c9d8c9d583cdfe95f52d6efabe476f86ac29f4e56f336ff80d3f38a
          │       │       └── [        606]  data
          │       └── [       4096]  ed
          │           └── [       4096]  ed7b1dfccfd019771f48bd2af5dcd516d0ea972d303b06f3c1dbb418b1346af7
          │               └── [     440526]  data
          └── [       4096]  repositories
              ├── [       4096]  cimg
              │   └── [       4096]  unity
              │       ├── [       4096]  _layers
              │       │   └── [       4096]  sha256
              │       │       ├── [       4096]  10fb62d84c33d2ce4babf3676fe51de556bae682bc9cb6e3cdd6f10301c14830
              │       │       │   └── [         71]  link
              │       │       ├── [       4096]  12e18843c097ded074be03559fa8c452e490ba9b505a8a6566e44f38c9ef3069
              │       │       │   └── [         71]  link
              │       │       ├── [       4096]  2b90d2b26936bcb151445decd7b3090669e94e7c442782ef4c23b63a6b72ecd3
              │       │       │   └── [         71]  link
              │       │       ├── [       4096]  2e9f6d8a64372a8b9e4d13b53d72daed57341a65a6683b630345893af4bcd92c
              │       │       │   └── [         71]  link
              │       │       ├── [       4096]  5604a14ddd886033b2daf58fcc8b739cf85c1898b3a48572673bf82660373804
              │       │       │   └── [         71]  link
              │       │       ├── [       4096]  56df1c47fb3430195a5ec8e46b6d5945d58d178546f6b5471792b868e14fe728
              │       │       │   └── [         71]  link
              │       │       ├── [       4096]  62a036a4d564d18ee8de3ebb3803ea84ac5d97fe8839c76273993936f1cb5584
              │       │       │   └── [         71]  link
              │       │       ├── [       4096]  68e7bb398b9ff421236990bfeaf5c1feab26c590eed93489e245375c23551e2a
              │       │       │   └── [         71]  link
              │       │       ├── [       4096]  81e20d5bf114e8e408cc465d9b67f6cd134d2c688158565a936cf6ce25d927c1
              │       │       │   └── [         71]  link
              │       │       ├── [       4096]  86ff50a23c5951ba1a297070dc6627beb153c268b26996937c28ca38c0662a59
              │       │       │   └── [         71]  link
              │       │       ├── [       4096]  97daab82b539cf6f6a4dd48ee0a21d4d688f41bd0ce9dcb029026b14831b2e64
              │       │       │   └── [         71]  link
              │       │       ├── [       4096]  9e9d525a370ddae32826a814ecc01553847c8543a3024cc7973aaa939a296ea1
              │       │       │   └── [         71]  link
              │       │       ├── [       4096]  a3efca40a7c9ecc35f7afe4e8b40379cead43e480b38dca3317035b27c214d3a
              │       │       │   └── [         71]  link
              │       │       ├── [       4096]  a67ef490b2756d033dd7b29a7cfb8da284b0b6d735d42482a7fd033b21e07728
              │       │       │   └── [         71]  link
              │       │       ├── [       4096]  af5d3e56b453711084ad3d38ebbdeaf8b6d2487d8e37a233a0113196cbda1c56
              │       │       │   └── [         71]  link
              │       │       ├── [       4096]  d15acaad07605937ab40a0b64c7c53c3bbfa0eca0070eaee4f4f5ea294fd2680
              │       │       │   └── [         71]  link
              │       │       └── [       4096]  d764ae9b7836de16d2e176930e00d3ec85102fdf9629b5eeb17215fcbf4b06b9
              │       │           └── [         71]  link
              │       ├── [       4096]  _manifests
              │       │   ├── [       4096]  revisions
              │       │   │   └── [       4096]  sha256
              │       │   │       └── [       4096]  24d9866c72b8bf308ad20399a0b8ae2cfde1ee09b31408e0eefc640088b1e00e
              │       │   │           └── [         71]  link
              │       │   └── [       4096]  tags
              │       │       └── [       4096]  9.9
              │       │           ├── [       4096]  current
              │       │           │   └── [         71]  link
              │       │           └── [       4096]  index
              │       │               └── [       4096]  sha256
              │       │                   └── [       4096]  24d9866c72b8bf308ad20399a0b8ae2cfde1ee09b31408e0eefc640088b1e00e
              │       │                       └── [         71]  link
              │       └── [       4096]  _uploads
              └── [       4096]  embedded-cluster-smoke-test-staging-app
                  ├── [       4096]  embedded-cluster
                  │   ├── [       4096]  charts.tar.gz
                  │   │   ├── [       4096]  _layers
                  │   │   │   └── [       4096]  sha256
                  │   │   │       ├── [       4096]  44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
                  │   │   │       │   └── [         71]  link
                  │   │   │       └── [       4096]  ed7b1dfccfd019771f48bd2af5dcd516d0ea972d303b06f3c1dbb418b1346af7
                  │   │   │           └── [         71]  link
                  │   │   ├── [       4096]  _manifests
                  │   │   │   ├── [       4096]  revisions
                  │   │   │   │   └── [       4096]  sha256
                  │   │   │   │       └── [       4096]  928e4a13084bcccb34db9932019598d7fa5b939f377a19ee0a5efba5e884aeaa
                  │   │   │   │           └── [         71]  link
                  │   │   │   └── [       4096]  tags
                  │   │   │       └── [       4096]  2eAqMYG1IEtX8cwpaO1kgNV6EB3-1261-appver-dev-329addc
                  │   │   │           ├── [       4096]  current
                  │   │   │           │   └── [         71]  link
                  │   │   │           └── [       4096]  index
                  │   │   │               └── [       4096]  sha256
                  │   │   │                   └── [       4096]  928e4a13084bcccb34db9932019598d7fa5b939f377a19ee0a5efba5e884aeaa
                  │   │   │                       └── [         71]  link
                  │   │   └── [       4096]  _uploads
                  │   ├── [       4096]  embedded-cluster-amd64
                  │   │   ├── [       4096]  _layers
                  │   │   │   └── [       4096]  sha256
                  │   │   │       ├── [       4096]  44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
                  │   │   │       │   └── [         71]  link
                  │   │   │       └── [       4096]  663fcd6603bbecda62424981d1d9c90f3fba42a21329e5b2bff15aaa13f53a03
                  │   │   │           └── [         71]  link
                  │   │   ├── [       4096]  _manifests
                  │   │   │   ├── [       4096]  revisions
                  │   │   │   │   └── [       4096]  sha256
                  │   │   │   │       └── [       4096]  2a2b68ac8c8a0b1c7ad026ed30789509cb9ad9f235c213d6030edadf55372f39
                  │   │   │   │           └── [         71]  link
                  │   │   │   └── [       4096]  tags
                  │   │   │       └── [       4096]  2eAqMYG1IEtX8cwpaO1kgNV6EB3-1261-appver-dev-329addc
                  │   │   │           ├── [       4096]  current
                  │   │   │           │   └── [         71]  link
                  │   │   │           └── [       4096]  index
                  │   │   │               └── [       4096]  sha256
                  │   │   │                   └── [       4096]  2a2b68ac8c8a0b1c7ad026ed30789509cb9ad9f235c213d6030edadf55372f39
                  │   │   │                       └── [         71]  link
                  │   │   └── [       4096]  _uploads
                  │   ├── [       4096]  images-amd64.tar
                  │   │   ├── [       4096]  _layers
                  │   │   │   └── [       4096]  sha256
                  │   │   │       ├── [       4096]  04f4a67bf46c032f001cfa9b922656dea915018ecb603d45250548ae17e5452b
                  │   │   │       │   └── [         71]  link
                  │   │   │       └── [       4096]  44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
                  │   │   │           └── [         71]  link
                  │   │   ├── [       4096]  _manifests
                  │   │   │   ├── [       4096]  revisions
                  │   │   │   │   └── [       4096]  sha256
                  │   │   │   │       └── [       4096]  eaf6678c3c9d8c9d583cdfe95f52d6efabe476f86ac29f4e56f336ff80d3f38a
                  │   │   │   │           └── [         71]  link
                  │   │   │   └── [       4096]  tags
                  │   │   │       └── [       4096]  2eAqMYG1IEtX8cwpaO1kgNV6EB3-1261-appver-dev-329addc
                  │   │   │           ├── [       4096]  current
                  │   │   │           │   └── [         71]  link
                  │   │   │           └── [       4096]  index
                  │   │   │               └── [       4096]  sha256
                  │   │   │                   └── [       4096]  eaf6678c3c9d8c9d583cdfe95f52d6efabe476f86ac29f4e56f336ff80d3f38a
                  │   │   │                       └── [         71]  link
                  │   │   └── [       4096]  _uploads
                  │   └── [       4096]  version-metadata.json
                  │       ├── [       4096]  _layers
                  │       │   └── [       4096]  sha256
                  │       │       ├── [       4096]  44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
                  │       │       │   └── [         71]  link
                  │       │       └── [       4096]  7afc713436b06101731111be08d107fd8339f5d36d10f3638d5f0d7bc050a2fc
                  │       │           └── [         71]  link
                  │       ├── [       4096]  _manifests
                  │       │   ├── [       4096]  revisions
                  │       │   │   └── [       4096]  sha256
                  │       │   │       └── [       4096]  c624f9bfc2ec04e9e8b7668766b61aada0d7b57eed22ae3aaf1021cd677901e2
                  │       │   │           └── [         71]  link
                  │       │   └── [       4096]  tags
                  │       │       └── [       4096]  2eAqMYG1IEtX8cwpaO1kgNV6EB3-1261-appver-dev-329addc
                  │       │           ├── [       4096]  current
                  │       │           │   └── [         71]  link
                  │       │           └── [       4096]  index
                  │       │               └── [       4096]  sha256
                  │       │                   └── [       4096]  c624f9bfc2ec04e9e8b7668766b61aada0d7b57eed22ae3aaf1021cd677901e2
                  │       │                       └── [         71]  link
                  │       └── [       4096]  _uploads
                  └── [       4096]  nginx
                      ├── [       4096]  _layers
                      │   └── [       4096]  sha256
                      │       ├── [       4096]  16eaaaf5f1c0db05389b2c1c2d90db8c9154289520fad38d580acdf2390e846e
                      │       │   └── [         71]  link
                      │       ├── [       4096]  249f59e1dec7f7eacbeba4bb9215b8000e4bdbb672af523b3dacc89915b026ae
                      │       │   └── [         71]  link
                      │       ├── [       4096]  3c854c8cbf469fda815b8f6183300c07cfa2fbb5703859ca79aff93ae934961b
                      │       │   └── [         71]  link
                      │       ├── [       4096]  5e845cc16269ce694cab9e13e1bdc03d82091f380f1b2975e6985767db717722
                      │       │   └── [         71]  link
                      │       ├── [       4096]  a0fbd691d7c1a07fbdeb8b338578f4e199a49e2491eff171105eb0dc7cf61628
                      │       │   └── [         71]  link
                      │       ├── [       4096]  b407bcc8063852cf7b980fa6d83d6caa2c17b2fa4c10e87835d72f21ed40c41a
                      │       │   └── [         71]  link
                      │       ├── [       4096]  da33b1ad0ac4b49641e40469216939f6488c1d8116b2513ba2caa561d4898067
                      │       │   └── [         71]  link
                      │       └── [       4096]  de5d475193dd13b444c2e58fc772d8a3297e370eb90e67e483095bb25f1861a6
                      │           └── [         71]  link
                      ├── [       4096]  _manifests
                      │   ├── [       4096]  revisions
                      │   │   └── [       4096]  sha256
                      │   │       └── [       4096]  e7ece7ddc2e44b2af69a1aa282047794bb31287a960f6c9f53e089c269bee1c7
                      │   │           └── [         71]  link
                      │   └── [       4096]  tags
                      │       └── [       4096]  1.24-alpine
                      │           ├── [       4096]  current
                      │           │   └── [         71]  link
                      │           └── [       4096]  index
                      │               └── [       4096]  sha256
                      │                   └── [       4096]  e7ece7ddc2e44b2af69a1aa282047794bb31287a960f6c9f53e089c269bee1c7
                      │                       └── [         71]  link
                      └── [       4096]  _uploads

196 directories, 87 files

@Lyndon-Li Lyndon-Li self-assigned this May 17, 2024
@Lyndon-Li Lyndon-Li added the downstream-integration The issue originated from downstream integration label May 24, 2024
@Lyndon-Li
Copy link
Contributor

Here are two reasons that BytesDone equals to TotalBytes before the restore actually completes:

  1. There is a traverse task and a data copy task, traverse task calculates TotalBytes while data copy task increases BytesDone. And these two tasks are running in parallel, not sequential.
  2. A item/file's size is added to the BytesDone at the beginning time of its process, instead of the end of the process. Therefore, say we have a file with 500G in size, you will see BytesDone is added by 500G and then it take a long time to complete the data copy

Therefore, with the current behavior, it is not reliable to decide the completion by comparing BytesDone against TotalBytes

@Lyndon-Li Lyndon-Li added depends-upstream The issue/PR that has dependency on external or upstream projects and removed Needs investigation labels May 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Lyndon-Li Lyndon-Li

Labels
area/fs-uploader depends-upstream The issue/PR that has dependency on external or upstream projects downstream-integration The issue originated from downstream integration
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@reasonerjt @cbodonnell @Lyndon-Li