Issue working with PV set as Amazon FSx Windows File server

[foggy-glasses]

Describe the bug
The trow server (pod) throws an error ERROR Failure cataloguing manifest <image> Os { code: 5, kind: Uncategorized, message: "Input/output error" } while attempting to push an image to the registry

To Reproduce

1. Deploy k8s v1.19 with csi-smb-driver on AWS ec2 nodes (1 master 2 worker). Nodes based on RHEL 8.5.
2. Container runtime used is CRI-O and podman is used to tag and push images.
3. Deploy nginx-ingress with hostname mapped to an ELB on AWS that proxies the traffic to the 3 nodes above.
4. Deploy a Windows File server (SMB) using Amazon FSx Windows File Server. This requires creating a AWS managed microsoft AD to which the Windows File Server will be linked.
5. All the above resources are deployed in the same VPC in AWS.
6. Deploy trow (v0.3) with a PV and PVC mapped to the SMB created in step 3.
7. Tag an image to the registry created via trow and attempt to push via podman.

Expected behavior
The image is to be pushed successfully and the repository listing via docker-ls should show the images pushed.

Output/Logs

[ec2-user@ip-10-0-9-28 trow]$ k -n dt-local logs -f trow-set-0
Starting Trow 0.3.4 on 0.0.0.0:8000

Maximum blob size: 8192 Mebibytes
Maximum manifest size: 4 Mebibytes

**Validation callback configuration

  By default all remote images are denied, and all local images present in the repository are allowed

  These host names will be considered local (refer to this registry): ["aws-ec2-registry.domain.com"]
  Images with these prefixes are explicitly allowed: ["k8s.gcr.io/", "docker.io/containersol/trow"]
  Images with these names are explicitly allowed: []
  Local images with these prefixes are explicitly denied: []
  Local images with these names are explicitly denied: []

2022-05-07T14:09:00 [rocket::launch] WARN 🔧 Configured for release.
2022-05-07T14:09:00 [rocket::launch_] WARN address: 0.0.0.0
2022-05-07T14:09:00 [rocket::launch_] WARN port: 8000
2022-05-07T14:09:00 [rocket::launch_] WARN workers: 256
2022-05-07T14:09:00 [rocket::launch_] WARN ident: Rocket
2022-05-07T14:09:00 [rocket::launch_] WARN keep-alive: 5s
2022-05-07T14:09:00 [rocket::launch_] WARN limits: bytes = 8KiB, data-form = 2MiB, file = 1MiB, form = 32KiB, json = 1MiB, msgpack = 1MiB, string = 8KiB
2022-05-07T14:09:00 [rocket::launch_] WARN tls: disabled
2022-05-07T14:09:00 [rocket::launch_] WARN temp dir: /tmp
2022-05-07T14:09:00 [rocket::launch_] WARN log level: critical
2022-05-07T14:09:00 [rocket::launch_] WARN cli colors: true
2022-05-07T14:09:00 [rocket::launch_] WARN shutdown: ctrlc = true, force = true, signals = [SIGTERM], grace = 2s, mercy = 3s
2022-05-07T14:09:00 [rocket::launch] WARN 🛰  Routes:
2022-05-07T14:09:00 [rocket::launch_] WARN (get_homepage) GET /
2022-05-07T14:09:00 [rocket::launch_] WARN (get_v2root) GET /v2
2022-05-07T14:09:00 [rocket::launch_] WARN (login) GET /login
2022-05-07T14:09:00 [rocket::launch_] WARN (healthz) GET /healthz
2022-05-07T14:09:00 [rocket::launch_] WARN (metrics) GET /metrics
2022-05-07T14:09:00 [rocket::launch_] WARN (readiness) GET /readiness
2022-05-07T14:09:00 [rocket::launch_] WARN (validate_image) POST /validate-image
2022-05-07T14:09:00 [rocket::launch_] WARN (get_catalog) GET /v2/_catalog?<n>&<last>
2022-05-07T14:09:00 [rocket::launch_] WARN (get_manifest_history) GET /<onename>/manifest_history/<reference>?<last>&<n>
2022-05-07T14:09:00 [rocket::launch_] WARN (delete_blob) DELETE /v2/<repo>/blobs/<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (list_tags) GET /v2/<repo_name>/tags/list?<last>&<n>
2022-05-07T14:09:00 [rocket::launch_] WARN (delete_image_manifest) DELETE /v2/<repo>/manifests/<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (post_blob_upload) POST /v2/<repo_name>/blobs/uploads
2022-05-07T14:09:00 [rocket::launch_] WARN (get_blob) GET /v2/<name_repo>/blobs/<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (get_manifest) GET /v2/<onename>/manifests/<reference>
2022-05-07T14:09:00 [rocket::launch_] WARN (put_image_manifest) PUT /v2/<repo_name>/manifests/<reference>
2022-05-07T14:09:00 [rocket::launch_] WARN (get_manifest_history_2level) GET /<user>/<repo>/manifest_history/<reference>?<last>&<n>
2022-05-07T14:09:00 [rocket::launch_] WARN (list_tags_2level) GET /v2/<user>/<repo>/tags/list?<last>&<n>
2022-05-07T14:09:00 [rocket::launch_] WARN (post_blob_upload_2level) POST /v2/<repo>/<name>/blobs/uploads
2022-05-07T14:09:00 [rocket::launch_] WARN (get_blob_2level) GET /v2/<name>/<repo>/blobs/<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (delete_blob_2level) DELETE /v2/<user>/<repo>/blobs/<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (delete_image_manifest_2level) DELETE /v2/<user>/<repo>/manifests/<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (put_blob) PUT /v2/<repo_name>/blobs/uploads/<uuid>?<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (patch_blob) PATCH /v2/<repo_name>/blobs/uploads/<uuid>
2022-05-07T14:09:00 [rocket::launch_] WARN (get_manifest_2level) GET /v2/<user>/<repo>/manifests/<reference>
2022-05-07T14:09:00 [rocket::launch_] WARN (put_image_manifest_2level) PUT /v2/<user>/<repo>/manifests/<reference>
2022-05-07T14:09:00 [rocket::launch_] WARN (get_manifest_history_3level) GET /<org>/<user>/<repo>/manifest_history/<reference>?<last>&<n>
2022-05-07T14:09:00 [rocket::launch_] WARN (list_tags_3level) GET /v2/<org>/<user>/<repo>/tags/list?<last>&<n>
2022-05-07T14:09:00 [rocket::launch_] WARN (post_blob_upload_3level) POST /v2/<org>/<repo>/<name>/blobs/uploads
2022-05-07T14:09:00 [rocket::launch_] WARN (get_blob_3level) GET /v2/<org>/<name>/<repo>/blobs/<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (put_blob_2level) PUT /v2/<repo>/<name>/blobs/uploads/<uuid>?<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (patch_blob_2level) PATCH /v2/<repo>/<name>/blobs/uploads/<uuid>
2022-05-07T14:09:00 [rocket::launch_] WARN (delete_blob_3level) DELETE /v2/<org>/<user>/<repo>/blobs/<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (delete_image_manifest_3level) DELETE /v2/<org>/<user>/<repo>/manifests/<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (get_manifest_3level) GET /v2/<org>/<user>/<repo>/manifests/<reference>
2022-05-07T14:09:00 [rocket::launch_] WARN (put_image_manifest_3level) PUT /v2/<org>/<user>/<repo>/manifests/<reference>
2022-05-07T14:09:00 [rocket::launch_] WARN (get_manifest_history_4level) GET /<fourth>/<org>/<user>/<repo>/manifest_history/<reference>?<last>&<n>
2022-05-07T14:09:00 [rocket::launch_] WARN (list_tags_4level) GET /v2/<fourth>/<org>/<user>/<repo>/tags/list?<last>&<n>
2022-05-07T14:09:00 [rocket::launch_] WARN (put_blob_3level) PUT /v2/<org>/<repo>/<name>/blobs/uploads/<uuid>?<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (patch_blob_3level) PATCH /v2/<org>/<repo>/<name>/blobs/uploads/<uuid>
2022-05-07T14:09:00 [rocket::launch_] WARN (post_blob_upload_4level) POST /v2/<fourth>/<org>/<repo>/<name>/blobs/uploads
2022-05-07T14:09:00 [rocket::launch_] WARN (get_blob_4level) GET /v2/<fourth>/<org>/<name>/<repo>/blobs/<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (delete_blob_4level) DELETE /v2/<fourth>/<org>/<user>/<repo>/blobs/<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (delete_image_manifest_4level) DELETE /v2/<fourth>/<org>/<user>/<repo>/manifests/<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (get_manifest_4level) GET /v2/<fourth>/<org>/<user>/<repo>/manifests/<reference>
2022-05-07T14:09:00 [rocket::launch_] WARN (put_image_manifest_4level) PUT /v2/<fourth>/<org>/<user>/<repo>/manifests/<reference>
2022-05-07T14:09:00 [rocket::launch_] WARN (get_manifest_history_5level) GET /<fifth>/<fourth>/<org>/<user>/<repo>/manifest_history/<reference>?<last>&<n>
2022-05-07T14:09:00 [rocket::launch_] WARN (list_tags_5level) GET /v2/<fifth>/<fourth>/<org>/<user>/<repo>/tags/list?<last>&<n>
2022-05-07T14:09:00 [rocket::launch_] WARN (put_blob_4level) PUT /v2/<fourth>/<org>/<repo>/<name>/blobs/uploads/<uuid>?<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (patch_blob_4level) PATCH /v2/<fourth>/<org>/<repo>/<name>/blobs/uploads/<uuid>
2022-05-07T14:09:00 [rocket::launch_] WARN (post_blob_upload_5level) POST /v2/<fifth>/<fourth>/<org>/<repo>/<name>/blobs/uploads
2022-05-07T14:09:00 [rocket::launch_] WARN (get_blob_5level) GET /v2/<fifth>/<fourth>/<org>/<name>/<repo>/blobs/<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (delete_blob_5level) DELETE /v2/<fifth>/<fourth>/<org>/<user>/<repo>/blobs/<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (delete_image_manifest_5level) DELETE /v2/<fifth>/<fourth>/<org>/<user>/<repo>/manifests/<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (get_manifest_5level) GET /v2/<fifth>/<fourth>/<org>/<user>/<repo>/manifests/<reference>
2022-05-07T14:09:00 [rocket::launch_] WARN (put_image_manifest_5level) PUT /v2/<fifth>/<fourth>/<org>/<user>/<repo>/manifests/<reference>
2022-05-07T14:09:00 [rocket::launch_] WARN (put_blob_5level) PUT /v2/<fifth>/<fourth>/<org>/<repo>/<name>/blobs/uploads/<uuid>?<digest>
2022-05-07T14:09:00 [rocket::launch_] WARN (patch_blob_5level) PATCH /v2/<fifth>/<fourth>/<org>/<repo>/<name>/blobs/uploads/<uuid>
2022-05-07T14:09:00 [rocket::launch_] WARN (post_blob_upload_6level) POST /v2/<sixth>/<fifth>/<fourth>/<org>/<repo>/<name>/blobs/uploads
2022-05-07T14:09:00 [rocket::launch] WARN 👾 Catchers:
2022-05-07T14:09:00 [rocket::launch_] WARN (not_found) 404
2022-05-07T14:09:00 [rocket::launch_] WARN (no_auth) 401
2022-05-07T14:09:00 [rocket::launch] WARN 📡 Fairings:
2022-05-07T14:09:00 [rocket::launch_] WARN Launch Message (liftoff)
2022-05-07T14:09:00 [rocket::launch_] WARN Set API Version Header (response)
2022-05-07T14:09:00 [rocket::launch_] WARN Shield (liftoff, response, singleton)
Trow is up and running!
2022-05-07T14:09:00 [rocket::launch] WARN 🚀 Rocket has launched from http://0.0.0.0:8000


2022-05-07T14:09:20 [_] WARN `TrowToken` request guard failed: ().
2022-05-07T14:09:20 [_] WARN Responding with registered (no_auth) 401 catcher.
2022-05-07T14:09:20 [_] ERROR No matching routes for HEAD /v2/busybox/blobs/sha256:eb6b01329ebe73e209e44a616a0e16c2b8e91de6f719df9c35e6cdadadbe5965.
2022-05-07T14:09:20 [trow_server::server] WARN Request for unknown blob: "./data/blobs/sha256/eb6b01329ebe73e209e44a616a0e16c2b8e91de6f719df9c35e6cdadadbe5965"
2022-05-07T14:09:20 [trow::client_interface] WARN Error getting manifest Status { code: NotFound, message: "No blob found matching BlobRef { repo_name: \"busybox\", digest: \"sha256:eb6b01329ebe73e209e44a616a0e16c2b8e91de6f719df9c35e6cdadadbe5965\" }", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Sat, 07 May 2022 14:09:20 GMT"} }, source: None }
2022-05-07T14:09:20 [_] WARN Response was `None`.
2022-05-07T14:09:20 [_] WARN Responding with registered (not_found) 404 catcher.
2022-05-07T14:09:20 [_] ERROR No matching routes for HEAD /v2/busybox/blobs/sha256:a217ebab39fc6f492b573b3d24df6d40b20ce1f830653bf16f73b1ddea5a8307.
2022-05-07T14:09:21 [_] ERROR No matching routes for HEAD /v2/busybox/blobs/sha256:1a80408de790c0b1075d0a7e23ff7da78b311f85f36ea10098e4a6184c200964.
2022-05-07T14:09:21 [trow_server::server] ERROR Failure cataloguing manifest busybox/latest Os { code: 5, kind: Uncategorized, message: "Input/output error" }

Output from the push command

[ec2-user@ip-10-0-9-28 trow]$ sudo podman push aws-ec2-registry.domain.com/busybox:1.0
Getting image source signatures
Copying blob eb6b01329ebe skipped: already exists
Copying config 1a80408de7 done
Writing manifest to image destination
Error: writing manifest "{\"schemaVersion\":2,\"mediaType\":\"application/vnd.docker.distribution.manifest.v2+json\",\"config\":{\"mediaType\":\"application/vnd.docker.container.image.v1+json\",\"size\":1457,\"digest\":\"sha256:1a80408de790c0b1075d0a7e23ff7da78b311f85f36ea10098e4a6184c200964\"},\"layers\":[{\"mediaType\":\"application/vnd.docker.image.rootfs.diff.tar.gzip\",\"size\":0,\"digest\":\"sha256:a217ebab39fc6f492b573b3d24df6d40b20ce1f830653bf16f73b1ddea5a8307\"}]}": uploading manifest 1.0 to aws-ec2-registry.domain.com/busybox: received unexpected HTTP status: 500 Internal Server Error

Trow Info

• Install method - Manual (modified the statefulset and service files from the install directory)
• Version/Image Name - containersol/trow:0.3

Kubernetes

• Kubernetes distro/host (AWS EC2 nodes running RHEL 8.5)
• Kubernetes version (1.19)
• Container Runtime (cri-o://1.19.2)

Additional context
I made few changes to the Trow statefulset. Running as user 333333 threw an error as "Operation Not Permitted". Setting the attribute uidNumber to 333333 on the AD for the user account used to access the SMB did not help. Hence, I updated the securityContext and set the runAsUser, runAsGroup and fsGroup to 0 and now got this error.

I confirmed that I'm able to successfully write to the SMB file share by running a busybox container and writing to a sub directory on the SMB (AWS FSx).

[amouat]

Interesting. The problem is it will take some effort to recreate this :(

[foggy-glasses]

@amouat I'll be happy to collaborate with you on a zoom link for an hour, if that's alright with you.