podman build adds "variant" to all images when building multiarch images on MacOS (Apple Silicon)

[edcdavid]

Issue Description

Describe your issue
Running `docker build --platform='linux/arm64,linux/amd64' on MacOS ARM (MAC M series processor) to generate multi architecture images for linux/amd64 introduces a variant from the ARM architecture (v8) in the docker manifest for all target architectures, including intel which does not define variants.

Steps to reproduce the issue

Steps to reproduce the issue

1. podman buildx build --platform=linux/arm64,linux/amd64,linux/s390x,linux/ppc64le --manifest <image_name> -f Dockerfile .
2. doker manifest push <image_name>
3. skopeo inspect --raw docker://<image_name>
4. or podman pull <image_name>

Describe the results you received

Describe the results you received

1. with Skopeo, the intel image, which should not have a variant has one:

    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:d8436858705389ce630788009....141ebbab6a6f09839cf9",
      "size": 2311,
      "platform": {
        "architecture": "amd64",
        "os": "linux",
        "variant": "v8"
      }
    }

2. the docker pull fails with this error:

podman pull <image_name>
Trying to pull <image_name>...
Error: choosing an image from manifest list docker://<image_name>: no image found in image index for architecture amd64, variant "", OS linux

Describe the results you expected

Describe the results you expected

1. with Skopeo, the intel image platform should have no variant in manifest

    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:d8436858705389ce630788009....141ebbab6a6f09839cf9",
      "size": 2311,
      "platform": {
        "architecture": "amd64",
        "os": "linux",
      }
    }

2. the docker pull the image with no error when running on intel architecture
   podman pull <image_name>

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

@nalind PTAL

[mkanoor]

I see the similar issue. When fetching the image on amd64 we get an error about the missing variant.

WARNING: image platform (linux/amd64/v8) does not match the expected platform (linux/amd64)

[nalind]

I'm not able to reproduce this with podman-5.0.3-1.fc40 running in local or remote mode on an aarch64 Linux box, or with the podman 5.0.3 bundled with podman desktop v1.10.3 on top of MacOS 14.4.1 building this example Dockerfile:

FROM busybox
RUN pwd | tee pwd.txt

More information about the environment where it's happening (the podman info output), or a more in-depth description of how to reproduce the problem (a Dockerfile, which would at least rule out bad data coming from base images), would be very helpful in figuring out what's going on here.

[edcdavid]

I was able to reproduce this with your dockerfile. Did you do a just a podman push or podman manifest push ? What is the output of skopeo in your case? This is my podman info:

host:
  arch: arm64
  buildahVersion: 1.35.4
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.39
    systemPercent: 0.03
    userPercent: 0.58
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2040
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.8.8-300.fc40.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 4560195584
  memTotal: 10146938880
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-3.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240426.gd03c4e2-1.fc40.aarch64
    version: |
      pasta 0^20240426.gd03c4e2-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 11h 28m 46.00s (Approximately 0.46 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 0
    stopped: 4
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 50206162944
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 150
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.3
  Built: 1715299200
  BuiltTime: Thu May  9 19:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/arm64
  Version: 5.0.3

[nalind]

I used podman manifest push for each of quay.io/nalind/testing:local (built using podman in the VM), quay.io/nalind/testing:remote (built using podman in the VM, via the podman system service), and quay.io/nalind/testing:mac (podman on the Mac host, talking to podman in a VM managed by Podman Desktop).

[edcdavid]

I rechecked my skopeo output and I also could not reproduce with your Dockerfile, sorry for the error. I created a new Dockerfile where I can reproduce the problem:

➜  podman ls
Dockerfile go.mod     main.go    manager
➜  podman cat Dockerfile 
FROM --platform=${BUILDPLATFORM} golang:1.22 as builder
ARG TARGETOS
ARG TARGETARCH

WORKDIR /workspace
# Copy the Go Modules manifests
COPY go.mod go.mod

# Copy the go source
COPY main.go main.go
RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH} go build -a -o manager main.go

FROM gcr.io/distroless/static:nonroot
WORKDIR /
COPY --from=builder /workspace/manager .
USER 65532:65532

ENTRYPOINT ["/manager"]
➜  podman cat main.go 
package main
import "fmt"
func main() {
    fmt.Println("hello world")
}
➜  podman cat go.mod 
module example.com

go 1.22.3
➜  podman