Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dockerhub replication: 429 Rate limit exceeded on project with lots of tags #19967

Open
Grimm75 opened this issue Feb 9, 2024 · 1 comment · Fixed by Grimm75/harbor#1 · May be fixed by #19984
Open

dockerhub replication: 429 Rate limit exceeded on project with lots of tags #19967

Grimm75 opened this issue Feb 9, 2024 · 1 comment · Fixed by Grimm75/harbor#1 · May be fixed by #19984
Assignees
@stonezdj
Labels
area/replication Stale

Comments

@Grimm75
Copy link

Grimm75 commented Feb 9, 2024

Expected behavior and actual behavior:
Harbor should be able to replicate artefact when Name and Tag filters are present on any dockerhub project.
Now it fails on projects with large amount of tags, because limit is exhausted before download by fetching all tags.

harbor/src/pkg/reg/adapter/dockerhub/adapter.go

Line 316 in 84b31aa

return fmt.Errorf("get tags for repo '%s/%s' from DockerHub error: %v", repo.Namespace, repo.Name, err)

Steps to reproduce the problem:

  • Create replica for dockerhub project with lots of tags ex.: apache/superset or envoyproxy/envoy and run it.
  • job will fail with 429
    harbor-replication

Versions:
Please specify the versions of following systems.

  • harbor version: [v2.8.4]
  • docker engine version: N/A (Kubernetes deployment)
  • docker-compose version: N/A (Kubernetes deployment)

Additional context:

  • Harbor config files: N/A
  • Log files:
...
2024-02-08T07:35:51+01:00 [INFO] [/pkg/reg/adapter/dockerhub/client.go:92]: GET https://hub.docker.com/v2/repositories/apache/superset/tags/?page=173&page_size=100
2024-02-08T07:35:51+01:00 [INFO] [/pkg/reg/adapter/dockerhub/client.go:92]: GET https://hub.docker.com/v2/repositories/apache/superset/tags/?page=174&page_size=100
2024-02-08T07:35:51+01:00 [INFO] [/pkg/reg/adapter/dockerhub/client.go:92]: GET https://hub.docker.com/v2/repositories/apache/superset/tags/?page=175&page_size=100
2024-02-08T07:35:51+01:00 [INFO] [/pkg/reg/adapter/dockerhub/client.go:92]: GET https://hub.docker.com/v2/repositories/apache/superset/tags/?page=176&page_size=100
2024-02-08T07:35:51+01:00 [INFO] [/pkg/reg/adapter/dockerhub/client.go:92]: GET https://hub.docker.com/v2/repositories/apache/superset/tags/?page=177&page_size=100
2024-02-08T07:35:52+01:00 [INFO] [/pkg/reg/adapter/dockerhub/client.go:92]: GET https://hub.docker.com/v2/repositories/apache/superset/tags/?page=178&page_size=100
2024-02-08T07:35:52+01:00 [INFO] [/pkg/reg/adapter/dockerhub/client.go:92]: GET https://hub.docker.com/v2/repositories/apache/superset/tags/?page=179&page_size=100
2024-02-08T07:35:52+01:00 [ERROR] [/pkg/reg/adapter/dockerhub/adapter.go:438]: list tags error: 429 -- {"detail": "Rate limit exceeded", "error": false}
@Grimm75 Grimm75 mentioned this issue Feb 14, 2024
Merged
@Grimm75 Grimm75 linked a pull request Feb 15, 2024 that will close this issue
Open
5 tasks
@Grimm75 Grimm75 reopened this Feb 15, 2024
@stonezdj stonezdj self-assigned this Feb 19, 2024
@github-actions GitHub Actions
Copy link

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

@github-actions github-actions bot added the Stale label Apr 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stonezdj stonezdj

Labels
area/replication Stale
Projects
None yet
Milestone
No milestone
2 participants
@stonezdj @Grimm75