Collect requirements/idea/features for v2.8.0

[OrlinVasilev]

Hello Community,

as we released v2.6 and we have plan and roadmap for v2.8 it's time to brainstorm the ideas and requirements for the feature list for 2.7!

Please take a look for all candidates for 2.7 that will be transferred to 2.8:
https://github.com/goharbor/harbor/issues?q=is%3Aopen+sort%3Aupdated-desc+label%3Acandidate%2F2.7.0+
And 2.8 candidates already added:
https://github.com/goharbor/harbor/issues?q=is%3Aopen+sort%3Aupdated-desc+label%3Acandidate%2F2.8.0+

Please share your thoughts here so we can all work on proper roadmap and achieving it!
Keep in mind that Harbor is community developed and driven project so not everything can be implemented right away :)
If you have an idea or need and eager to work on, we will support you and help you get there!

Thank you being essential part of the community!

Orlix
Harbor Community Manager

[sagikazarmark]

I'd like to bring up the IAM story/experience again: improving that experience would be crucial for this project IMO.

A couple thoughts I already shared with you in the 2.6 thread: #16214 (comment)

Some of the major pain points again:

• Multi-project teams either require admin access or have to refer to the admins for management
• API access is problematic when OIDC is enabled
• No manual group mgmt when OIDC is enabled

[Vad1mo]

My dream would be to extend Casbin functionality that is used in some areas in Harbor across the whole app.
Also, expose the currently hard coded policies of Casbin to the end user. This allows the Harbor team to ship what we think is a good setup, but allow the user to customize all aspects of AIM to their own needs if they need that.

[lengrongfu]

We also encountered some problems when using OIDC:

1. If the user has not logged in, it does not exist in the system, and the administrator cannot authorize it.
2. The user password used for docker push/pull is different from the password used for OIDC login, so it is very confusing when used in this way.

[captn3m0]

+1 for API Access issues when OIDC is enabled.

[wy65701436]

@captn3m0 what issues do you mean?

[captn3m0]

It's very difficult to be able to use the Harbor API (beyond push/pull) with programmatic access once OIDC is enabled.

• Robot accounts don't have access to the full API: Robot accounts should be allowed to access the full Harbor api #8723, so that's not viable.
• The documented workarounds are problematic, and involve too much overhead. This should be much easier.

Either Robot accounts should get access to the complete API, or the standard user tokens should continue to work with OIDC enabled.

[dioguerra]

Can we split Normal registry operations (push/pull images) from other secundary features (webhooks, image scans, replications)?

#17607

[gwagner681105]

We are currently fighting with proxy and their URL context. Since we are using GitLab with Kaniko to build docker images. Kaniko supports an option like registry mirror (--registry-mirror=myprivateharbor.com), but does not support an additional context, like --registry-mirror=myprivateharbor.com/proxy.

Do you see a benefit in the idea to allow another external_url for a harbor per proxy context? Example:

All internal images will be pulled over companyharbordomain.com/{harborproject}
All proxy pulls will be pulled over companyharborproxydomain.com/{harborproxyname}

With the latter, one can put an Reverse Proxy in Front and forward the request from companyharborproxydomain.com -> companyharborproxydomain.com/{harborproxyname}/v2

[stonezdj]

Harbor doesn't support multiple FQDN for a single host

[wy65701436]

it sounds like a setup problem?

[dkulchinsky]

Hey @OrlinVasilev!

I would like to raise the issue of GC performance and reliability for consideration in v2.8

A good summary for the challenges and issues is described here: #16010

Unfortunately, some of the core problems with GC have not been addressed for over 2 years now (primarily: #12948), in large deployments (> ~1.5TB) GC becomes inoperable and results in considerable cost and wasted resources over time.

It is also a security concern as artifacts linger in the storage bucket when they are expected to be removed regularly.

I was hoping some of this work would be slotted for v2.7, however looks like it didn't make the cut (?), so here's hoping for v2.8 🤞🏼

/cc @Vad1mo

[Vad1mo]

Agree with @dkulchinsky.
GC is not a failsafe, especially if the object storage responds to errors or timeouts. The whole GC job just stops midway.

[dkulchinsky]

Some recent changes in GC made it more fault tolerant, however I think the biggest challenge right now is performance.

In our environment (>1.6TB), GC runs for 24 hours and manages to clean ~200 artifacts, it also gets reaped as it exceeds the max job duration deadline, we delete more artifacts in a day than GC cleans up and our backlog is >180,000 artifacts to delete

The bottleneck is well described here: #12948 (comment) by @twhiteman

[captn3m0]

We're facing issues running GC on a couple of TB worth of cleanup. Reported on Slack here, but from what it looks its mainly #12948.

Would be great for this to be considered in 2.8

[wy65701436]

We've resolved some of them, like Failure tolerance. But some of items listed depends on the upstream distribution. I will continue working with the distribution team to try ro resolve its performance problem.

[qnetter]

Thanks - unfortunately, the 2.7 window was a couple of weeks shorter due to week-long holidays.

[sagikazarmark]

Oh, I know one: ability to configure databases when installing with the operator. It's kinda annoying, that each Harbor instance requires a completely new RDS cluster. :)

[OrlinVasilev]

Hi @sagikazarmark goharbor/community#188 please check this effort here

[sagikazarmark]

Excellent, thanks!

[slist]

Hello,
I would like to be able to scan vulnerabilities with more than one scanner.
Currently, we have trivy by default, but if we add another scanner, we can not use both.
If, for example one scanner is better for Java, the other better for python, and other better for Debian and the other for Alpine... it would be nice to consolidate scan results from all scanners.
Regards

[prahaladdarkin]

@slist If I understand the requirement correctly, what you are proposing is a mechanism to associate scanners (Trivy, PA etc) with types of image classes (Java, Ubuntu, CentOS, Linux etc..). Assuming that the image classes would be specified by the end user and potentially associated with images using some mechanism such as tags or any other means, then when the scan job runs it would instantiate the respective scanner for that class and trigger the scan. The vulnerability reporting however should be able to aggregate the results across all scanners. Is this a fairly correct understanding?
Of course, another approach would be to have a mechanism within the UI and API to specify the scanner to be used for each image but this wont scale when scanning hundreds and thousands of images. Hence, associating scanners with image classes would be the more natural and scalable way to go. cc @wy65701436 @OrlinVasilev

[wy65701436]

to support multiple scanner is a big topic in Harbor, we would like to see if someone from the community can have a proposal on this part, the questions I can immediately think of are:

1, data base scheme update
2. API&UI update
3. how to apply the pull policy bases on the multiple CVE report? one allows your to pull, but others doesn't?
4. how to ensure consistent data from multiple scanners.
5. whether to consolidate scan results from all scanners, why and how?
6. the performance issue on scan all.
7. the CVE export behavior update.

[dkulchinsky]

#17125 didn't make it into 2.7 but was an important candidate (#16775 (comment))

Hope this can get into v2.8

@OrlinVasilev @qnetter

[AlenversFr]

Feature N°1 : exclude trivy scans from last pull datetime indication

When trivy is running a full scan of the registry, it pulls the image in order to scan it.
This is considered as a pull and then the last pull statistic is refreshed on the database.

If we want to implement a retention policy based on the last pull criteria it doesn't work because of trivy.

[dkulchinsky]

absolutely necessary

and would also be an enabler for #17125 which I raised above and in the 2.7 roadmap discussion

[slushysnowman]

This is 100% necessary - we basically CANNOT use image retention policies to auto-remove images at the moment due to this. We want to enable 'time-since-last-pull' image removal for certain types of tags, and since every time Trivy scans an image it is counted as 'pulled' this is basically impossible.

If I had to pick only ONE feature to get into 2.8.0 it would be this one.

[wy65701436]

#14638
This one has been marked to be resolved in v2.8.0

[AlenversFr]

Feature N°2 : RBAC

possibility to disable local admin users except the default system admin when external OIDC is activated

[AlenversFr]

Feature N°3 : make intelligent SCAN ALL

As for now in Harbor we have only 2 options in order to launch a scan on an existing image : Full Scan or Scan artifact from the API.
It is really important to keep security scans up to date and when you have thousands of images the situation is not really managable.

Idea 1 => Ability to schedule a Scan of selected projects
We could configure "batchs of scans" during the week based on how we use the projects objects in harbor.

Idea 2 => have an intelligent scanning system
The scan function acts automatically based on 3 parameters : batch capacity (number of images per batch), schedule (cron like to activate the scan), expiration period (number of days for a valid scan).

Once it is configured, every time de intelligent scan is activated it gets the (batch capacity) oldest scanned images in order to refresh the scan result. It exclude all the images that have a valid scan result based on the expiration period.

[AlenversFr]

From my point of view, the scanning rule based on last pull is not ideal because it depends on how you use "image pull policy" on your infrastructures.
In our case we have 100+ kubernetes clusters and 700 VM with docker engine. We do not want every single host to pull images all the time because in 90% of the time we apply the immutability concept.
There's no need to always pull immutable image like mysuperapp:1.3.2 if it's already cached on the host (or in Dragonfly for some edge cases).

Furthermore (this might differ between end users), when we deploy Harbor there's at least 1 or 2 Trivy containers running waiting for newly created images. We currently have around 700 new images pushed / day, the trivy containers are not used all the time especially during the night.
These 2 containers could be use to refresh the overall scanning status of the registry.

Anyway for sure we need more configuration on scanning policies !
Supply Chain is a major concern, the Harbor project should focus on this subject.

[dkulchinsky]

Hey @AlenversFr, I agree that it's not ideal, however it provides a reasonable method to filter out images that were not pulled from the registry in a fairly lengthy time to ignore them during scans.

We also use the IfNotPresent image pull policy, but with cluster autoscaling and frequent cluster updates/refreshes that any well maintained environment would have, the images that are still in fact in use would be pulled (e.g. on new nodes) which provides a good signal that can determine what should and shouldn't be periodically rescanned.

the above, coupled with things like:

• project inclusion/exclusion rules
• repository inclusion/exclusion rules
• lazy background scanning as you suggested
• etc..

[wy65701436]

@zyyw

[wy65701436]

This idea could be added to the backlog.

[slushysnowman]

As a side note to this - it would be great if for daily scans, Trivy would scan throughout the day, instead of all images at once at the scheduled time - the problem with the current approach, is it creates huge load spikes. Instead if a process looked at the time an image was last scanned, and when it goes over 24h or so, starts the scan, that would spread load across the day

[AlenversFr]

Feature N°4 : possibility for a maintainer user to manually delete imutable tag

The configuration of immutabilty is based exclusively on tag syntax rule and it's not easy to manage for a maintainer because the maintainer is in our case a high level manager who doesn't know anything about containers. A client gets basically 2 harbor projects, one with immutability and the other one without.

In the documentation a maintainer is able to delete image and we'd like this permission to bypass immutability rule.
This action can be done if a tech ask the manager to do it, we have this processus in place for build artifacts.

it's actually available in other solutions like jfrog.

[dkulchinsky]

Support more versatile filtering expression (i.e. regexp) in replication/retention/immutability/etc... rules

This has been raised multiple times over the last few years and is recorded in numerous open issues:

#16131
#8614
#11975
#12877

[slushysnowman]

Agreed, one of those issues is one I created - the current setup results in a lot of 'waste' in terms of storage space, or excess replication rules etc etc

[wy65701436]

cc @qnetter
To use the regexp as the filter for rules could be added into the backlog. But, we should consider:
1, whether to support both double star and regexp.
2, how to deal with the existing data.
3, data migration?

[slushysnowman]

I'd still REALLY like to see #12306 get sorted out - it seems to have not made 2.7 candidacy, but it's a really important feature IMO

As an enterprise running Harbor, and aiming to allow teams to do a lot of self-service, the fact that project admins have the ability to set a project as 'public' with like 2 clicks, while maybe unaware of the implications of doing so, is a very scary prospect.

[OrlinVasilev]

Hi @slushysnowman - are you able to spare some time and to work on the development of that we can support you in that if you are willing to work on it?
Thank you!!

[slushysnowman]

I'll see what I can do! Unfortunately my go isn't exactly up to scratch, so I can't guarantee that my abilities will allow me to get this done, but if I can contribute I will, and it might be good practice for me!

[OrlinVasilev]

Fantastic! you can always reach out to @goharbor/all-maintainers for help and guidance :) slack is open or issue here plenty of options! Thank you!

[thcdrt]

Hello,

A feature that I would love to have, is a global storage quota.
For now, Harbor support project quotas (which is already great), but we would like to have a wider quota that would apply to all the Harbor instance, and that would be ideally configured in Harbor configuration file (not modifiable by Harbor users).
Bonus: When a certain percentage of quota is reached, display a warning message at top of Harbor UI (like for read-only for example).

Thomas,

[thcdrt]

Hello @OrlinVasilev,
I would really love to, but I'm not sure I will have enough time to work on it before the next release. As it would be my first Harbor contribution it will take time to understand base code, and setup dev environment.

[OrlinVasilev]

I suppose we need that to be in 2.8 so we can at least have it in the list https://github.com/goharbor/harbor/labels/target/2.8.0 and https://github.com/goharbor/harbor/labels/candidate/2.8.0

[thcdrt]

Do you want me to create the associated issue ?

[OrlinVasilev]

yes please pining @qnetter for visibility as well!

[thcdrt]

#18011

[OrlinVasilev]

Closing this discussion as we have the 2.9 open now :) #18396