Accelerated Image Integration For Harbor

[imeoer]

Why we need accelerated image?

Pulling image is one of the time-consuming steps in the container lifecycle. Research shows that time to take for pull operation accounts for 76% of container startup time FAST '16. The P2P-based file distribution system can help reduce network latency when pulling images at runtime, saving network bandwidth and greatly reducing the impact of registry single point of failure, which is very important for large-scale container scaling scenarios.

This research FAST '16 also shows that only about 6.4% of the data is read at runtime, meaning that most of the data take up and wastes network bandwidth and container startup time when pulling images or doing advance preheat of images. Accelerated image formats like Nydus and eStargz aim to address this issue, and they do a lot of work on data de-duplication, end-to-end data integrity, and the compatibility for OCI artifacts spec and distribution spec.

It would be very useful if Harbor could provide accelerated image integration directly. Harbor combines the P2P system and accelerated image format to provide a large-scale efficient, secure joint image solution for the container ecosystem.

How to integrate accelerated image?

User can use the nydusify or ctr-remote tools to convert image to Nydus or eStargz image, respectively. But we can introduce a standalone image conversion service (ICS for short) designed to provide Harbor with the ability to automatically convert image to accelerated image.

When a user does something such as artifact push, Harbor will make a request to ICS to complete the corresponding image conversion through its integrated Nydus, eStargz, etc. drivers, see more detail in this proposal.

There are currently two ways considered to integrate ICS with Harbor, one more loosely and the other more deeply.

Option 1: Integrate ICS with webhook

Design

The design principle of this option is to decouple the ICS component from the harbor core as much as possible to keep it independent, with ICS acting as a webhook server and harbor using webhook request to notify the ICS service to trigger image conversion action. For example, an accelerated image conversion can be triggered when an artifact push event occurs, involving the following webhook configuration:

• Notify Type: ICS acts as an HTTP server to listen for webhook request, so select to HTTP;
• Event Type: Usually the artifact push is chosen, but other options can also be considered;
• Endpoint URL: The endpoint address of ICS HTTP server listens;
• Auth Header: The HTTP auth header checks configured on the ICS side are used by the ICS to verify the webhook request source;
• Verify Remote Certificate：Determine whether the webhook should verify the certificate of ICS endpoint;

The following features are not yet supported in the current version of harbor and should be proposed and implemented:

• Webhook Filter: Harbor should provide some filtering rules to allow users to decide which images can be converted by ICS, such as limiting to a specified repository or tag;
• Webhook Re-schedule：The ability to automatically re-trigger a webhook request or allow the user to manually trigger it if the webhook request fails to be sent (e.g. ICS endpoint is unreachable), or if the image conversion fails on ICS;

Workflow

• Create a new HTTP type webhook, set the endpoint URL provided by ICS, allows to configure when to trigger image conversion based on event types (such as artifact push) and filter conditions (such as repository name);
• When a user pushes an image that meets the webhook rules for conversion, a webhook request will be sent to ICS, wait a while, and the ICS will complete the conversion.

Questions

1. How does the user know if a conversion task has succeeded or failed?
2. How does the user trigger the conversion of an existing image?

Option 2: Integrate ICS with job service

Design

The design will integrate ICS more deeply with Harbor for a better user experience. When the push image event occurs, harbor sends an image conversion job to the job service based on conversion configuration to schedule the ICS to complete image conversion.

Workflow

• Adding access configuration to the ICS in the Harbor web portal, including the ICS endpoint, image conversion rules (e.g. which projects need to have conversions enabled).
• When a user pushes an image, the event controller dispatches the event to the conversion controller.
• The conversion controller creates a conversion job and runs it in the job service, which uses the ICS client to request that the ICS convert image.
• After the ICS completes the conversion, it will push the converted image to the Harbor registry, and the event controller will receive the event and write the association between the original image and the converted image to the database.
• User can trigger image conversion through the Harbor web portal, or by calling the conversion API.

What do you think about image conversion integration?

[ktock]

Thank you for proposing this! Some thoughts on the proposal:

How does ICS get creds to access the registry? According to the discussion in goharbor/community#167 (comment), ICS should obtain creds on its own instead of getting them from harbor.
I think there are at least two possible ways to configure creds for ICS:

• configuring ~/.docker/config.json on the node
• allowing access to Secrets of k8s API

either way, I think we need ways to manage registry access permissions per user (but I'm not sure how we can achieve it as of now).

According to the proposal, the registry client can know the association of the original image and the converted one only through Web Portal, right? WDYT about baking the association to the image index (discussed in goharbor/community#167) for supporting other registry clients? For avoiding the race condition mentioned in goharbor/community#167, I guess we need something like opencontainers/distribution-spec#251?

[imeoer]

@ktock Thanks for sharing your thoughts.

How does ICS get creds to access the registry?

Configuring creds directly for ICS is a feasible option, but I think there is another way to do it in Harbor.

Harbor can create a robot account automatically, the HTTP request sent by Harbor to ICS can take the robot account secret, which is used by ICS to pull/push the image, and this secret only has permissions of accessing the image. This approach has been used in the Harbor scanner, see the implementation.

How the registry client knows the association between the original image and converted one?

Agree with your point, at this stage, using a manifest index for image association is a natural way to do it, can eStargz image be put in the manifest index? Distribution support for conditional requests seems to be a solution for the race issue, and it would look great if it can be supported by Harbor/Registry. @steven-zou @reasonerjt @wy65701436 @liubogithub , WDYT?

[reasonerjt]

AFAIK, in the spec there's no API to query what index an image belongs to, hence, if we use the index approach, when a client tries to pull the transformed image via the URI there's no way to know the original image unless we create the reverse index to track all reference relationships?

[imeoer]

@reasonerjt Nydus image that is placed in the manifest index is added a platform -> os.features flag, and Nydus is currently modifying CRI to allow user to select which images (original/converted) need to be used. For docker or other clients, they are select the original image by default. Does eStargz have any experience with using image index @ktock?

[ktock]

@imeoer

eStargz have any experience with using image index

eStargz doesn't directly change to the manifest index and we don't have restrictions about the structure. But the conversion can indirectly change the digest of that index because it converts each layer to eStargz (mediatype will be gzip) and adds an annotation to each layer. Users can push the converted image either using the same tag or a different one (e.g. adding -esgz prefix).

[dioguerra]

Hello, i was just getting into this, and the implementation (currently choosen) seems to be option 1.
https://github.com/goharbor/community/pull/167/files

Are we still planing on opting to go for 2, in similar context with what happens with Trivy?

Otherwise i could do some contribution to the helm chart...

[imeoer]

@dioguerra Thanks for your attention, yes, trivy used option 2-like design, currently we are working on option 1 for accelerated image conversion, and option 2 will not be implemented in short term.