How should we distribute piped agent?

[kentakozuka]

Background

Before the introduction of CUSTOM_SYNC stage (issue #1089), PipeCD only supported built-in platform providers.

We want to give users more freedom to sync/apply/reconcile their applications via CUSTOM_SYNC stage, in which they can use $SHELL commands to deploy scripts. With that change, we thought piped needed a way to manage which binary/commands are available and which are not, which led to the introduction of asdf as a binary management tool for all (ref: docs dev).

The problem is that in some cases, asdf only installs a binary and does not ensure the installed binary works (issue #4376).

Based on the fact that we should focus on the functionality of the piped agent while keeping the cost of maintaining manifests and dependencies as low as possible (simpler is better), we think instead of trying to find the best binary management which fits all the cases, we should keep the focus on the piped running functional and lets users handle the environment.

Ideas

• Core team provides binaries and minimal images only
• The community will create the images to be able to use sam commands, etc.
• Documentation should clearly state how to use custom images.
• Remove asdf
• Clean up piped-base. Remove INSTALL for each command

[kentakozuka]

• Community members contribute to a repository for creating custom images for Custom Sync.

[kentakozuka]

Just thought;
We simply create asdf plugin for the requirements of Python installation.
@khanhtc1202 wdyt?

Though I am not sure it would be possible...

[khanhtc1202]

That's a point. If we choose to go that way, we have to keep supporting asdf plugins like that. And besides, even if we have a way to install Python using asdf, as the example, the sam-cli users have to know and configure the piped/application externalTools to install python then sam-cli which is not very natural to me.

I mean, compare installing all required commands via declarable Dockerfile vs pipecd asdf plugin management python then sam-cli, I think the first way is easier to understand.

[kentakozuka]

That's true 👍
Let's go a way to the first one 🚀

[kentakozuka]

What is the best name for a repository where the community can maintain piped agent images?

[kentakozuka]

What about piped-images?

[khanhtc1202]

@kentakozuka I think that name is a bit confusing, but I don't have a better one now as well 😓 Lets use that name and add description to its readme 👍

[khanhtc1202]

Update:

• We removed the asdf installation/usage from the pipecd project (via Remove external tool management #4383)
• Remove all external tools installed by default from the piped-based image (via Remove pre-installation tools from piped-base image #4400)

[ffjlabo]

We will discuss about preparing the tools by container image later 👍

[Jennifer2112]

To distribute piped agents effectively, you need to assess each agent's workload and skills, and then use a routing system to send calls to the most suitable agent based on factors like availability and expertise. It's important to automate this process as much as possible to reduce errors and save time. Monitoring agent performance and gathering feedback regularly can help you optimize the distribution system for better results. Training agents regularly is also crucial to improve their skills and efficiency in handling piped calls.