Handling the piped component in a fully serverless architecture

[ouchi2501]

When working with a current system based on a serverless architecture using API Gateway and Lambda, I want to integrate PipeCD for Lambda's Continuous Delivery and Terraform workflows. Since Piped is expected to run continuously, it seems necessary to set up infrastructure like ECS Fargate for Piped. What are the best practices for managing Piped in a serverless architecture?

[t-kikuc]

Thank you so much!

For AWS, at first, I'll try running the piped on AWS App Runner.

[t-kikuc]

@ouchi2501
I tried and found that there is one difficulty on security, even though it's possible to run a piped on AWS App Runner and deploy apps.
The difficulty is that you need to open the piped container's port of TCP or HTTP in order to pass the health check of App Runner.

For example, the below article uses netcat.
https://speakerdeck.com/gotok365/run-non-apps-with-app-runner?slide=20

[t-kikuc]

I succeeded in passing the health check with:

• port: 9085
• path: /healthz

see below:

• port:

  pipecd/cmd/pipecd/server.go

  Line 104 in 49627aa

  adminPort: 9085,

• path:

  pipecd/cmd/pipecd/server.go

  Line 368 in 49627aa

  admin.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {

[t-kikuc]

Only one difficulty remaining is 'how to pass the piped-config on starting the piped/launcher'.

The problem is that we cannot pass the secret of piped-config, which is fetched from AWS Secrets Manager, in StartCommand of App Runner like launcher --config-data=$PIPED_CONFIG_DATA.
That is because App Runner does not expand env variables in StartCommand.

I created the PR #4780 to solve that.
That enables to read an env variable for piped-config like launcher --config-env-key=PIPED_CONFIG_DATA ,
which is the same as launcher --config-data=$PIPED_CONFIG_DATA

[t-kikuc]

I created the PR #4785, which is better than #4780.
It enables to fetch piped-config from AWS Secrets Manager.

[t-kikuc]

Since #4785 is merged into master,
we can fetch piped-config from AWS Secrets Manager on launching an App Runner service

[t-kikuc]

My below articles are examples for running piped in a serverless infrastructure on AWS.
If anyone has any feedbacks, plz share them!

• sample built by Terraform
  https://github.com/t-kikuc/pipecd-iac/tree/main/piped-on-apprunner

• explanation of it (in Japanese)
  https://t-kikuc.hatenablog.com/entry/2024/02/22/193046