otaru

Over-Engineering at Its Finest.

Bare-Metal Home Lab for Kubernetes and Technical Playground.

• otaru
  • Architecture
  • Hardware
  • Cluster Components
  • IaaS, PaaS, and SaaS
  • Bootstrap Cluster
  • Oopsy
  • Repository Configuration

Architecture

Hardware

ID	Device	HAT	Role	/dev/mmcblk0	/dev/nvme0n1
raspberrypi-00	Raspberry Pi 4 Model B 8GB	Waveshare PoE HAT (B)	Master	SanDisk Extreme 32 GB	-
raspberrypi-01	Raspberry Pi 4 Model B 8GB	Waveshare PoE HAT (B)	Worker	SanDisk Extreme 32 GB	-
raspberrypi-02	Raspberry Pi 4 Model B 8GB	Waveshare PoE HAT (B)	Worker	SanDisk Extreme 32 GB	-
raspberrypi-03	Raspberry Pi 5 8GB	Raspberry Pi Active Cooler + Pineberry Pi HatDrive! Bottom	Worker	SanDisk Extreme 32 GB	Samsung 980 PRO NVMe™ M.2 SSD 2TB (MZ-V8P2T0BW)

Cluster Components

Category	Name	Remarks
Application	AdGuard Home	Ad and tracker-blocking DNS server
Application	CyberChef	The Cyber Swiss Army Knife by GCHQ
Application	Home Assistant	Home Automation
Application	Jellyfin	Home Media System
Application	Repave	Daily restart of workloads within the cluster
Application	SFTPGo	SFTP for Jellyfin
Application	冗PowerBot	Telegram bot tracks and counts individual message counts in groups.
CI/CD	Argo CD	GitOps, drift detection, and reconciliation
Connectivity	Cloudflare Tunnel	Cloudflare Zero Trust Edge
Connectivity	Istio	Inbound North-South and East-West traffic with mTLS
Connectivity	MetalLB	Internal bare-metal network load-balancer with L2 operating mode
Connectivity	httpbin	Generic health check service
Monitoring	Kiali	Monitor Istio Network; Read-Only
Scheduling	KEDA	Event Driven Autoscaler
Scheduling	descheduler	Evicts pods for optimal cluster node utilisation
Security	1Password Connect	Proxy service for 1Password; acts as a secret provider
Security	External Secrets Operator	Extracts secrets from a secret provider
Security	cert-manager	Manages TLS certificates via Let's Encrypt and ACME protocol
Storage	Longhorn	Distributed block storage system; backup and restore from/to remote destinations

IaaS, PaaS, and SaaS

Category	Name	Service	Remarks
CI/CD	Github	Actions	Run Terragrunt
Connectivity	Cloudflare	Access	Edge Access Control
Connectivity	Cloudflare	DNS	Authoritative DNS Service
Connectivity	Cloudflare	Tunnel	Edge Connectivity
Connectivity	Cloudflare	WARP	VPN to Internal Network
Monitoring	Healthchecks.io	Healthchecks.io	Health Check - Heartbeat
Monitoring	UptimeRobot	UptimeRobot	Health Check
Security	1Password	Connect	Secrets Automation
Security	Let's Encrypt	Let's Encrypt	Certificate Authority
Storage	AWS	S3	Terraform Remote State
Storage	Backblaze	B2	Volume Backup

Bootstrap Cluster

1. Install Tooling

   brew install ansible go-jsonnet helm kubectl terraform terragrunt

2. Add SSH Keys to known_hosts

   for i in {00..03}; do ssh-keygen -R "raspberrypi-$i.local"; done && for i in {00..03}; do ssh-keyscan "raspberrypi-$i.local" >> ~/.ssh/known_hosts; done

3. Set Up 1Password Credentials
   • Follow the 1Password Connect Doc to create 1password-credentials.json.
   • Save the access token to the file token.

     ❯ tree $(pwd) -L 1
     /path/to/project/otaru
     ├── 1password-credentials.json
     ├── 1password-credentials.json.sample
     ├── ...
     ├── token
     └── token.sample

4. Bootstrap Cluster

   make main

5. Update AdGuard Home Password
   • Update the password in the ConfigMap.

Oopsy

make maintenance

make upgrade-cluster

make nuke-cluster

make rebuild-cluster

make restart-all

Repository Configuration

Secrets for GitHub Actions

Key
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
B2_APPLICATION_KEY
B2_APPLICATION_KEY_ID
CLOUDFLARE_ACCOUNT_ID
CLOUDFLARE_API_TOKEN
CLOUDFLARE_TUNNEL_SECRET
CLOUDFLARE_ZONE
CLOUDFLARE_ZONE_ID
CLOUDFLARE_ZONE_SUBDOMAIN
CLOUDFLARE_ZONE_TUNNEL_IP_LIST
GH_ADD_COMMENT_TOKEN
GH_DELETE_UNTAGGED_IMAGES_TOKEN
UPTIME_ROBOT_API_KEY