Building AWS at Home
A 3-year journey through distributed systems, from Step Functions to LLM inference.
There's a difference between knowing how systems work in theory and building them in production. I've spent the last 3 years learning this difference the hard way—first at AWS, then at RBC, and now at home. This is the story of how I learned distributed systems by owning every layer: from workflow orchestration to hardware networking, from GitOps to AI agents.
Homelab: Building AWS from Scratch
May 2025 – Present
The question: How does LLM serving work at scale? The only way to answer that was to build an entire cloud-like platform with SaaS fundamentals from scratch. 4 bare-metal machines, 1 GPU node, 1 Dell PowerEdge (now in the garage due to noise), and 2 mini-desktops. All running Talos Linux, a Kubernetes-native OS designed for immutability.
Infrastructure Layer
4-node bare-metal cluster (3 control plane + 1 worker) running Talos Linux—immutable, API-driven OS designed for Kubernetes. All wired ethernet to avoid etcd consensus issues.
- • Compute: Talos Linux nodes, machine config via Terraform
- • Networking: Cilium CNI, nginx ingress, Cloudflare Tunnel for zero-trust external access
- • Storage: Longhorn for distributed block storage with disk tagging, MinIO for S3
- • GitOps: ArgoCD with multi-source Applications and sync waves
Talos Cluster Topology
Control planes, worker GPU, storage, and networking architecture
GitOps Evolution
Started with Terraform for everything—but constant reconciliation of pods and configmaps created chaos. Brought in ArgoCD and established a clear split:
- • Terraform: Talos machine config (barely changes, no drift)
- • ArgoCD: CRD-driven observer pattern for K8s resources (changes sync automatically)
- • Design principle: Infrastructure should be easy for AI to modify. Declarative config + clear abstractions = easier for models to reason about
GitOps Deployment Flow
ArgoCD for Kubernetes and Terraform for Talos infrastructure
AI/ML Platform
Self-hosted LLM inference with multi-tenant GPU scheduling and 60% latency reduction.
- • Inference: vLLM serving Qwen3-32B with INT4 quantization
- • Multi-model: Ollama for smaller models, hot-swapping
- • Embeddings: TEI for text embeddings and reranking
- • Orchestration: KServe + custom Go API gateway
- • Scheduling: GPU node affinity with sm70/Volta constraints
LLM Inference Architecture
Gateway routing to vLLM, Ollama, and TEI on GPU worker node
Request Lifecycle
Client → nginx → gateway → predictor → streaming response
Data Platform
- • PostgreSQL: CloudNativePG operator with HA, automated failover
- • Vector DB: pgvector extension for AI embeddings
- • Streaming: Kafka/Redpanda for event-driven architecture
- • Workflows: Temporal for durable, long-running processes
Identity & Security
- • SSO: Authentik OIDC provider with custom claims
- • RBAC: Kubernetes RBAC synced with Authentik groups
- • Secrets: SOPS-encrypted secrets in git, decrypted at deploy time
- • TLS: cert-manager with DNS-01 ACME via Cloudflare
- • Powerline adapters killed etcd consensus at 200ms latency. Ethernet to garage solved it.
- • Network is foundational—hardware matters more than code sometimes.
- • Design infrastructure so AI can modify it easily. Declarative + immutable = no surprises.
Poimen: The AI Agent System
Building
Current LLM architecture works seamlessly with lambda/serverless patterns. The key insight: if AI is powerful because of context, tool-calling, and memory, what if we provide the right context in the right environment?
- •poimen-memory: Graph-RAG with wiki-link indexing, 7B model for instruct QA, pgvector + OpenSearch hybrid search
- •poimen-workflows: Natural language → executable WorkflowSpec via reasoning model, Temporal for durable execution
- •Activity Knowledge Base: Informs LLM about timeouts, retry policies, dependencies before routing
- •Generic state machine: JSON workflow spec + JSONPath parameter chaining. Temporal enables redrive + free self-hosted
Build a skill factory where a model generates workflows and completes them. Each Temporal activity is independently refined—enabling a general-purpose workflow orchestrator that improves through iteration.