CDN-ON-A-DIME

Building a Kubernetes-Integrated CDN: A Personal POC Journey (AKA CDN-ON-A-DIME)

(See the bottom of this page for a diagram to follow along with)
(Also follow along with the githib repo at https://github.com/cstradtman/cdn-on-a-dime)

During my job hunt, I decided to challenge myself by building a proof-of-concept to see if I could extend Kubernetes with CDN-like functionality. This experiment wasn’t about lowering costs for enterprise-scale content delivery—it was a personal project aimed at honing my technical skills and exploring how Kubernetes can serve as the backbone for dynamic, streaming infrastructure.


Interestingly, as I focused on service meshes, I discovered that for my use case, the overhead of sidecars and the networking assumptions inherent in many service mesh implementations meant that a simple custom configuration endpoint—combined with the HTTP Traefik provider—was the most straightforward, least convoluted solution.


One of the challenges I faced was establishing a linkage between the external Traefik instances running in Podman and the external interfaces of the cluster itself. I evaluated several technologies, including Consul, etcd, Nomad, Istio, and Caddy. In the end, I wound up using one of the oldest and most reliable key-value stores in existence: DNS. 

Why I Built This

I’ve always been fascinated by the power of Kubernetes and its potential to orchestrate not just containerized applications, but also more complex systems—like a CDN. This project was my playground:

• Exploration: Experimenting with how Kubernetes objects (Deployments, Services, ConfigMaps) can be leveraged to build network functionality beyond the usual.
• Learning: Integrating various technologies such as Traefik for dynamic proxy configuration, externalDNS for extra cluster name resolution, and even ZeroTier for secure overlay networking.
• Showcase: Creating a technical portfolio piece to demonstrate my ability to design and implement non-trivial, integrated systems.

Key Components

Demo Workloads

• gstreamer and nginx (Video Streaming Demo): One of the core demo workloads is the video streaming setup. In this configuration, a gstreamer deployment streams video content. The deployment includes an init container that waits for the nginx service to be ready—verified via a TCP probe—before the main gstreamer container begins streaming video. The nginx deployment then acts as a content aggregator for the video stream.
• whoami (External Access Demo): To illustrate external accessibility, I deployed a lightweight whoami service. This service serves as a minimal example of how a containerized application can be exposed externally while remaining integrated within Kubernetes.

Dynamic Configuration and Proxy

Traefik and Dynamic Configuration: I developed a dynamic configuration service using a custom Python script (dynamic_config.py). This service monitors Kubernetes ConfigMaps and automatically updates Traefik’s routing rules through its HTTP provider. This real-time configuration capability is key for handling dynamic traffic patterns and making on-the-fly adjustments to the proxy layer.

Secure Overlay Networking

ZeroTier Overlay Network: To enable secure connectivity across different environments, each Traefik proxy was assigned a unique ZeroTier IP address. An upstream router running ZeroTier takes care of routing both the load balancer and node address ranges to these proxies, which are hosted in Vultr. This integration effectively embeds the overlay network into the overall CDN architecture.

External Infrastructure

ExternalDNS and MetalLB: While the demo workloads (gstreamer+nginx and whoami) showcase content delivery and external access, the entire system relies on robust external infrastructure. ExternalDNS automates DNS record management, and MetalLB handles pseudo-external IP assignments for all services (rfc1918 address that are routed via ZeroTier)—whether they are part of video streaming, dynamic configuration, or the whoami demo. These tools ensure that external access and routing remain consistent across every component of the system.

Architectural Snapshot

In my repository (https://github.com/cstradtman/cdn-on-a-dime), each component is organized into a logical section that mirrors the architecture. For example, you’ll find:

• Ansible Playbooks: To automate setup (e.g., configuring Traefik via Podman and joining nodes to ZeroTier).
• Kubernetes Manifests: For shared configurations (like ConfigMaps) and workload definitions (for gstreamer, nginx, and whoami deployments).
• Dynamic Configuration Service: A dedicated service that pulls from Kubernetes and updates Traefik when needed.

Here’s a glimpse of the directory structure for clarity:

Repository Structure

.
├── ansible/ # Ansible playbooks for automation and configuration
│ └── playbooks/
│ ├── configure_traefik_podman.yml
│ ├── configure_zerotier.yml
│ └── traefik.toml
├── docs/ # Documentation and architectural diagrams
│ ├── diagram.mermaid
│ └── diagram.png
├── dynamic_config/ # Dynamic configuration service for Traefik
│ ├── Dockerfile
│ ├── dynamic_config.py
│ └── k8s/ # Kubernetes manifests for traefik-config service
│ ├── rbac-traefik-config.yaml
│ ├── traefik-config-deployment.yaml
│ └── traefik-config-service.yaml
├── kubernetes/ # Shared Kubernetes manifests
│ └── configmaps_and_services/
│ ├── traefik-video-config-cm.yaml
│ └── traefik-whoami-config-cm.yaml
└── workloads/ # Workload definitions
├── gstreamer-service/ # Deployments for gstreamer and nginx with services
│ ├── gstreamer-deployment.yml
│ ├── nginx-deployment.yml
│ └── service.yml
└── whoami-service/ # Deployment and service for whoami
├── deployment.yml
└── service.yml

What I Learned

This project was as much about personal growth as it was about technical exploration. Here are a few takeaways:

• Integration Complexity: Extending Kubernetes beyond its common use cases revealed challenges in orchestration and dynamic configuration.
• Overlay Networking: ZeroTier added a robust layer of secure connectivity, but it demanded careful planning of routing and monitoring.
• Tool Synergy: Combining gstreamer, nginx, Traefik, and Kubernetes showcased how open-source tools can be re-purposed in creative ways to build custom systems.
• Shortcomings in Current Service Meshes: Although many service meshes attempt to offer “drop-in” functionality, the design of this project—especially the minimal Traefik proxy hosting—highlighted several limitations. Many of the meshes assumed full Kubernetes-to-Kubernetes connectivity. While I could have run K3s on the proxies, that approach introduced unnecessary overhead. Even in cases where the service meshes allowed for loading sidecars or service mesh proxies in Podman, the most common flaw was that they pointed the Podman endpoints to internal cluster addresses rather than to the overlay network–accessible addresses.

Next Steps

This POC was never meant to be a production platform, however, there are improvements that could be made to it. Maybe as future projects.

• HTTPS – Both the internal services backending the Traefik proxies and the external services provided by the Traefik proxies would benefit from being migrated to HTTPS rather than HTTP
• Authentication – Like HTTPS both the internal and external services would benefit from having an authentication layer made available.

Final Thoughts

Building this Kubernetes-integrated CDN was more than just a technical exercise—it was a journey of discovery, skill sharpening, and investigating what can be done with modern cloud-native tools. I hope my experiences inspire others to think outside the box and experiment with blending diverse technologies to solve unique problems.

If you have questions, ideas, or simply want to chat about awesome tech projects, feel free to drop a comment below!

CDN-ON-A-DIME architecture