Kube-Demo, Covid-19, Docker and Kubernetes

Kube-Demo

BOINC

When looking for a demo workload to use in demonstrating Kubernetes on this low budget cluster, I decided that I would at least try to lend a hand where I can to help with the pandemic. I decided to use 2 BOINC projects Rosetta@Home and OpenPandemics. Both of these projects are working to help scientists to find a cure or vaccines. Both of these projects use a system known as BOINC(
Berkeley Open Infrastructure for Network Computing). BOINC allows volunteers to run distributed computations for projects in a sandboxed environment. It exists for Windows, Linux, macOS, and Android. The Boinc people have an official docker container image on the docker hub. I picked this as an easy way to demonstrate manifests and container deployment on Kubernetes.

DOCKER

Having an official image to start with definitely made it easier to get the effort going. On docker hub the BOINC page lists a number of containers. It appears that most of the tags relate to GPU optimization in each container. Since the VMs that the Kubernetes cluster is running on have no GPUs, I chose the "latest" tag which is the same as the "baseimage-ubuntu" according to the digest. It also appears that there are container images for 32 and 64-bit arm platforms.

KUBERNETES

With the official docker hub container to start with, it was a fairly simple matter to come up with a manifest to deploy the container to the cluster. One of the problems with the container label is that Kubernetes uses the label to decide if the container in the repo has changed. Since the label used by the container creators is "latest", that means that the container from Kubernetes' point of view is ALWAYS current. A workaround to this is to do one of two things (from the Kubernetes configuration best practices doc):
imagePullPolicy: Always: every time the kubelet launches a container, the kubelet queries the container image registry to resolve the name to an image digest. If the kubelet has a container image with that exact digest cached locally, the kubelet uses its cached image; otherwise, the kubelet downloads (pulls) the image with the resolved digest, and uses that image to launch the container.
imagePullPolicy is omitted and either the image tag is :latest or it is omitted: Always is applied.
I chose to use "imagePullPolicy: Always" since I believe listing it explicitly makes it easier to understand.


Now one approach is to just issue patches against the manifest every so often to make Kubernetes go check for itself. Somehow that felt like using a sledgehammer to drive a tack. I then discovered that there was no real good way to check the hash on the docker repo without having Docker installed locally. Since the machine I was using only had 2G ram total to manage the OpenNebula cluster and run the CD stuff, I felt that installing Docker simply to check a hash value was also overkill. So I created my first opensource project in GitHub: docker-repo-info. Docker-repo-info uses the Docker repo API to fetch metadata about repositories and tags. One of the programs in the package "etagcli" simply takes the repo name, container and tag and returns the hash associated with that container. This can easily be put in a bash script to trigger and rolling upgrade using kubectl.


If you are interested in the nitty-gritty of how this was implemented check out these pages:

Live Pod Deployments
Poor Mans Continuous Deployment

Sharing this Blog

I've made the decision to start sharing this blog even though it's not anywhere near done.   I sometimes fall in the trap of continuing to work on a project like this until I feel it's perfect before I share it and then I never share it because there is always "one more thing to finish".   Future pages and posts are definitely going to cover creating Kubernetes deployments, deployment telemetry,  port forwarding for troubleshooting deployments, using Visual Studio Code as a programmers interface to the cluster,  taints, node selectors, anti affinities.   Other possible pages and posts might be using Terraform to drive OpenNebula and Kubernetes, running Kubernetes under LXD.  I'm open to suggestions on what the next posts should be about.

Using USB ethernets in this project

I know every system and network purist is going to say "WHAT DO YOU THINK YOU ARE DOING!".  Even I was asking myself that question.   It started as just an easy simple way to attach the storage server to the management network.  Then the coronavirus shows up and most retail and not a few online suppliers close.  So I'm faced with an architecture that requires 2 NICs per node and 3 machines with 1 builtin NIC each.  I consider trying to move everything to a single connection use VLAN tags, but it complicates the configuration a good bit and doesn't really mimic how it would be done in the real world.    I dig through my parts bin and come up with a handful of USB/ethernet NICs.   So, I think, why not give it a try, you don't have anything else to use.  As it turns out, so far they have held up to the demand just fine.    I ran a number of tests on them and they have performed flawlessly.  Now the fact that they will never be asked to go beyond 100Mb/s may have a lot to do with that.  However, this is just a simple test lab for learning and isn't going to be running an enterprise client VM load, it appears I have found a way to keep plugging forward with building this demo.  As a note, all of the USB ethernet adapters I have coincidentally wound up being ASIX chipsets.

updates on this project

Unfortunately, the COVID shelter in place order has kind of thrown a few wrenches into this project.  The group that I was buying my used gear from had to close their doors for an undetermined period.  The group that I was working with is https://www.freeitathens.org/.  They are a great group of people doing a good job providing technology to those who can't afford it.   Once this COVID stuff is over, please check them out.  Also, if you happen to have any used gear you are looking to get rid of and are anywhere near Athens GA, please consider donating it to this group.

However, I have plugged on with this project.  I was able to borrow a pair of machines from a friend.  They are equipped with phenom chips.  Given the command set on the chips, they should work fine as virtualization engines.  However, my first stab at building a private cloud was via Ovirt and it turns out that it doesn't recognize the Phenom chips as supported.  I tried several ways to work around this, but couldn't find a simple way around it.   So my next stab for building a private cloud will be OpenNebula

Reason for this blog

I've been working for several years with containers and Kubernetes, and I've been working with hypervisors/virtual machines for at least a decade.  Most commercial production work has been implemented in the Google Cloud, Digital Ocean, AWS, and friends.   However, in the interest of saving on the commercial cloud costs(I was working for a startup),  I also built a private cloud out of older machines that had been hanging around the organization.  This cloud wound up doing a lot of the long term code builds, testing, simulations and things that didn't require the millisecond response time the outward-facing products needed.   So this blog is documenting my process of building a working Kubernetes cluster out of cheap surplus hardware.  Hopefully, this will allow hobbyists, small schools, clubs, and other groups to get a good working knowledge of Kubernetes clusters without having to take out a second mortgage 😊.  Let's get started with the first page showing the initial design of the private cloud.