Table of contents of the article:
Kubernetes has presumably won the container wars. However, Kubernetes is still difficult and causes a lot of pain.
I think I should give a little preface to this article. Kubernetes is the new runtime for many applications, and when used correctly it can be a powerful tool for getting complexity out of your development lifecycle. However, in recent years I have seen many people and companies stumble upon the desire to manage their own installation. It often remains in the experimental phase and never goes into production.
How does Kubernetes work?
In broad terms, Kubernetes, or K8s, appears to be very simple. The nodes (machines) running Kubernetes are divided into (at least) two types: the master and the workers. The master(s), by default, doesn't run any actual workload; that's the work of the workers. The Kubernetes master includes a component called the API server, which provides an API you can communicate with using kubectl. It also includes a scheduler, which makes decisions about which containers run where (scheduling containers). The final component is the controller-manager, which is actually a set of multiple controllers responsible for managing node outages, managing replication, merging services and pods (collections of containers), and finally managing service accounts and API access tokens. All data is stored in etcd, which is a highly consistent key-value database (with some really cool features). So, to summarize, the master is responsible for managing the cluster. No wonder. The worker, on the other hand, manages the actual workloads. To do this, it again comprises a number of components. First, it runs the kubelet, which is again an API that works with containers on that node. There's also the kube-proxy, which forwards network connections, containerd for running containers, and, depending on the configuration, there may be other things like kube-dns or gVisor. You'll also need some sort of network overlay or integration with your underlying network configuration so that Kubernetes can manage the networking between your pods.
Kubernetes ready for production
This, so far, doesn't sound too bad. Install a couple of programs, configurations, certificates, etc. Don't get me wrong, it's still a learning curve, but it's nothing an average sysadmin hasn't dealt with in the past. However, the simple manual installation of Kubernetes isn't exactly production ready, so let's talk about the steps involved in getting this thing up and running. First, the installation.
You really want to have some sort of automated installation. It doesn't matter if it's Ansible, Terraform, or other tools, you want it to be automated. kops, for example, helps with this, but using kops means you don't know exactly how it's set up and can cause problems when you want to debug something later. This automation should be tested and tested regularly. Next, you need to monitor the Kubernetes installation. So right away you need something like Prometheus, Grafana, etc. Do you run it inside your Kubernetes? If your Kubernetes has a problem, is your monitoring interrupted? Or do you manage it separately? If so, then where do you manage it?
Also noteworthy are the backups.
What will you do if your master crashes, the data is unrecoverable, and you need to restore all pods on the system? Have you tested how long it takes to run all the jobs in your system again? Do you have a disaster recovery plan? Now, since we are talking about the CI system, we need to run a Docker registry for the images. This, of course, can be done again in Kubernetes, but if Kubernetes crashes. The CI system is obviously an issue too, as is the version control system. Ideally, isolated from your production environment so that if that system has a problem, at least you can access your git, redeploy, etc.
Data Storage
Let's talk about the elephant in the room: memorization. Kubernetes in and of itself does not provide a storage solution. Of course, it is possible to mount a folder from the host machine, but this is neither recommended nor simple. Rok, for example, makes it relatively easy to use Ceph as the underlying block memory for your data storage needs, but my experience with Ceph is that it has a lot of values and configurations that need tuning, so you're not out of the question at all. trouble simply advancing to the next step.
Debugging
When discussing Kubernetes with developers, a common pattern appeared fairly regularly: when using managed Kubernetes, people had trouble debugging their applications. Even simple issues, like a container not starting, caused confusion. This, of course, is an education issue. Over the past few decades, developers have learned to debug "classic" configurations—reading log files in /var/log, etc.—but with containers, we don't even know which server the container is running on, so it represents a paradigm shift.
The complexity problem
You may have noticed that I'm skipping over the things cloud providers provide, even if it's not fully managed Kubernetes. Of course, if you use a managed Kubernetes solution, that's great, and you won't need to worry about any of this, except debugging. Kubernetes has a lot of moving parts, and Kubernetes alone doesn't provide a complete solution stack. RedHat OpenShift, for example, does , but it costs money, and you still have to add things yourself. Right now, Kubernetes is on the cusp of Gartner's hype cycle; everyone wants it, but few truly understand it. In the coming years, many companies will have to realize that Kubernetes isn't the solution to all problems and figure out how to use it correctly and efficiently.
I think getting your Kubernetes running is only worth it if you can afford to dedicate an operations team to the issue of maintaining the underlying platform for your developers.
