I feel like I was lied to about what kubernetes is and comics like this sorta partake in it.
The sales pitch is basically akin to "mongo is webscale" but "kubernetes is scaling/portability/reproducability."
Maybe my workplace just did it wrong, but the way they've set up kubernetes is exponentially more work than just scaling up (akin to an AWS autoscaling group). It has its own networking system, competing languages for manifests, competing auth mechanisms, multiple ingress/routing mechanisms, different CI/CD tools, helm/kustomize, multiple different competing scaling mechanisms that can sometimes be used together. And don't get me started on CRDS and such.
In some ways it's closer to build your own AWS than it is just autoscaling. Which sort of doesn't make as much sense to me if your company is full of 10-year experts on AWS and you're suddenly reinventing the wheel on all of this.
Which isn't to say it's strictly bad, but when I see what my work team has built it's very clear that local testing is well out of the realm of possibility and their work has already lead to multiple outages. And most of the administration isn't "clickops" but IMO it's even worse -- people typing admin commands at their console with no review -- type-ops.
It wasn't the UI that made click-ops bad, it was the fact that it was a bunch of manual admins changes not using PRs and automation.
Kubernetes is very much a choose your own adventure, and it’s very easy for your infra people to choose an adventure through Dante’s 12 levels of SaaS hell.
It doesn’t help that the popular tools like Helm or Kustomize are still so over-general and YAML-happy that they only seem to add complexity.
If I was going to brave these waters again, I would probably write my own tooling / templates with very narrow customization per “service”, and all the YAML generation would be done from TypeScript or similar function calls & composition, or at worst Terraform. The Kustomize or Helm stuff that has YAML tempting YAML… painful.
If you want kubernetes just for autoscaling, you can just use deployment with autoscaling. It's probably few 10s of lines to do this. And 10 lines for service/ingress.
In fact the complain that you need to learn helm or networking or auth or CRDS is unfair. If you don't want all the goodness of helm you can do it the old fashioned way, ie creating your own docker container and using deployment.
I understand your point -- is my complaint about kubernetes itself or just the way that people at my job set it up?
But in practice perhaps my feeling is that if a tool/ecosystem has too many viable/competing ways to do things and/or it makes adding complexity too easy, the it does encourage devs to get way out their depth. That's less of a problem if it's some tiny service only they have to deal with, but a threat to the whole company if they're using a multimillion dollar company's uptime as their little playground.
That may be less of a criticism of kubernetes itself and more of the ecosystem/trend, but I still think it's relevant. I'd compare it to this - I have often found frontend intolerable not because the tools are bad, but because the rate-of-change of the "in" tools is so excessive.
I know, it's supposed to spark excitement and make me long for Kubernetes, but… all I'm feeling is dread.
When our company was still small, I decided to go with Docker Swarm mode on Hetzner vServers to avoid the complexity of Kubernetes, until we've grown bigger™. Swarm mode got kinda-sorta-canceled, but by now I've built so much automation and architecture around it, it's working pretty stable, is super cheap to run, and doesn't involve too much maintenance.
Then again, demands are rising, and I'm starting to feel boxed-in by the hardware nodes the cluster runs on—I still need to account for the number of CPU cores to estimate the proper number of replicas for a worker container, for example. We also just got new funding, so this is probably the time I should finally be considering that Kubernetes migration, and do it properly once and for all.
But every time I reach this point, I start to dive into the documentation, the Big Cloud pricing charts, the upgrade procedures… I feel like I can avoid steering into the complexity maelstrom just a little longer.
> Then again, demands are rising, and I'm starting to feel boxed-in by the hardware nodes the cluster runs on—I still need to account for the number of CPU cores to estimate the proper number of replicas for a worker container, for example.
Have you switched to Hetzner bare metal? I'm pretty sure that'll raise your performance ceiling significantly.
> But every time I reach this point, I start to dive into the documentation, the Big Cloud pricing charts, the upgrade procedures… I feel like I can avoid steering into the complexity maelstrom just a little longer.
You can run k8s on Hetzner. Downside: Running k8s yourself. Upside: Not dealing with big cloud providers.
The analogy I came up with when I first learned about kubernetes in its early days was: "This is like a Bally's Total Fitness gym membership. Once signed up, you're locked in."
Companies are like bodybuilders:
- Many will try to look like Arnold Schwarzenegger (Mr. Universe), but few will succeed. - A significant number of others will focus on developing certain muscle groups more than others. The resulting appearance will look disproportionate or deformed. - The majority will just limp along, because we're paying for the membership and can't easily get out.
The key point is that it takes incredible discipline, knowledge, and resources (including money) to achieve the "nirvana" state, and then maintain it. And as a company ages, it will require even more effort to keep up that level of fitness and performance.
Just like a bodybuilder, achieving and sustaining peak condition is extremely challenging, even for the most dedicated and well-resourced companies. Most will fall short of the ideal, resulting in an unbalanced or suboptimal outcome.
Consulting companies are the real winners here. Kubernetes is their cash cow, as they can provide the expertise to help companies (try) achieve and maintain their "bodybuilder" status.
After seeing this comic, my v2 analogy especially after seeing the motorcycle slides is this:
If I ( === company) get a Yamaha Tenere 700 I'll be just like Pol Tarres!
- https://www.youtube.com/watch?v=eDxBGc9TD4c - https://www.youtube.com/watch?v=7PngVqi5wT8 - https://www.youtube.com/watch?v=3DgzzjiFn7g
Guessing the individual riding the motorcycle is the IT director ... to me this depiction screams "motorcycle squid".
For the record, I am not against using Kubernetes, but I highly question the decisions made to use Kubernetes. Majority of the time I get the sense someone "drank the koolaid" vs a techincal reason. The analogy is meant to highlight the significant challenges, resources and discipline required to truly master and maintain such a complex system.
This was incredible. Much better than the giraffe comic from forever ago, which I also liked!
- [deleted]
- [deleted]