It is also common for filesystems to reserve a small percentage to the root user. I think the ext4 default is still 5% (which can be quite a bit more than 1GB on modern drives!)

We used to refer to this as a "cork file" back in the day.

> a third world slum situation when it happens

> why would operating systems allow themselves to run out of headroom entirely in this day and age?

Following the analogy, perhaps for the same reason the richest cities in the richest, most developed Western nations, are rapidly beginning to look very much like third-world slums?

Over-optimization is the name of the game. All systems need some amount of slack in them to stay flexible, robust (and livable). Unfortunately, cutting into the slack is always profitable on the margin, so without top-down intervention, the slack will be cut into until it's gone entirely. "Look, these machines are utilized only to 75% of capacity; adding this new system will only increase it by 5%"; "look, we can save XX$/month by cutting on compute, the machines will still be maxing out at 90%, so we'll still have a buffer". "Oh, this new telemetry service will bump that only by 1%".

"Look, there's so much free space here in between these blocks of flats; adding another block won't hurt."

And so on. Until your machines are running at 99% capacity and you risk global outage every time someone sneezes near the server room. Until your city starts to look like London, and if you're from Central Europe like me, you may start to realize that being a few months or years behind on the most recent gadgets is small price to pay in exchange for cities that are affordable and clean.

They usually don’t? If they did, they would have just crashed. If they are still running, then they allowed themselves headroom. The real question is who gets to use that headroom and how? What happens when the administrative processes use “too many” resources?

There are tons of tools built into modern OSes to manage prioritization of processes, etc. Whether it’s worth the tradeoff for you to deal with the operational overhead of harnessing those features is up to you.

Because it makes computer slower and people pay for compute.

I think consumer systems do it. macos shows the "quit some app, you're out of ram" but the system itself works.

but if you are asking that OS knows this is containers process and this is control plane process and treat them differently, I think no one does that.

Imagine how frustrating would it be if your os does it wrongly and you end up paying 10x because the os nerfed your main process because it thought something more important was running

Yep - my understanding of https://github.com/kubernetes/kubeadm/blob/main/docs/ha-cons... is that Kubernetes doesn't usually control that load balancer (nor should it, since you could accidentally tell Kubernetes to take down the control plane LB, then not be able to get it back up again!).

I suppose one could set up a "fast pass lane" kubeconfig that adds a header that haproxy would understand, and route to a priority class in its queue with e.g. https://www.haproxy.com/documentation/haproxy-configuration-... . But there's no easy `kubectl --with-priority` (or, to my knowledge, good guidelines for the various gitops solutions) that follows this pattern out of the box.

There’s a whole set of features built[0] to prevent this exact scenario - runaway controllers consuming all api resources. Not sure why it didn’t work, maybe they are running an old release

[0] - https://kubernetes.io/docs/concepts/cluster-administration/f...

Based on my understanding of the plugin code it is the default. The way cache.Indexer works is it's continuously streaming resources from APIServer using Watch API and updates internal map. I think if Watch API is down it just sits there and doesn't purge anything but I haven't tested that. The 20 min expiry is probably referring to CodeDNS cache stanza which is a separate plugin[0].

[0] - https://coredns.io/plugins/cache

They meant manuscript I assume.

This was most definitely not my choice or preference, but you know how it goes.

Excellent article, but the typesetting / justification in that pdf is horrendous.

The text columns look like the side of that hallway rubber mat that my dog keeps chewing on.

  spend a lot of time trying
  edge. However, as someone
  lieve that true progress is
  mes, and for the chickens
  y zombies, and the polite
  to eat your brain to acquire
  be prepared; thus, in the
  e scientific breakthroughs,
  ast inevitably becomes
  he main thing that I ponder is
  post-apocalyptic survival
  ag-tag group of associates.
  cruit: a locksmith (to open
  ith has run out of ideas);
  row snakes at my enemies
  g is a reasonable way to
  ble in my ultimate success

That was an amazing read. Thanks for linking it.

So sad that the account isn’t posting anymore.

Not if you connect to all the nodes