vmMemoryOverheadPercent design is causing over provision

[bnetzi]

Description

I created a new issue instead of Re-opening
#6611
As I don't have a permission to re-open it.

The reason I want to re-open it is documented in the comments there but I will submit it here as well:

I think the fact that karpenter is reducing 7.25% from the allocatable machine memory is a bug or at least a huge design flaw. The fact that karpenter can't know upfront the allocatable memory does not mean it should assume a constant overhead for all instances types and for all node pools. Just like it does not do that for the CPU.
It should be at least be a factor that depends on instance size - 7% overhead of the memory is acceptable for small instances, but for large instances it creates a huge overhead.
In my opinion the bare minimum would be to allow this to be configured per node pool.
Node overlay seems like an ok solution too but it would be harder to configure. Node pool seems like a better place (although I do get why it is harder to implement it there).

Also - I still don't understand, why does karpenter needs to know upfront? it can launch an instance, check if it fits, and if not - kill it and launch another instance. Another option is to cache the allocatable memory for each instance yype from each AMI used.

The bottom line is: In our env it created a 34% overhead. There is no way this should be acceptable as a default behavior, and I don't think most of the users understand the implications of this setting as it is really hard to measure over-provision, especially at scale.

[Balraj06]

@bnetzi Setting vmMemoryOverheadPercent to zero can solve the issue of overestimating the overhead right? Did you give it a try?

[bnetzi]

@Balraj06 Hi, for sure tried and it helps for the specific use case I presented.
Having said that - it is still a bug due the following reasons:

1. It is a deployment level variable, and with a mix environment with a variety of nodePools, pods and instance sizes - too large value is causing a waste of memory that we are not allocating / over-provision. A small value can cause pods that are pending forever as karpenter miss calculating the allocatable memory.
   My whole point is that it should be controlled per node size or in nodePool or per nodeClass (or with the future feature of node overlay as suggested in the comments of my previous issue)

2. This behavior, although might be controlled to some degree, is not well documented and probably most of the people that are using large instance (Ones with more than 100G memory) - have over allocation and they don't even realize it as it is hard thing to measure. Only advanced metrics are showing it or a deep dive into the logs, events, pod and node specs.

[Balraj06]

@bnetzi Thank you for the brief explanation. Can you send me the solution of node overlay?

[bnetzi]

@Balraj06 Sure kubernetes-sigs/karpenter#1305

[jukie]

I'd prefer something like VM_MEMORY_OVERHEAD_MB that supported setting a specific value rather than percentage.

[bnetzi]

@jukie good point

[njtran]

Another option is to cache the allocatable memory for each instance type from each AMI used.

The solutions for this problem are covered by #5161. Is it possible to discuss this there?