Account for additional costs per node

[harryw]

Description

What problem are you trying to solve?

I'd like to account for the costs of vendor-provided software and services that are licensed per node in our EC2 instance selection. I have not been able to find a way to configure Karpenter to include an additional cost to each node when evaluating the most cost-efficient set of nodes to use.

This seems quite similar to #4697 in that it brings in additional cost information into the node selection that is not necessarily available without more configuration.

For example, I might have a contract with some vendor that says they will provide support for my K8s cluster if I pay them $200 per node per month. My workload might run fine on 3 m6a.2xlarge instances (~$750 per month + $600 support), but a more cost-efficient option is 1 m6a.8xlarge (~$1000 + $200).

How important is this feature to you?

Pretty important. Without some way to build this cost into Karpenter's decision making, we'll end up having to be much more specific about the EC2 instance types we use to try to optimize for license costs, and we'll probably end up with much more need for review and churn in the Karpenter config. At the same time we'll have less resilience due to reduced flexibility - if those smaller instance types are all that's available then we'd prefer to use them than nothing.

I also, personally, think that this is the right choice from a design perspective. It's great that Karpenter can do so much to evaluate cost 'out of the box' by using the AWS APIs, but there will inevitably be costs and benefits to the choices it makes that are external to AWS - or any other cloud provider that Karpenter might eventually work with. It seems right to allow knowledgable users to provide that context through configuration.

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[ellistarn]

Somewhat related to #5050

[zmpeg]

We have per-node licensing costs and this would be an amazing feature to have.

[wmgroot]

I'd need to look more at the code to be sure, but this might require support upstream of Karpenter in the EC2 Fleet API.

[jonathan-innis]

Looks to be related to kubernetes-sigs/karpenter#1305. Also looks to be related to some thinking in issues like kubernetes-sigs/karpenter#1149 where there is a desire for Karpenter to be cost-aware when deciding to continue to schedule more pods onto a node or add a new node (though this is sort-of difficult, there may be some heuristics that we could introduce here)