Allow resource requests be set in build pipelines for individual steps, not just resource limits, steps could burst into the free capacity in the worker node
long-term
S
Shamrock green Planarian
Assigning resource limits to individual steps in a pipeline does not work well, because each container (harness step) finishes its task before the next container starts in a sequential setup. One can assign each container a resource limit, which is accounted for as the reserved (blocked) capacity of the worker node.
This issue forces us to choose larger worker nodes when it’s not really necessary. If we could set resource requests instead of resource limits, we could place the container on a node with enough capacity to perform the job, and allowing jobs to burst into the extra capacity of the worker. Since the previous steps have finished their activities, the requested resource limit is blocked and cannot be used for the following steps (containers) in the current implementation of the build pipeline.
Log In
N
Nofar Bluestein
Merged in a post:
Option to setup only Resource Requests in steps
M
Maximum Tapir
Currently the at the step level only the resource limits can be configured as per https://developer.harness.io/docs/continuous-integration/use-ci/set-up-build-infrastructure/resource-limits/#override-resource-limits . However, in cases where users don't want it to set the limit, just the request, there isn't any provision currently available.
N
Nofar Bluestein
M
Maximum Tapir
This image illustrates exactly what we're going through. To avoid the throttling we are having to set more CPU than the application actually need, causing us to spend more resources.
S
Shamrock green Planarian
you can only set resource limit for containers, not any other objects in Kubernetes
S
Shamrock green Planarian
The harness does not sum the resource limits, but Kubernetes does , each step + add-ons resources are containers in single pod, in Kubernetes, you can assign resource limits on container level, so in order for the pod to be scheduled to a worker node (kubelet) , the resource limits are summed up. that is why resource requests are more useful since you can ensure stability and use the free overhead. but when you set limit, your workload can not burst when it need to. this issue is very real when a monoliths or big application with a lot of tests cases need to bust on some of the test cases
N
Nofar Bluestein
long-term
N
Nofar Bluestein
Hey,
Harness has a mechanism to help us determine the resources needed for the pod. In case of 2 sequential steps, we the steps are calculated as ( maximum resources needed amongst sequential steps ) + ( add-on resources ). We don't sum the resources of the sequential steps, but ask for the minimum that will allow them to run sequentially.
Please learn more about how Harness calculate resource allocation here:
Thank you,
Nofar Bluestein,
CI product team
S
Shamrock green Planarian
I wish this was not changed to a long term effort as it is very simple change since the resource limit is already set in the configuration, so resource request is just adding another field. in big monolith application builds that requires a lot of resources, setting resource limit is not very good option to schedule the workloads