Remove resource limits for the observability components #9785
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some time ago we decided to avoid using resource limits but somehow it was not
consistently implemented for these components.
The trigger for this change is a non self healing condition that we observed
recently: when the vertical pod autoscaling has an anomaly (not understood) and
it scales the resource requests of a pod to the lower bound recommendation
instead of the target recommendation (e.g. 1m core and 2.6MB for the blackbox
exporter), the limits are scaled proportionally (e.g. 13MB).
The problem is that such low memory limits do not manifest in OOM failures, but
the container itself can not be started in the first place because the `runc`
process is killed due to the out of memory condition on the cgroup. This
manifests in a RunContainerError in the pod status and can not self heal.
State: Waiting
Reason: RunContainerError
Last State: Terminated
Reason: StartError
Message: failed to create containerd task:
failed to create shim task: context canceled: unknown
With this change the limits are removed. Even if there is a momentary anomaly in
the vertical pod autoscaling, the pod will be able to start and the autoscaler
will correct the resource requests in the next iteration.
gardener-prow
bot
added
area/monitoring
gardener-prow
bot
added
the
size/S
|
/lgtm I checked in the prometheus-operator documentation that setting value 0 is what we want:
|
|
LGTM label has been added. Git tree hash: e8a629897bac70cbecb91f81db4e36bb83d0d899
|
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: rfranzke The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
gardener-prow
bot
added
the
approved
ialidzhikov
added a commit
to ialidzhikov/gardener
that referenced
this pull request
Jun 3, 2024
…ener#9785)" This reverts commit 0be0fdc.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
How to categorize this PR?
/area monitoring
/kind enhancement
What this PR does / why we need it:
Some time ago we decided to avoid using resource limits but somehow it was not consistently implemented for these components.
With this change the limits are removed for the observability components. Even if there is a momentary anomaly in the vertical pod autoscaling, the pod will be able to start and the autoscaler will correct the resource requests in the next iteration.
Which issue(s) this PR fixes:
The trigger for this change is a non self healing condition that we observed recently: when the vertical pod autoscaling has an anomaly (not understood) and it scales the resource requests of a pod to the lower bound recommendation instead of the target recommendation (e.g. 1m core and 2.6MB for the blackbox exporter), the limits are scaled proportionally (e.g. 13MB).
The problem is that such low memory limits do not manifest in OOM failures, but the container itself can not be started in the first place because the
runcprocess is killed due to the out of memory condition on the cgroup. This manifests in aRunContainerErrorin the pod status and can not self heal.Special notes for your reviewer:
cc @vicwicker @rickardsjp
fyi @dguendisch @ialidzhikov
Release note: