Kubelet TCP/HTTP probes: improve network resources utilization #115143
+363
−2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
k8s-ci-robot
added
release-note
|
/assign @SergeyKanzhelev @thockin |
k8s-ci-robot
added
the
priority/important-soon
k8s-ci-robot
removed
the
needs-priority
|
/sig node |
k8s-ci-robot
added
area/kubelet
sig/node
aojea
commented
Jan 17, 2023
aojea
force-pushed
the
linger
branch
3 times, most recently
from
0363f7a
to
42d6a44
Compare
aojea
commented
Jan 18, 2023
| } | ||
| switch result.Result.String() { | ||
| case "Failure": | ||
| t.Errorf("Failure %s on contantinerID: %v Pod %v", probeType, result.ContainerID, result.PodUID) |
There was a problem hiding this comment.
the test fails on failed probes
There was a problem hiding this comment.
nit: Can you make it an explicit comment in code? It was hard to understand when the test will fail.
|
/retest |
probe util dial set linger to 1s
Change-Id: I323b472606eaf3242b665022afe2a79ecf3b8358
Change-Id: I1e49943531b569b5e02f82369750a9ca899ae726
k8s-ci-robot
removed
the
lgtm
|
added comment on the choice for linger socket time #115143 (comment) and it removed the lgtm, removing the hold and waiting for final lgmt @SergeyKanzhelev /hold cancel |
k8s-ci-robot
removed
the
do-not-merge/hold
k8s-ci-robot
added
the
lgtm
|
LGTM label has been added. Git tree hash: 630ad365977ca27c2e3b8a3704506211db2c9010
|
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: aojea, SergeyKanzhelev, thockin The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
This was referenced Jan 23, 2023
k8s-ci-robot
added a commit
that referenced
this pull request
Jan 30, 2023
…-of-#115143-upstream-release-1.23 Automated cherry pick of #115143: add custom dialer optimized for probes
k8s-ci-robot
added a commit
that referenced
this pull request
Jan 30, 2023
…-of-#115143-upstream-release-1.24 Automated cherry pick of #115143: add custom dialer optimized for probes
k8s-ci-robot
added a commit
that referenced
this pull request
Jan 30, 2023
…-of-#115143-upstream-release-1.25 Automated cherry pick of #115143: add custom dialer optimized for probes
k8s-ci-robot
added a commit
that referenced
this pull request
Jan 30, 2023
…-of-#115143-upstream-release-1.26 Automated cherry pick of #115143: add custom dialer optimized for probes
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
/kind bug
/kind cleanup
/kind feature
What this PR does / why we need it:
For every connection created, the OS opens a socket and, once it finish it doesn't free all the resources immediately, it goes through a TIME-WAIT state (60 seconds by default).
Kubelet probes create short lived connections (usually less than 1 seconds), each of these connection allocate a socket, ephemeral port, conntrack entry, ... and it takes 60 seconds (TIME-WAIT) state to be freed.
This causes a problem at scale on systems that are already using a considerable amount of network resources, when there are a considerable number of probes at a high rate, they will be consuming a considerable amount of them. Since Kubernetes is a distributed system, it relies heavily on the network, and is common to reach the limits of the network resources with the users workloads.
In addition, the failures caused by the exhaustion of this network's resources are very difficult to troubleshoot, since the symptoms vary from failed probes, to connectivity issues with Services and Ingresses, controllers failing to reach apiserver, ...
In order to be more efficient on the kubelet, we can use the option SO_LINGER for each of the network probes to set the TIME-WAIT to 1 second. This is not a big problem since the probes use the connection just to check the status of the Pod, and the data transferred is relatively small.
Fixes #89898