Codecov Report

Merging #5029 (b594596) into release-1.20 (baade70) will increase coverage by 0.00%. The diff coverage is 100.00%.

@@              Coverage Diff              @@
##           release-1.20    #5029   +/-   ##
=============================================
  Coverage         41.21%   41.22%           
=============================================
  Files               117      117           
  Lines              9489     9490    +1     
=============================================
+ Hits               3911     3912    +1     
  Misses             5126     5126           
  Partials            452      452           

@dcbw: The following test failed, say /retest to rerun all failed tests:

Full PR test history. Your PR dashboard.

<details>

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. </details> <!-- test report -->

Problem

I'm running into the following problem when I run docker run --network=host --hostname=<hostname> --user <user> -v /etc/group:/etc/group:ro <image>.

docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: setup user: unable to find groups for spec kochiku-worker: bufio.Scanner: token too long: unknown.

My current finding shows me that this error is only thrown if a line in /etc/group exceeds 65536 characters.

Diagnosis

I'm not familiar with runc code or code in general, but i tried my best to find what's going on. I was able to find where this error message is thrown unable to find groups for spec in https://github.com/opencontainers/runc/blob/48d76adf7b8d074296edbc8c4475ebede1062116/libcontainer/user/user.go#L359

And from there, I find that the error likely came from running ParseGroupFilter function in https://github.com/opencontainers/runc/blob/48d76adf7b8d074296edbc8c4475ebede1062116/libcontainer/user/user.go#L339

Reading ParseGroupFilter function, https://github.com/opencontainers/runc/blob/48d76adf7b8d074296edbc8c4475ebede1062116/libcontainer/user/user.go#L178-L211, I suspect that we're hitting a token limit in bufio.NewScanner (which happens to be 65536 according to https://golang.org/pkg/bufio/#pkg-constants). This is likely given that another project https://github.com/sirupsen/logrus/issues/564 has hit a similar issue as well.

I encountered same issue, but solved by using WSL2 kernel package, you can have try: https://docs.microsoft.com/en-us/windows/wsl/install-win10#step-4---download-the-linux-kernel-update-package

We are triggering this error in 3.0.1 via:

$ podman generate kube c1d2e96b6fc0 -s -f podman-pod.yaml 
Error: unable to find user yelgeb: no matching entries in passwd file

@QiWang19: The following tests failed, say /retest to rerun all failed tests:

Full PR test history. Your PR dashboard.

<details>

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. </details> <!-- test report -->

@QiWang19: The following tests failed, say /retest to rerun all failed tests:

Full PR test history. Your PR dashboard.

<details>

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. </details> <!-- test report -->

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: <a href="https://github.com/cri-o/cri-o/pull/5029#" title="Author self-approved">dcbw</a> To complete the pull request process, please assign giuseppe after the PR has been reviewed. You can assign the PR to them by writing /assign @giuseppe in a comment when ready.

The full list of commands accepted by this bot can be found here.

<details open> Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":["giuseppe"]} -->

Ongoing sandbox requests cannot be (or are not) canceled by kubelet, leading to a situation where short-lived pods (especially Kubernetes e2e tests for stateful sets) cause overlapping sandbox requests. If the CNI plugin needs to wait for network state to converge, it's pointless to wait for a sandbox who's pod has been deleted so the plugin should cancel the request and return to the runtime. However, it's impossible to do that race-free without the pod UID the sandbox was created for, since the there is a gap between when kubelet requests the sandbox creation and when the plugin gets the pod object from the apiserver when the pod could have been deleted and recreated, and the CNI plugin would retrieve information for the new pod, not the pod the sandbox was created for.

Passing the pod UID to the plugin allows the plugin to cancel the operation when the pod UID retrieved from the apiserver during plugin operation does not match the one the sandbox was created for.

@trozet @haircommander @mrunalp

Cherry-pick of https://github.com/cri-o/cri-o/pull/5026

/kind feature

CNI plugins are now passed a K8S_POD_UID environment variable containing the pod UID this sandbox was started for.

Codecov Report

Merging #5028 (39efb80) into release-1.21 (d3e59a4) will increase coverage by 0.00%. The diff coverage is 100.00%.

:exclamation: Current head 39efb80 differs from pull request most recent head c512346. Consider uploading reports for the commit c512346 to get more accurate results

@@              Coverage Diff              @@
##           release-1.21    #5028   +/-   ##
=============================================
  Coverage         43.26%   43.27%           
=============================================
  Files               107      107           
  Lines              9839     9840    +1     
=============================================
+ Hits               4257     4258    +1     
  Misses             5128     5128           
  Partials            454      454           

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: <a href="https://github.com/cri-o/cri-o/pull/5028#" title="Author self-approved">dcbw</a> To complete the pull request process, please assign kolyshkin after the PR has been reviewed. You can assign the PR to them by writing /assign @kolyshkin in a comment when ready.

The full list of commands accepted by this bot can be found here.

<details open> Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment </details> <!-- META={"approvers":["kolyshkin"]} -->

Ongoing sandbox requests cannot be (or are not) canceled by kubelet, leading to a situation where short-lived pods (especially Kubernetes e2e tests for stateful sets) cause overlapping sandbox requests. If the CNI plugin needs to wait for network state to converge, it's pointless to wait for a sandbox who's pod has been deleted so the plugin should cancel the request and return to the runtime. However, it's impossible to do that race-free without the pod UID the sandbox was created for, since the there is a gap between when kubelet requests the sandbox creation and when the plugin gets the pod object from the apiserver when the pod could have been deleted and recreated, and the CNI plugin would retrieve information for the new pod, not the pod the sandbox was created for.

Passing the pod UID to the plugin allows the plugin to cancel the operation when the pod UID retrieved from the apiserver during plugin operation does not match the one the sandbox was created for.

@trozet @haircommander @mrunalp

/kind feature

CNI plugins are now passed a K8S_POD_UID environment variable containing the pod UID this sandbox was started for.

@dcbw: #5026 failed to apply on top of branch "release-1.21":

Applying: vendor: bump ocicni to 4ea5fb8752cfe
Using index info to reconstruct a base tree...
M	go.mod
M	go.sum
M	vendor/modules.txt
Falling back to patching base and 3-way merge...
Auto-merging vendor/modules.txt
Auto-merging go.sum
CONFLICT (content): Merge conflict in go.sum
Auto-merging go.mod
CONFLICT (content): Merge conflict in go.mod
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 vendor: bump ocicni to 4ea5fb8752cfe
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

<details>

In response to this:

/cherry-pick release-1.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. </details>

/cherry-pick release-1.21

A friendly reminder that this issue had no activity for 30 days.

@QiWang19: QiWang19 unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file.

<details>

In response to this:

/override ci/prow/e2e-gcp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. </details>

/override ci/prow/e2e-gcp

@dcbw: new pull request could not be created: failed to create pull request against cri-o/cri-o#release-1.22 from head openshift-cherrypick-robot:cherry-pick-5026-to-release-1.22: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"No commits between cri-o:release-1.22 and openshift-cherrypick-robot:cherry-pick-5026-to-release-1.22"}],"documentation_url":"https://docs.github.com/rest/reference/pulls#create-a-pull-request"}

<details>

In response to this:

/cherry-pick release-1.22 /cherry-pick release-1.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. </details>

@dependabot[bot]: The following tests failed, say /retest to rerun all failed tests:

Full PR test history. Your PR dashboard.

<details>

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. </details> <!-- test report -->

/cherry-pick release-1.22 /cherry-pick release-1.21

A friendly reminder that this issue had no activity for 30 days.

A friendly reminder that this issue had no activity for 30 days.

A friendly reminder that this issue had no activity for 30 days.

A friendly reminder that this issue had no activity for 30 days.

A friendly reminder that this issue had no activity for 30 days.

weird this sounds like a kubelet bug. I would expect cri-o to fail to create a duplicate pod while the first request is ongoing, and synchronously wait on cni plugin, thus preventing duplicate calls.

this change is fine to me, but I fear we're putting a bandaid on a bigger wound

@haircommander the scenarios are something like this:

Scenario 1: pod recreated during sandbox wait

1. pod created, kubelet notices, asks CRI to create the sandbox
2. CRI creates sandbox, execs CNI plugin
3. CNI plugin gets pod from apiserver, starts setting up networking
4. something deletes and recreates the pod
5. CNI plugin waiting for networking to converge

Now in this scenario, the plugin could create a pod watch for delete events. But that's not race-proof since the pod could be deleted + recreated between steps 3 and 4 and the sandbox would be for an old pod (and be subsequently torn down by kubelet at some point in the future). Pod UID allows the plugin to notice the pod instance it gets in (3) or (5) is different and exit early.

Scenario 2: pod deleted during sandbox init

1. pod created, kubelet notices, asks CRI to create the sandbox
2. CRI creates sandbox, execs CNI plugin
3. something deletes and recreates the pod
4. CNI plugin gets pod from apiserver, starts setting up networking
5. CNI plugin still waiting for networking

in this scenario, the CNI plugin gets the new pod instance, which is still wrong for this sandbox setup. Pod UUID immediately tells the plugin that its pod is gone and it can exit early.

In all cases, kubelet will just tear the sandbox down anyway, and what we're trying to prevent is waiting longer than necessary before noticing that this sandbox is useless.

I suppose the real fix would be to allow a sandbox delete + CNI DEL while an existing add was going on, or a CANCEL operation that kubelet could execute to tell the CRI and plugins to stop the request. But that's a much longer arc to make happen (but it should happen, and at least on the CNI side we are working on that via gRPC).

@dependabot[bot]: The following tests failed, say /retest to rerun all failed tests:

Full PR test history. Your PR dashboard.

<details>

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. </details> <!-- test report -->

/retest

Please review the full test history for this PR and help us cut down flakes.

/retest

Please review the full test history for this PR and help us cut down flakes.

@sshnaidm The HTTP status code is problematic here. If streaming straight from c/images (as currently written), we write to the client before knowing the final results.

@cdoern I do not like seeing all the status messages in one string. That is going to break clients who have built state machines around the expected output stanzas.

One implementation, could be to buffer the channel output from c/image in the handler and then present it as expected to the clients and with a better status code. That only burns memory in the service.

/cc @mheon