Blockers for v1.2.0

[AkihiroSuda]

Blockers for rc.1 (#3963):

• libcontainer: dmz: fix "go get" builds #4101
• libcontainer: remove all mount logic from nsexec #3985
• configs: validate: add validation for bind-mount fsflags #3990
• runc-dmz: Inheritable capabilities are dropped when they previously weren't #4125
• Fix runc-dmz error printing #4172
• never send procError after the socket closed #4173

Blockers for GA:

• Tests broken in debian due to PR: rootfs: remove --no-mount-fallback and finally fix MS_REMOUNT #4093
• Release runtime-spec v1.1.1 v1.2.0 Release v1.2.0 runtime-spec#1242
• runc doesn't work with go1.22  #4233
• Revert "Set temporary single CPU affinity before cgroup cpuset transition" #4283 (comment)
• Verify compatibility with well-known downstreams
  • Dockerfile: update to runc v1.2.0-rc.2 moby/moby#47666
    • Fix failing moby tests in CI Moby CI is failing due to stdio issue introduced in f2f16213 init: close internal fds before execve #4384
  • update runc (1.2.0-rc.3) containerd/nerdctl#3153
    • [v1.2 regression] SCMP_ACT_NOTIFY rule for fcntl causes runc to hang, before connecting to the seccomp listener agent #4328
    • [v1.2 regression] [cgroup v1 + rootless] nerdctl run -d --name=bar --pid=container:foo ; nerdctl rm -f bar hangs #4394
  • containerd works fine, CI scripts need fixes
    • PR to fix containerd's CI scripts: CI fixes for upcoming runc 1.2.0 containerd/containerd#10449
    • cri-tools PR backported for v1.30.1: [release-1.30] userns: Skip tests if the host doesn't support idmap mounts kubernetes-sigs/cri-tools#1492

Needs discussion (probably deferrable to v1.2.1 or v1.3.0):

• Issue joining cgroups cpuset with kernel scheduler task "random" distribution #3922

@opencontainers/runc-maintainers Feel free to edit this issue

[AkihiroSuda]

We don't have timeline for v1.2.0, but we should try to release v1.2.0 by the end of the year.

[kolyshkin]

Well, we have the 1.2.0 milestone (which I am trying comb through regularly), and everything that's in there is kind of a blocker for 1.2.0.

Feel free to modify milestones for any issues/PRs, but at the very least we have one more blocker:

• regression: can't kill and delete the container with shared(host) pid ns when the init process has dead #4047 (which I'm actively working one, hope to come to a closure this week)
  (and I guess @cyphar can add more)

[AkihiroSuda]

Well, we have the 1.2.0 milestone (which I am trying comb through regularly), and everything that's in there is kind of a blocker for 1.2.0.

Not everything.
At least this one can be postponed to v1.3 or later

• Add mon groups for resctrl. #2523

[kolyshkin]

Not everything.

Well, strictly speaking, everything that is under that milestone needs to either be included into 1.2.0, or be reassigned to a different milestone. Whis is why I said

Feel free to modify milestones for any issues/PRs

[cyphar]

I still think #3985 is needed for 1.2.0. There are several issues with mount propagation (not to mention the serious restriction of idmapped mounts) that are fixed with #3985. We also need #3990 if we want to make sure people scream at us during the -rc1 if it breaks something.

Other than those, I don't have any strong opinions on any remaining PRs.

[h-vetinari]

Blockers for GA: Release runtime-spec v1.1.1

Given that it took well over 3 years to get 1.1.0 finished (incl. half a year from rc1 to GA), is it really necessary (resp. judicious) to block runc 1.2 on a spec-release?

[AkihiroSuda]

Blockers for GA: Release runtime-spec v1.1.1

Given that it took well over 3 years to get 1.1.0 finished (incl. half a year from rc1 to GA), is it really necessary (resp. judicious) to block runc 1.2 on a spec-release?

Necessary, because we have been depending on its main branch:

• features: Expose idmap support #3993

[utam0k]

This task has been done 👍

Release runtime-spec v1.1.1 v1.2.0 opencontainers/runtime-spec#1242

[testinfected]

Hi guys,

Eagerly waiting for that 1.2 release with fixes on mount options. What do you think is a realistic release date?

Thanks

[testinfected]

Ping

[AkihiroSuda]

I think we are ready to release rc.2, and release GA after verifying compatibility with Docker, containerd, BuildKit, etc.

cc @opencontainers/runc-maintainers

[AkihiroSuda]

Turned out that we need to make a decision on this one too

• Revert "Set temporary single CPU affinity before cgroup cpuset transition" #4283 (comment)

[MaxXor]

Any ETA when 1.2.0 will be released?

[AkihiroSuda]

Any ETA when 1.2.0 will be released?

As soon as the compatibility issues get resolved

• Dockerfile: update to runc v1.2.0-rc.2 moby/moby#47666
• update runc (1.2.0-rc.3) containerd/nerdctl#3153

[h-vetinari]

#3922 is noted as a blocker here, and a recent comment notes

@kolyshkin: This is going to be implemented via opencontainers/runtime-spec#1253

Since opencontainers/runtime-spec@7017384 is not yet in a runtime-spec release, does that mean you'll need another spec release to unblock runc 1.2?

From the peanut gallery: Given that #3922 is a year old already (and the issue apparently exists back until runc 1.0.2 at least), I'm skeptical that this really has to block runc 1.2? After being 28 months behind the originally intended release date, perhaps it would be reasonable to stop the scope-creep and just focus on fixing any remaining hard regressions and then release?

[AkihiroSuda]

Let me move #3922 to the "probably deferrable to v1.2.1 or v1.3.0" list

[rata]

I've tested runc 1.2.0-rc.2 with containerd and opened a PR to fix the issues (some fixes belonged to the cri-tools repo, that are already merged and included in a patch release). All the fixes belong to the containerd/cri-tools repos, nothing to do here in runc: containerd/containerd#10449.

Can someone please take a look at the moby issue? moby/moby#47666

[kolyshkin]

Looking into moby/moby#47666, as well as other dmz issues, I'm thinking maybe we should remove it entirely from runc-1.2?

[rata]

@kolyshkin are you sure that is dmz related? That moby PR is compiling with nodmz, there might be something on the say we disable it, but it felt like another issue given that it fails in the same way with and without the nodmz buildtag

[kolyshkin]

@kolyshkin are you sure that is dmz related? That moby PR is compiling with nodmz, there might be something on the say we disable it, but it felt like another issue given that it fails in the same way with and without the nodmz buildtag

Yeah, probably not. I will take a look.

[rata]

It was indeed something on the way runc_nodmz is disabled, work-around here: #4345 (still need to fix cc_platform.mk).

I didn't manage to debug the failing tests now that docker compiles and need to leave now. Please someone else take a look :)

To use a patched runc, I used this moby PR to just compile my fork: moby/moby#48161. Compilation works fine here, but the test fail. You can use something like that to try fixing the tests on the CI.

[kolyshkin]

I figured out the runc compile issue in moby; see moby/moby#48160

[apostasie]

Hey folks.

Would it be possible to cut an RC3? Ideally that would allow closing containerd/nerdctl#3153

Thanks!

[AkihiroSuda]

If somebody can take a look at incompatibility issues with Moby (moby/moby#48336), that will be helpful for releasing the next RC (and GA)

[rata]

@AkihiroSuda I can't repro the issue locally (even without apparmor=0 in the kernel cmdline), I created a VM in Azure with a stock ubuntu (to see if there was something in that setup), but all test seem to pass just fine. I've tested CI without apparmor, etc, no differences (I'll update the moby PR with more details). @kolyshkin has been testing several things also and he can't repro either.

The issues in CI seem real to me, its' 100% reproducible that the new runc causes failures and the old doesn't. I guess there is something in the moby CI setup that we don't have in the other envs we are testing this.

@AkihiroSuda Can you have a look, please?
@thaJeztah can you have a look too, pretty please?

[AkihiroSuda]

f2f1621 init: close internal fds before execve seems closing stdout too earlier

• Moby CI is failing due to stdio issue introduced in f2f16213 init: close internal fds before execve #4384

[rata]

Okay, I think I understand now the issues with moby: #4384 (comment). They are on moby, though, and not runc. @thaJeztah it would be great if you can take a look though, as it is quite moby related.

IMHO we can tag a new rc now. @AkihiroSuda @kolyshkin @cyphar @lifubang @thaJeztah (or any other maintainer reading this) what do you think?

[lifubang]

IMHO we can tag a new rc now.

I agree, if there is no begin, I'll open the release PR this weekend.

[cyphar]

@lifubang I'll do the release this time.

[AkihiroSuda]

• update runc (1.2.0-rc.3) containerd/nerdctl#3153

Tried to run nerdctl tests with runc v1.2.0-rc.3, but something seems hanging on Ubuntu 20.04 (cgroup v1):

• [v1.2 regression] [cgroup v1 + rootless] nerdctl run -d --name=bar --pid=container:foo ; nerdctl rm -f bar hangs #4394

[lifubang]

#4408

[rata]

@lifubang thanks! IMHO, we can cut rc4 when @cyphar fixes that issue

[cyphar]

I'll send a patch for it later today. (This will be our first logging output in filepath-securejoin, I guess I'll just use logrus...)