SELinux policy changes in Bottlerocket 1.3.0

[bcressey]

How is the SELinux policy changing?

In Bottlerocket 1.3.0, we will be enforcing Multi-Category Security (MCS) in the SELinux policy, which will add an additional layer of security between pods (Kubernetes) and tasks (Amazon ECS) on the same host.

What does this mean to me?

The majority of the use cases that work today will continue to work.

• Every Kubernetes pod gets an MCS pair assigned automatically, which allows all the containers in the pod to interact with any of the other processes and files in the pod.
• Each container in an Amazon ECS task gets a different MCS pair assigned automatically. If containers in the task need to interact with other processes or files in the same task, then the task PID mode can be set so they will share the same MCS pair.
• On Kubernetes, volume types like emptyDir and Amazon EBS volumes are automatically relabeled with the MCS pair when the pod is brought up, so containers in that pod can always read and write to them. Volume types like hostPath and Amazon EFS volumes are not relabeled, which allows reads and writes from all containers.
• On Amazon ECS, Docker volumes are automatically relabeled for shared usage by default, so any container can read and write to them. Amazon EFS volumes and bind mounts are not relabeled, which allows reads and writes from all containers.
• Privileged containers bypass all MCS-related restrictions.

What could go wrong?

In certain cases, unprivileged containers will be unable to access files or processes that belong to other containers.

• Unprivileged containers that interact with processes or files in another pod may encounter access denials. This could happen if the container shares the host PID namespace, such that it can see other processes, or if it has access to the entire host root filesystem, such that it could encounter files for other pods while traversing directories.
• Unprivileged containers that interact with processes in privileged containers may receive an access denial, even if the containers are running in the same pod. This could happen if the containers share a process namespace, and the unprivileged container runs a supervisor process that attempts to signal processes in the privileged container.
• Unprivileged containers with custom SELinux labels may not be able to interact with other processes or files in the same pod. Custom SELinux labels can be specified through the SELinuxOptions for a Kubernetes pod or container, or by setting the dockerSecurityOptions for an ECS task.

Can I turn it off?

No. Bottlerocket aims to constantly improve the security posture of the host OS, and it is important to ensure that unprivileged containers do not have access to unexpected resources.

Help! I found a problem!

All SELinux related access denials will be logged in dmesg and in the host’s journal. You can use these resources to verify an incompatibility with the new SELinux policy restrictions. In most cases, you can work around this by adding SELinux options to pod specs or task definitions.

If you continue to have difficulty, you can downgrade to the previous Bottlerocket version, or pin to a specific Bottlerocket version in your update settings. Please reach out through AWS Premium Support or open a GitHub issue for further guidance.

[anthr76]

I'm aware this will fix #1651 are there any AMIs published to test this build? Very exciting changes!

[Halama]

It would be great to able to test the image before the release!

[bcressey]

We don’t currently have a way for users to test official builds before release, but we have a couple things to suggest:

Once the 1.3.0 release is ready, it’ll be rolling out slowly, over the course of about two weeks. You could test early in this cycle by setting settings.updates.ignore-waves=true to let your hosts update immediately. See updates settings for details. (The best place to watch for the release is our releases page, and GitHub can send you notifications if you use the “watch” feature at the top of the page.)

All of the expected pull requests for the 1.3.0 have been merged, including #1733 that introduced the MCS changes, so you could also build your own test images. You can use them just like official ones, except that they won’t be able to upgrade to future official releases.

[Halama]

thanks, settings.updates.ignore-waves=true works great!

[joekir]

Is there ever an intention that pods should need to share resources using MCS? In my experience of how MCS is implemented that would be quite painful, as all categories need to be present to allow interaction.

Example with 2 pods:

1. all files in pod1 get assigned ...s0:c0
2. all files in pod2 get assigned ...s0:c1

Is there a scenario where pod1 would need to access a process/file/etc.. in pod2?
If so there's 2 approaches I can think of to achieve this:

1. Each pod (or process within a pod needing access to the shared resource) is relabeled so that they have MCS of ...s0:c0,c2 and ...s0:c1,c2 respectively and the shared file is then labeled as ...s0:c2 (as MCS doesn't support OR, only AND)
2. Whatever process in pod1 that needs access to the file in pod2 is relabeled as ...s0:c0,c1
3. (is there another approach I'm missing?)

Approach #1 is probably the only secure approach, but it seems like it could get complicated if sharing among more than 2 pods is needed, the options then would be sharing between all N pods, or needing a lock abstraction.

Approach #2 seems dangerous (or at least sub-optimal) as then the whole of pod2 is opened up to that process in pod1.

What's the missing piece to my mental model?

[bcressey]

Another approach would be to add another object type - custom_file_t - and rules that only permitted ...:s0:c0 or ...:s0:c1 to read and write to it. This is along the lines of what udica attempts to do with its dynamic policy generation. It's also essentially how access to shared volumes like host paths or NFS filesystems works today. These directories receive a different label that does not have MCS pairs associated with it, which all containers are allowed to write to.

Integrating this into the container execution workflow by default would seem to require some grouping above the pod level that represents another trust boundary. For example, we might say that all pods within a namespace are more trusted than other pods, but less trusted than containers within the pod. To restrict file access to pods in the namespace, there could be a new access mode with the desired semantics.

Then we could allocate a third category, add it as a prefix to all pods in that namespace (container_t:s0:c3,...), and use it to label volumes with the namespace access mode (namespace_file_t:s0:c3). The namespace categories would need to come out of a dedicated range but we could expand the category range from 1024 to 2048, and use the upper half for those categories without giving up any allocatable pairs for pods.

[joekir]

Another approach would be to add another object type - custom_file_t - and rules that only permitted ...:s0:c0 or ...:s0:c1 to read and write to it.

I didn't know you could combine categories in a .te policy file.
Quoting RedHat docs:

The MCS check is applied after normal Linux Discretionary Access Control (DAC) and Type Enforcement (TE) rules, so it can only further restrict security.

Do you have an example to link to? It sounds useful!

---

After some searching with your additional info, I think I've found some prior examples of this usage it seems sVirt (secure virtualization) is the canonical example https://danwalsh.livejournal.com/30565.html of using this, leveraging MCS for ~namespacing of type enforcement. Note though, no intent for sharing cross-category as that's the whole feature that it cannot share.

---

Now that I'm re-reading your section on What does this mean to me? it sounds like all those intended usage patterns may be condensed to:

1. By default, pods will not be able to access "files" of other pods, regardless of whether the SELinux type label is the same. This is due to MCS acting as a pod namespace
2. You can only share "files" across pods by either
   1. Relabeling the requestor's SELinux domain entrypoint label to match the owner's MCS pair
   2. Accessing from a privileged container
   3. Removing SELinux labeling from the "file" (e.g. EFS scenario)

Does that sound right?
It does feel like going further than that would add more complexity and hence more chance for availability edge cases.

[bcressey]

That all sounds right.

Dan Walsh ported his usage of MCS for sVirt to the go-selinux library used by Docker, and later by containerd's cri implementation, which is the container-level mechanics came into Bottlerocket.

The specific change in the Bottlerocket 1.3.0 release was adding the SELinux policy rules to make MCS categories meaningful; unlike type enforcement in a deny-by-default setup, MCS categories don't deny anything in the absence of rules.

[chlunde]

Is there a way to get the same automatic MCS labels for all pods in the same namespace? Sometimes they share the same volume, and we think that this is the root cause for permissions errors we see.

[bcressey]

Is there a way to get the same automatic MCS labels for all pods in the same namespace? Sometimes they share the same volume, and we think that this is the root cause for permissions errors we see.

You could do this with an admission controller of some sort to override & set the seLinuxOptions for all pods created in the namespace. However, I suspect there might be an implementation issue here, such as volumes that "should" be shared getting labeled as private, or not getting relabeled when reused on either the same node or a different one.

Can you open an issue with more details, such as the specific AVC denial from dmesg and the relevant CSI volume configuration? I'm happy to dig deeper.

[liam-mackie]

Though I hate necro-posting, this is the only place I've been able to see any details about the implementation that seems to be blocking me. Our current situation is that we have an NFS server container which uses an emptyDir to host temporary files used across multiple pods. Since we're not using nfs-ganesha we need to run the pod in privileged mode, since it's required for NFS.

This NFS container hosts temporary executables which are then used by pods created by a controller. It looks like the pods are unable to execute the files exposed via NFS due to the MCS labels being in-place:

Jul 24 07:13:50 ip-10-0-19-55.ap-southeast-2.compute.internal audit[10860]: AVC avc:  denied  { execute } for  pid=10860 comm="nfsd" name="bootstrapRunner" dev="nvme1n1p1" ino=17060223 scontext=system_u:system_r:kernel_t:s0 tcontext=system_u:object_r:data_t:s0:c61,c301 tclass=file permissive=0

I guess my questions are as follows:

• Is there a consistent MCS pair that pods use?
  • If there is, is the fix to simply label the exported directory in the NFS server pod?
• Is there a way for us to automatically allow access to the privileged container via a customised SELinux security context on that container?

I'm happy to provide logs or additional details if required, and appreciate any help you might be able to provide!

[liam-mackie]

After a bunch more investigation, I don't think this is due to MCS labelling. I've logged an issue here rather than trying to get support on this thread - but I hope that if anyone has issues similar to me the issue I logged will help them diagnose :)