Runtime fails to mount /sys when --tpuproxy is provided #10795
Comments
|
Hi Peyton, thanks for reporting this bug. The error you're seeing likely happening while building the internal gViosr sysfs, not because |
|
iirc, you can't run tpuproxy via the similar command which works for nvproxy because it has not implemented in tpuproxy, then tpu devices are not accessible in your docker container. |
|
@manninglucas sure thing here it is: @milantracy Does this mean we can't use |
|
afaik, I tried the raw |
|
@milantracy would you mind sharing the command you're using to start #/bin/bash
sudo runsc --tpuproxy --root=/home/peyton/tputesting/runroot do --root=/home/peyton/tputesting/jax-rootfs -- env -u LD_PRELOAD /bin/bashpeyton@t1v-n-901fc2b8-w-0:~/tputesting$ ./start.sh
starting container: starting root container: starting sandbox: failed to setupFS: mounting submounts: mount submount "/sys": failed to mount "/sys" (type: sysfs): no such file or directory, opts: &{{false false false false} false {true 0xc000620990} false}EDIT: I'm also getting the same behavior with peyton@t1v-n-901fc2b8-w-0:~/tputesting$ cat start.sh
#/bin/bash
sudo runsc --root=/home/peyton/tputesting/runroot --tpuproxy run --bundle=/home/peyton/tputesting my-jax-container
peyton@t1v-n-901fc2b8-w-0:~/tputesting$ ./start.sh
running container: starting container: starting root container: starting sandbox: failed to setupFS: mounting submounts: mount submount "/sys": failed to mount "/sys" (type: sysfs): no such file or directory, opts: &{{true false false false} true {true 0xc0005a2420} false}And my {
"ociVersion": "1.0.2",
"process": {
"terminal": true,
"user": {
"uid": 0,
"gid": 0
},
"args": [
"/bin/sh"
],
"env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"LANG=C.UTF-8",
"PYTHONUNBUFFERED=1"
],
"cwd": "/"
},
"root": {
"path": "jax-rootfs",
"readonly": false
},
"hostname": "jax-container",
"mounts": [
{
"destination": "/proc",
"type": "proc",
"source": "proc"
},
{
"destination": "/dev",
"type": "tmpfs",
"source": "tmpfs",
"options": [
"nosuid",
"strictatime",
"mode=755",
"size=65536k"
]
}
],
"linux": {
"namespaces": [
{
"type": "pid"
},
{
"type": "network"
},
{
"type": "ipc"
},
{
"type": "uts"
},
{
"type": "mount"
}
]
}
} |
|
I've managed to track down the error to this function: gvisor/pkg/sentry/fsimpl/sys/pci.go Line 223 in e0643b8 Specifically, this function fails when passed peyton@t1v-n-901fc2b8-w-0:~/gvisor$ ls -alh /sys
total 4.0K
dr-xr-xr-x 13 root root 0 Aug 20 16:23 .
drwxr-xr-x 19 root root 4.0K Aug 20 16:24 ..
drwxr-xr-x 2 root root 0 Aug 20 16:23 block
drwxr-xr-x 40 root root 0 Aug 20 16:23 bus
drwxr-xr-x 68 root root 0 Aug 20 16:23 class
drwxr-xr-x 4 root root 0 Aug 20 16:23 dev
drwxr-xr-x 15 root root 0 Aug 20 16:23 devices
drwxr-xr-x 6 root root 0 Aug 20 16:23 firmware
drwxr-xr-x 9 root root 0 Aug 20 16:23 fs
drwxr-xr-x 2 root root 0 Aug 20 19:22 hypervisor
drwxr-xr-x 16 root root 0 Aug 20 16:23 kernel
drwxr-xr-x 152 root root 0 Aug 20 16:23 module
drwxr-xr-x 3 root root 0 Aug 20 19:22 powerI'll continue to investigate why this directory can't be opened. |
|
It seems to me that there's something wrong with the mount. When I look inside the sandbox's namespace, /sys does not exist, but I expect it to: I'm not sure where to go from here - any pointers on what this should look like would be appreciated. |
|
it has been a while since I did it last time, I will share the runsc command with you later.
also, can you share with me what the |
|
@milantracy When I don't pass peyton@t1v-n-901fc2b8-w-0:~/tputesting$ ./start.sh
Child PID: 82314
Press Enter to continue...
# ls -alh /sys
total 0
drwxr-xr-x 12 root root 0 Aug 20 21:47 .
drwxrwxr-x 2 2004 2004 60 Aug 20 21:47 ..
drwxr-xr-x 2 root root 0 Aug 20 21:47 block
drwxr-xr-x 2 root root 0 Aug 20 21:47 bus
drwxr-xr-x 4 root root 0 Aug 20 21:47 class
drwxr-xr-x 2 root root 0 Aug 20 21:47 dev
drwxr-xr-x 4 root root 0 Aug 20 21:47 devices
drwxr-xr-x 2 root root 0 Aug 20 21:47 firmware
drwxr-xr-x 3 root root 0 Aug 20 21:47 fs
drwxr-xr-x 2 root root 0 Aug 20 21:47 kernel
drwxr-xr-x 2 root root 0 Aug 20 21:47 module
drwxr-xr-x 2 root root 0 Aug 20 21:47 powerWhen I do pass |
|
When I spin up a cluster in GKE and run with tpuproxy, this is the sandbox spec that gets used. I would try to copy this spec wrt the mounts and devices sections specifically and see how that works. I see you don't have a |
|
@manninglucas thanks for the config! I've tried this with a {
"ociVersion": "1.0.2",
"process": {
"terminal": true,
"user": {
"uid": 0,
"gid": 0
},
"args": [
"/bin/sh"
],
"env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"LANG=C.UTF-8",
"PYTHONUNBUFFERED=1"
],
"cwd": "/"
},
"root": {
"path": "jax-rootfs",
"readonly": false
},
"hostname": "jax-container",
"mounts": [
{
"destination": "/proc",
"type": "proc",
"source": "proc"
},
{
"destination": "/dev",
"type": "tmpfs",
"source": "tmpfs",
"options": [
"nosuid",
"strictatime",
"mode=755",
"size=65536k"
]
},
{
"destination": "/sys",
"type": "sysfs",
"source": "/sys",
"options": [
"nosuid",
"noexec",
"nodev",
"ro"
]
}
],
"linux": {
"namespaces": [
{
"type": "pid"
},
{
"type": "network"
},
{
"type": "ipc"
},
{
"type": "uts"
},
{
"type": "mount"
}
]
}
}and the run: For what it's worth, this error message is not coming from the inner container - it's coming from the From my testing, I'm actually not sure how the GKE sandbox example works. It looks to me like |
|
I see the issue - the code is looking for TPU devices in specific paths to see if it should bind down to the container. The issues is that in GCE VMs, those paths don't exist! I'm not sure how they work in the first place then, though :) |
|
That behavior is very strange to me. FWIW here's what I see in my VM when I What do you get when you run an unsandboxed TPU workload? How did you create your TPU VM? |
|
@manninglucas It turns out that at least part of this issue was the TPU VM image I was using. I was using I've now switched to ➜ ~ gcloud compute tpus tpu-vm create another-peyton-tpu \
--zone=us-central1-a \
--accelerator-type=v5litepod-1 \
--version=tpu-ubuntu2204-base \
--project=<redacted>While this is getting further, it's now failing at a later step in the parsing: Do you know what image the GKE VMs are using? |
|
I may need to use |
|
@manninglucas I've tried with the new image with no luck. It looks like the device layout still does not match what GVisor expects. If you know what VM image GKE uses that would be helpful. Here is some output: peyton@t1v-n-9becfdd7-w-0:~/tputesting$ python3 -c "import jax; print(jax.device_count()); print(repr(jax.numpy.add(1, 1)))"
1
Array(2, dtype=int32, weak_type=True)
peyton@t1v-n-9becfdd7-w-0:~/tputesting$ sudo find /sys/class/vfio/
/sys/class/vfio/
/sys/class/vfio/0
peyton@t1v-n-9becfdd7-w-0:~/tputesting$ sudo find /dev/vfio/
/dev/vfio/
/dev/vfio/0
/dev/vfio/vfio
peyton@t1v-n-9becfdd7-w-0:~/tputesting$ ./start.sh
running container: creating container: cannot create sandbox: cannot read client sync file: waiting for sandbox to start: EOFAnd here are the relevant logs again: |
|
I believe the image based on COS, should be something like "tpu-vm-cos-109" |
|
@manninglucas Nice those paths exist on that image: peyton@t1v-n-00ca9571-w-0 ~ $ sudo find /dev/vfio/
/dev/vfio/
/dev/vfio/0
/dev/vfio/vfio
peyton@t1v-n-00ca9571-w-0 ~ $ sudo find /sys/class/vfio-dev/
/sys/class/vfio-dev/
/sys/class/vfio-dev/vfio0This image is painful to work with because of the read-only filesystem, though. I may have to bite the bullet and figure out how to do the device mapping on |
|
I will have a patch up soon that will hopefully fix the issue for the ubuntu image you're using. Seems like /sys/class/vfio-dev/vfio0 just corresponds to /sys/class/vfio/0. |
You can remount the filesystem with btw, COS has a tool called |
Ubuntu TPU images do not have the vfio-dev directories that COS images do, so we need a more robust way of setting up the sandbox chroot to handle this case. This change implements a way to get devices and minor numbers into the sandbox with minimal support from the host filesystem and cleans up a few methods to reflect their current usage. Addresses #10795 PiperOrigin-RevId: 667652314
Ubuntu TPU images do not have the vfio-dev directories that COS images do, so we need a more robust way of setting up the sandbox chroot to handle this case. This change implements a way to get devices and minor numbers into the sandbox with minimal support from the host filesystem and cleans up a few methods to reflect their current usage. Addresses #10795 PiperOrigin-RevId: 667652314
Ubuntu TPU images do not have the vfio-dev directories that COS images do, so we need a more robust way of setting up the sandbox chroot to handle this case. This change implements a way to get devices and minor numbers into the sandbox with minimal support from the host filesystem and cleans up a few methods to reflect their current usage. Addresses #10795 PiperOrigin-RevId: 667652314
Ubuntu TPU images do not have the vfio-dev directories that COS images do, so we need a more robust way of setting up the sandbox chroot to handle this case. This change implements a way to get devices and minor numbers into the sandbox with minimal support from the host filesystem and cleans up a few methods to reflect their current usage. Addresses #10795 PiperOrigin-RevId: 667652314
Ubuntu TPU images do not have the vfio-dev directories that COS images do, so we need a more robust way of setting up the sandbox chroot to handle this case. This change implements a way to get devices and minor numbers into the sandbox with minimal support from the host filesystem and cleans up a few methods to reflect their current usage. Addresses #10795 PiperOrigin-RevId: 667652314
Ubuntu TPU images do not have the vfio-dev directories that COS images do, so we need a more robust way of setting up the sandbox chroot to handle this case. This change implements a way to get devices and minor numbers into the sandbox with minimal support from the host filesystem and cleans up a few methods to reflect their current usage. Addresses #10795 PiperOrigin-RevId: 667652314
Ubuntu TPU images do not have the vfio-dev directories that COS images do, so we need a more robust way of setting up the sandbox chroot to handle this case. This change implements a way to get devices and minor numbers into the sandbox with minimal support from the host filesystem and cleans up a few methods to reflect their current usage. Addresses #10795 PiperOrigin-RevId: 674363342
|
Hey @pawalt were you able to get this working for your needs? |
|
hey @manninglucas we deprioritized getting this working. I think Peyton was maybe going to restart the effort when the patch landed: #10795 (comment). |
|
Gotcha. The patch has finally landed (290789b), let me know when you're able to test this out again! |
|
@manninglucas thanks! The container is now starting up. I'm seeing a different issue when trying to use The command hangs on the My startup script: I'm using a jax image exported from the build below: FROM python:3.11
RUN pip install jax[tpu] -f https://storage.googleapis.com/jax-releases/libtpu_releases.html |
|
@pawalt let's follow up with a new issue. Looks like libtpu is looking for some metadata that might be stored in an environment variable. Can you run |
|
@manninglucas I've opened #10923 |
Description
I'm testing out TPU support with the runsc docker shim. When I use
runscnormally, everything works fine, but when used with--tpuproxy, it fails to mount /sys. This is surprising to me because the mount is definitely there.cc @thundergolfer
Steps to reproduce
I've configured docker to use my custom runsc script:
This happens despite the mount existing:
I'm using a v5lite tpu:
runsc version
peyton@t1v-n-901fc2b8-w-0:~/tputesting$ runsc -version runsc version release-20240807.0 spec: 1.1.0-rc.1docker version (if using docker)
uname
runsc debug logs (if available)
The text was updated successfully, but these errors were encountered: