Can't boot Pi 5 via NVMe behind PCIe switch / bridge #1833
Comments
|
It's not supported right now |
|
What "PCIe switch" do you have? |
|
@peterharperuk - I'm testing with this I/O Crest switch: |
|
I've just come across a HAT+ (Geekworm X1004) with an ASM1182e PCIe switch for two SSDs and the manufacturer claims it can't be used for booting at this time. It looks related. |
|
@cnxsoft - It is most definitely related! And thanks for you post on that board, I really do hope Raspberry Pi can support at least a few switches (the asmedia ones seem extremely popular for smaller companies. I don't see Pericom that often). |
|
I heard the guys from Pineberry are also already experimenting with a hat that supports 2 m.2 devices... (no promises though.. ) 2 NVME drives or 1 NVME drive and room for a pcie coral TPU would be nice but of course we need to be able to boot from it too if a pcie switch is involved :) |
|
Removed off topic comment about a CM4. This issue is specifically about Pi5 / BCM2712 |
|
That's covered in the documentation. From https://www.raspberrypi.com/documentation/computers/raspberry-pi-5.html#pcie-gen-3-0:
If Gen 3 was guaranteed to be stable then it would be enabled by default. |
|
@Ronald1817 - The title of this GitHub issue is "Can't boot Pi 5 via NVMe behind PCIe switch / bridge" and is meant to discuss that topic, not general NVMe SSD issues you may be having. The Raspberry Pi forums, or the manufacturer of the HAT / Bottom you're using, would be a better avenue for questions about Pi boot issues with NVMe. That is all that is implied by @timg236—this GitHub repository is for issues pertaining to the Pi firmware, and this issue is about a specific feature request. These comments don't have anything to do with that, so typically the maintainers will clean them out at some point. |
|
A new dual NVMe rpi5 product has appeared with a PCIe switch in it https://pimoroni.com/nvmeduo |
|
@greyltc - Looks like @timg236 / @pelwell — it seems like by some stroke of luck, all the main vendors of PCIe addon boards have settled on the asmedia |
|
https://pineberrypi.com/products/hatdrive-ai-coral-edge-tpu-bundle-nvme-2230-2242-gen-2-for-raspberry-pi-5 |
|
Recently I have bought an PCIe HAT from aliexpress.com. The link is https://www.aliexpress.us/item/3256806347359812.html. But I found two interesting things:
Let me explain below what I did, I didn't go in depth to find out how it worked. But at a high level,
I didn't know what I did except expanding the root partition size to utilize the full NVMe (512G) size, (SD card was 128G), I resized the partition in NVMe. While preparing to answer this, I found that the below in lsblk command. mmcblk0 179:0 0 116.2G 0 disk Both boot and root partitions are used from NVMe only. Now this is running as my home router / gateway. So, I can't do much tests immediately. But I will try to reboot multiple times/ try without SD card and see what's happening when I get a chance and post it here. |
|
Can I please add the ASM1184e PCIe packet switch chip to this issue? It shows up on the ZS ZHISHANG PCI-E X1 to 4 PCI-E X16 Expansion Riser as documented by @geerlingguy. The ASM1182e is mentioned above. It's a 2 by PCIe chip vs the ASM1184e which is 4 by PCIe chip; so similar(ish). Boards available on Amazon via:
I will note that this PCIe bridge works fine once the system is booted. |
In principle any switch that follows the spec should work with generic firmware (once written). Supporting / debugging individual switch/PCIe devices would be up to the HAT designer i.e. debugging low level electronic issues. |
|
To be clear, my obeys the spec I mean that anything that requires a ton of quirks an interop fixes just won't be supported. No immediate plans to do this though |
Agreed. The intention would be to support booting from an NVMe driver behind a switch, probably only simple hardware topologies would be supported |
|
@timg236 - The other use case I have is to have multiple boot drives in a compact solution, so a Pi could boot into different OSes more easily than with a bunch of USB stuff hanging off it (or PXE boot). That... would be interesting to do from the Pi OS default firmware experience, though a UEFI bootloader could have boot order priority. I don't know the default way the bootloader picks which device to boot from first, if there are multiple on one bus like USB. Is it by device ID? |
|
The bootloader enumerates MSD in parallel and you get the first one that looks like valid OS. It’s mentioned in the msd docs. |
|
@geerlingguy The output of lspci would be interesting. To fully debug you devices I'd need the output of "sudo lspci -xxx" If you attach it as a file I'll try and take a look. If you see the same problems when booting from an SD card with rootfs on the NVMe disk then that would rule out your issues being caused by some side effect from the bootloader. |
|
@peterharperuk it seems like the Phison E13 controller has aspm issues behind the switch. The latest firmware is dated August 2022, maybe they'll update it at some point. I tested all the problematic drives from WD and they boot normally with ASM1182 and the latest firmware. @geerlingguy instead of globally disabling aspm please use: dtparam=pciex1_no_l0s=on and it'll work fine. |
|
@mikegapinski - Ah, thanks. I have edited Everything seems to be working fine now, and my impromptu benchmark gives: So getting PCIe Gen 2 without issue now. Thanks! Now the race begins to figure out which drives work better than others out of the box behind a switch :D |
I can think of at least 5 SKUs that throw errors only behind a switch (Not only Phison but also Maxio controllers). @peterharperuk at least initially pciex1_no_l0s could be set by default when you boot behind a switch. Having to remove a drive and put it in the enclosure to set it up after NVMe boot is enabled is not ideal. I don't know why ASPM errors on certain drives are only thrown behind a switch, maybe it could be fixed in the firmware update. |
|
Changing global default to workaround drive specific issues is generally the wrong thing to do. It's possible to set these as board level options via the bootloader flash config. https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#config_txt |
|
What you mean is that instead of adding "pcie_aspm=off" to "/boot/firmware/cmdline.txt" we should: Is the above method correct? |
|
dtparam=pciex1_gen=3 - does not make sense with a switch, those are mostly Gen 2 |
OK, we are interested in this configuration. And remove "pcie_aspm=off" from "/boot/firmware/cmdline.txt" |
|
If the HAT meets the M2.HAT spec or the system has booted from NVMe then these lines are redundant. |
|
I don't have any drives that suffer from this problem and I've got the cheapest drives I could find from Amazon. Where can I get these problematic drives from? |
Those have the controller that only throws errors behind a switch (Phison E13, make sure it has the latest 2022 firmware. The last character in the firmware version corresponds to NAND type):
If you want something that always throws ASPM errors and barely works in GEN 3 on regular M.2 HAT+ boards go with the Lexar NM620 |
|
ok, found one drive (Patriot) that seems to have a problem. Oddly, it only has an issue if it's plugged into the second downstream bridge. |
|
I had this issue yesterday as well but a sudo rpi-update solved it for me so I´m up and running with my DIY NAS :) I´m using the below parts:
|
|
A side note for @peterharperuk (if I may) ...
Sadly the $ env | egrep ^TEST_
$
$ TEST_A=1 TEST_B=2 TEST_C=3 env | egrep ^TEST_
TEST_A=1
TEST_C=3
TEST_B=2
$
$ TEST_A=1 TEST_B=2 TEST_C=3 && env | egrep ^TEST_
$
$ TEST_A=1 TEST_B=2 TEST_C=3 sudo env | egrep ^TEST_
$
$ TEST_A=1 TEST_B=2 TEST_C=3 && sudo env | egrep ^TEST_
$
$ (export TEST_A=1 TEST_B=2 TEST_C=3 && env | egrep ^TEST_ )
TEST_A=1
TEST_C=3
TEST_B=2
$
$ (export TEST_A=1 TEST_B=2 TEST_C=3 && sudo env | egrep ^TEST_ )
$You've got to do the environment variable allocation after the sudo. $ sudo bash -c "TEST_A=1 TEST_B=2 TEST_C=3 env" | egrep ^TEST_
TEST_A=1
TEST_C=3
TEST_B=2
$This is a thing in all shell's dating back since the start. That all said, the $ TEST_A=1 TEST_B=2 TEST_C=3 sudo --preserve-env env | egrep TEST_
TEST_A=1
TEST_C=3
TEST_B=2
$ TEST_A=1 TEST_B=2 TEST_C=3 && sudo --preserve-env env | egrep TEST_
$Hence my final recommended version would be: $ SKIP_KERNEL=1 SKIP_FIRMWARE=1 SKIP_SDK=1 SKIP_VCLIBS=1 sudo -E rpi-update
...
$Hope that helps. (PS: can't wait to test the new boot this weekend as I also have NVMe's installed after a switches) |
|
@mahtin It might be better to move rpi-updapte params to an issue on that repo. I'd recommend just running vanilla rpi-update because you almost certainly want the latest 6.6 kernel when testing pre-releases of the Pi5 firmware. BETA bootloader releases were dropped from rpi-eeprom APT and made available because it's easier to bisect releases via commit ids with rpi-update and on Pi5 the bootloader SPI flash image is effectively start5.elf.
|
|
Thanks @mahtin I have updated the original post in case anyone finds it again. |
|
Thank you @peterharperuk,
I think the anyone these days will mainly be some LLM scraper; but, I digress. I took @timg236's advice and went and looked at https://github.com/raspberrypi/rpi-update and sure enough it gets it right by using the somewhat simpler (and specific to $ sudo SKIP_KERNEL=1 SKIP_FIRMWARE=1 SKIP_SDK=1 SKIP_VCLIBS=1 rpi-update
...
$Sorry to distract from the task at hand. [1] Insert oblatory quote: If you learn something new every day, you can teach something new every day. - Martha Stewart |
|
Hi, I followed manual on https://pineboards.io/blogs/tutorials/how-to-configure-the-google-coral-edge-tpu-on-the-raspberry-pi-5 firstly it didn't boot from the NVMe behind switch and it did boot without it. the TPU works only connected to RPi (but I successfully saw the TPU before I started messing with the NVMe booting) I have tried every manual, all here what I could read. the only thing I haven't try is the "NVME_Controller=0", because I don't know where to write it, but not sure if it help. (I would rather have the NVMe on pcie1 so the cooling works better, but it does not work. pls see below info: |
|
Your comment doesn't seem to be about booting from NVME behind a PCIe switch. |
it is, because it doesn't work on pcie1 and also there is the secondary problem with TPU |
It is drive related, some of them need a PCIe compat flag for ASPM behind a switch. I don't know anything about Shenzhen Longsys Electronics controllers, but adding one of those overlays should make it work: @HonzaJaros generally you should contact our support before raising an issue here, it is slightly offtopic (as @pelwell highlighted) |
|
I see. The TPU issue is a distraction here. It is worth trying the TPU with combinations of the compatibility flags @mikegapinski mentions above, but that won't affect booting. |
@pelwell Yes and no, drives with Phison E12 controllers won't boot past bootloader without the no-l0s flag. Coral needs no-mip. But that's a different story |
|
Yes, that's possible - but then it would be a kernel issue, not a firmware issue. |
Hi Mike, I did, 2x (email and form), sorry I didn't realised that its not related. |
Describe the bug
I am unable to boot a Raspberry Pi 5 from an external NVMe SSD if used behind a PCIe switch (e.g. not as the root device on the external connection).
To reproduce
BOOT_ORDER=0xf25416)Expected behaviour
I would expect the NVMe SSD to be selected for boot, wherever it is enumerated on the PCIe bus.
Actual behaviour
The Raspberry Pi 5 bootloader attempts to load
nvmebut fails, likely due to it only enumerating devices directly attached to the external port, and not walking down the tree of any other connected PCIe bridges...System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:
cat /etc/rpi-issue)?vcgencmd version)?uname -a)?Logs
Click to expand full bootloader log (captured via UART)
Additional context
When behind the bridge, here is the hierarchy according to
lspci:I know for the Compute Module 4, the concern was a lack of space in the bootloader to successfully enumerate all PCIe devices, no matter where they are on the bus. Does the Pi 5's bootloader overcome that limitation? I don't expect this to work on launch day, but it is something I think a lot of people would like to do (e.g. stack an 'NVMe + 2.5G Ethernet' HAT, or 'NVMe + WiFi 7' HAT, etc. on top).
The text was updated successfully, but these errors were encountered: