Basic AMD ROCm setup guide

[shanecbauman]

First of all thanks for such a great program. The bulk TTS generator is exactly what I was looking for and I have been getting a ton of use out of it.

I have gotten this successfully setup with my RX 6750XT and I wanted to share my findings with anyone else who is thinking about running this on an AMD card. Ive only tested the regular model that is downloaded as part of the setup, and have not tried to implement deepspeed or done any model tuning as the vanilla model is sufficient for my use case:

• This was done on linux using an RX 6750XT
• I followed the QUICK SETUP - Standalone Installation guide
• On step 4 during the ./atsetup.sh process, I installed the base requirements for AMD machines
• At this point I could launch ./start_alltalk.sh and tts would work, but it was running on cpu and was very slow.
  • You can check what its running on by the output in the terminal console. The model will either be loaded into cpu or cuda(GPU)
• To get it running on my AMD card I had to uninstall the original pytorch libraries in the conda environment, and re-install the ROCm version of them.

# Enter conda env
$ ./start_environment.sh
# Remove original pytorch libs
$ pip3 uninstall torch torchaudio
# Install ROCm pytorch libs
$ pip3 install --pre torch torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm5.7/

• Once this finishes, exit the conda environment. Re-launch alltalk and verify the model is now loading into cuda.
• NOTE: If you have an older AMD card you may need to run export HSA_OVERRIDE_GFX_VERSION=10.3.0 before launching alltalk in order to get it running properly. (I had to do this for my RX 6750XT)

I saw that ROCm support is on the feature request list which is great to hear. Hopefully this information will help out anyone else who is running AMD.

[erew123]

Hi @shanecbauman

Wow! After ZLUDA getting law suited to high heaven https://github.com/vosen/ZLUDA by Nvidia, I kind of thought getting ROCm to just work as a CUDA device would be dead in the water.

You're not running ZLUDA, right? Just plain old ROCm (latest obviously) and the steps you said above? Either way is great news as I've not been able to test out any theories of getting an AMD card working (not having one personally).

Feel free to feed back anything you notice and I have no idea if DeepSpeed will or wont work on an AMD flavour of CUDA, if the card is working with CUDA instructions, Ill give it a 30-40% chance it might, so you may be able to get a bit more of a speed up.

Thanks so much for the instructions, hopefully someone else can give it a test at some point! :)

If anyone else sees this and is willing to give it a test, please feed back and let us know how you get on!

Thanks so much @shanecbauman! I really appreciate you taking the time to let me know how it worked/went and writing out some instructions.

Really awesome.

[shanecbauman]

Wow, ZLUDA looks great! Its a shame seeing compatibility efforts being stifled so much by the big players, hopefully progress on it will continue.

But no im only using the latest ROCm libraries. I think the work AMD has done with HIP has really helped with making pytorch applications more portable between the GPU vendors. Ive had pretty good luck running automatic-1111s SD webui on my AMD card, so I was using the GPU prerequisites check in webui.sh as a reference for troubleshooting this.

I will definitely be trying out deepspeed once I have more time to look into it, that speed improvement would really be nice. If I have any more success getting the rest of the features working Ill try to update here with my findings.

[erew123]

Huh, well that's really interesting! I wonder if AMD are silently pushing better CUDA integration, perhaps from ZLUDA, into their drivers.

Really good to hear though! Ill have to get someone else to test. I guess it should theoretically work on windows too!

Thanks so much for the info!

[RenNagasaki]

So I tried getting that working on a windows machine. Sadly ROCM ist not supported by windows, to be more clear PyTorch isn't.
And I failed to get ZLUDA to work as well.

[RenNagasaki]

Seems like I need to disable CUDNN. Any Idea which file I need to edit to disable that for PyTorch?

torch.backends.cudnn.enabled = false needs to be set

[RenNagasaki]

[erew123]

Off the top of my head, I don't, though obviously it may just be specific calls that arent supported by AMD cards (too Nvidia specific)! Can you actually see it processing/generating on your GPU when you send a request over?

[RenNagasaki]

ROCm seems to slowly start supporting wsl. When that's possible, I'll try again.

[RenNagasaki]

Off the top of my head, I don't, though obviously it may just be specific calls that arent supported by AMD cards (too Nvidia specific)! Can you actually see it processing/generating on your GPU when you send a request over?

Got that fixed, but cufft_internal_error is the next problem. Seems to be the problem that there's no PyTorch Version for ROCm for Windows.

[RenNagasaki]

Sadly that won't work, since I want the end user to use this to inference live while gaming. So it HAS to run on windows.

[RenNagasaki]

Since I for the moment just develop this for me and a few friends, its fine.
If I ever plan to release it for everyone to use, I hope there's a solution.

[yujikaido]

Any update? I have a Radeon 6600m and I have successfully gotten stable diffusion via sd.net running on my mini pc running windows using Zluda after installing ROCm and patches for my GPU.. its works great compared to CPU only, I have been trying to figure out how to use Zluda with XTTSv2 but I only have had help from Chatgpt and haven't made much progress yet.

[erew123]

@yujikaido I personally havnt made any progress, due to a lack of an AMD card, so I cant test it. As I think I mentioned, I believed/understood that Zluda should just intercept PyTorch CUDA calls and convert them over to AMD ROCm calls..... but @RenNagasaki attempt above seemed to show it didnt. I dont know if that would be a limitation of Zluda or maybe Coqui's TTS engine or what..... On the flip side of that, I have broken out all the loaders in V2 of AllTalk, so Im hoping to find someone with some coding experience and an AMD card whom might like to take a pop at getting things working. Because the loader code is broken out, its much easier for someone to work on/test and understand what does/doesn't work. So hopefully I can find someone to give it a shot.

[yujikaido]

ok Thank you, I fiddled with it myself and got similar error even though it was detected and loaded into cuda. Guess I will try Ubuntu and see if it works there. Edit, I just got it working with my setup on ubuntu using the Rocm 6.0 following the instructions in the first post. Thank you erew123 and Thank you shanecbauman for the tip.
Only issue I seem to be having is long texts input seem to be an issue so i will have to keep it shorter.
Now I am going to try and get deepspeed to work with it.

[Neoony]

Just to mention

V1 Seems to work in Windows on WSL2 with Ubuntu 22.04 and Radeon 7900 XTX
Now that the AMD drivers support that ( but only some GPUs :( )

Uses my GPU

Instructions:
Might not be proper instructions, took a while until I got there and not sure what is actually needed or not.

Got the requirements (already had it since I tried Stable Diffusion)

sudo apt update
wget https://repo.radeon.com/amdgpu-install/6.1.3/ubuntu/jammy/amdgpu-install_6.1.60103-1_all.deb
sudo apt install ./amdgpu-install_6.1.60103-1_all.deb
sudo amdgpu-install -y --usecase=wsl,rocm --no-dkms

sudo reboot

Then in alltalk-tts after its set up already

./start_environment.sh
pip3 uninstall torch torchaudio
pip3 install --pre torch torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm6.1

Then I also did this I think (based on SD Next instructions on Discord)
Not really sure if this is correct
Inside venv

"Patch PyTorch"

location=`pip show torch | grep Location | awk -F ": " '{print $2}'`
cd ${location}/torch/lib/
rm libhsa-runtime64.so*
cp /opt/rocm/lib/libhsa-runtime64.so.1.2 libhsa-runtime64.so

And then I needed to also copy this file into the lib folders overwriting the one there
Otherwise I would get errors that libhsa-runtime64.so expects different libstdc version
Outside of the venv

sudo cp -f /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /somelocation/alltalk_tts/alltalk_environment/env/lib/
sudo cp -f /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /somelocation/alltalk_tts/alltalk_environment/conda/lib/

just make sure to correct the path

And start it up and then it works.
Something like that :P

This would take around a minute to generate on my 5950X CPU

I also read some advices about using a newer kernel than the one in ubuntu 22.04
So that it can utilize the 7900 XTX better
I will have to try (but I guess WSL2 has its own kind of kernel...hmm maybe thats only something for non WSL linux)

[Neoony]

the v2 is not working however (xtts)
I also tried v2 before v1 and resulted in same error as now
Now I tried again after v1 is ok, but same error

[erew123]

@Neoony Thanks for all your feedback on this! Its very appreciated.

Best I can see with this error and having a look around the internet, this would suggest that the pip3 install --pre torch torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm6.1 isn't properly installed/set-up. You may want to try it with the --upgrade --force-reinstall

pip3 install --upgrade --force-reinstall --pre torch torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm6.1

Ive seen pip sometimes be very funny when there is a local cached version of Pytorch and it not actually bothering to install the version you tell it to.

[Neoony]

v2 does not seem to work, no matter what I try (also tried the command you mentioned)

The thing is, I also get some amdsmi errors in SD.next and ComfyUI when I use the official nightly pytorch, instead of the ones recommended from AMD
The wheels from AMD: https://rocm.docs.amd.com/projects/radeon/en/latest/docs/install/wsl/install-pytorch.html

e.g. in SD.next with the official nightly pytorch
/home/someuser/.local/lib/python3.10/site-packages/torch/cuda/init.py:701: UserWarning: Can't initialize amdsmi - Error code: 34
warnings.warn(f"Can't initialize amdsmi - Error code: {e.err_code}")

But that does not prevent it from working, everything works just fine
(no idea why this error happens)

The wheels from AMD dont seem to have torchaudio and they also seem to require python 3.10
With those the amdsmi error does not happen anywhere
But as mentioned, I dont see the torchaudio from AMD, and I would somehow have to make the v2 use python 3.10

hmm :/