I may or may not have splurged on a 128GB AMD Ryzen AI Max 395 (strix halo) system for ‘AI stuff’ (told you I was a noob).

I’ve been running Ubuntu on it with the AMD drivers (think its ROCm?), ollama and seems to be working fine.

An LLM told me to change the RAM/VRAM ratio to 50:50 (so 64GB for the CPU, 64GB for the GPU). I dunno if that was correct, seems like a waste tbh. Feels like I could give the GPU more resources and run bigger models.

I’ve read about Lemonade being better than Ollama on strix halo? Also, I realised that I might not be using the NPU as extra work is required to get that up and running.

I’m looking for advice from users on the same hardware. What OS are you using? How do you have the RAM/VRAM ratio configured? What’s your stack? That sorta thing.

PS - If my wife asks, the machine only cost like £250 and is a second-hand floor model.

  • [object Object]@lemmy.caEnglish
    151·
    19 days ago

    I use Strix Halo

    On Linux you can just go into the bios and map like 512mb video and then update your grub config and you’re fine, Linux knows to use the unified memory correctly. I don’t know what your LLM told you, but you do not need a 50/50 split. The whole thing has worked for over a year now smoothly, and there are guides like kuz0’s online to help you get everything set up.

    Lemonade is better than Ollama for a few reasons:

    • lemonade is open source, ollama is a for profit closed source company that gives you simple and terrible defaults. They want you to pay for their cloud API. Ollama is also a hard fork of llama.cpp that never contributes back, and has unique bugs.
    • lemonade can run many different engines, llama.cpp/stable diffusion, whatever, and AMD ships them precompiled for lemonade, so the rocm support is good
    • lemonade does a reasonably good job making things easy to use but you can add params for whatever llama.cpp flags you want. It’s a good middle ground, but the default context window sizes need to be bumped (typically to 128*1024 on 128gb strix, more can get quite slow). Ollama has the same issue.

    However, on the negatives, lemonade lags behind upstream engines, so if you wanted to say use Qwen 3.8 125b that just came out, you have to build it and link to rocm yourself. Usually it’s fine waiting a couple weeks, but sometimes I run a custom build and I point lemonade at it.

    My stack is NixOS+Podman. I run lemonade in a container it all works great. I use Caddy + Authelia for networking and i serve it over tailscale to my tailnet.