Brett

What are you using to run the model? Llama.cpp will automatically split the model between your system ram and graphics card’s vram.

Qwen 3.6 is a mixture of experts model with only 3B parameters active at a time. Even without quantization your card could easily run that.

Nice callback

Is that quantized? 4 bit Qwen 3.6 can get 22tps on a 1060.

Saved. Awesome stuff 🥕

This is a fantastic post. I am very happy to have seen it tonight.