It took me months to get this right for my RTX 6000 ADA 48GB, so I'm sharing it here. Obviously, this is on Linux - Ubuntu to be specific. I get ~60 tokens per second, full context and can run ~16 concurrent sessions.
Helper commands
sudo systemctl enable vllm-27b-fp8
sudo systemctl start vllm-27b-fp8
sudo systemctl stop vllm-27b-fp8
sudo systemctl status vllm-27b-fp8
sudo systemctl disable vllm-27b-fp8
journalctl -u vllm-27b-fp8 -f
sudo nano /etc/systemd/system/vllm-27b-fp8.service
BEGIN SERVICE ENTRY
[Unit]
Description=vLLM …