There are dozens of us! Dozens! (4x B70)

Finally added the next two B70s to the rig (H12SSL-i Epyc 7402, 128GB 8-channel 2133).

Currently serving 4x Strata engines running Qwen 3.8 Flash Next Q4 K M with 262k 8-bit KV. All engines are served through a custom router that dynamically serves users based on GPU availability and pre-existing cache (users mid-conversation stay on their engine).

All four engines attach to ONE expert mirror in pinned host RAM, so the expert weights remain the same regardless of engine count, which I’m pretty proud of.

Current speeds: \~1300-1400tps prefill on long prompts, \~50tps single stream decode.

EDIT: Switched to vLLM and am getting 3.1k prefill and 138t/s single stream decode. Four streams sees over 300t/s now!