Time To First Token stats?
Hi, do any of you have benchmarks for Time To First Token for different model architectures and inference engines? I'm wondering if any local setup could match or beat hosted on TTFT latency for something like a 7B model?
Hi, do any of you have benchmarks for Time To First Token for different model architectures and inference engines? I'm wondering if any local setup could match or beat hosted on TTFT latency for something like a 7B model?