What factors affect LLM API latency? How do you optimize?

Answer preview

Factors: Model size, output length, queue depth, network, region, context size.…