5 posts
When people estimate token costs, they usually watch TTFT, TPOT, and throughput. What actually makes bills differ by 10× is whether the KV cache hits. The model, server, and user layers all have to line up.
Demand for GPT-5.5 and Opus 4.7 is nearly infinite, the mid-tier has vanished, and low-to-mid-range compute sits idle. The token economy sounds like selling electricity, but it's more like a gas station: 98-octane is sold out, diesel tanks are full yet self-service only, and 95-octane sits empty.
The Gemini 3.5 Flash launch barely talked about intelligence; Zhipu’s GLM-5.1 High-Speed edition hit 400 token/s. Both tell the same story—once inference speed crosses that 5× threshold, the model unlocks an entirely different product category.
I ran seven hundred rounds of AI Infra experiments, and thirty-five hours were entirely eaten up by environment startup. At first I thought GPT-5.5 fast mode wasn't fast enough, but later realized it wasn't the model thinking—it was the model waiting for the CPU. Intel has already tightened the server CPU:GPU ratio from 1:8 to 1:1.
V4 capabilities sit around the Opus 4.6 tier, but pushing FP4 to production, making million-token context the default, and day-0 adaptation for domestic chips is a disaster for everyone in the inference infra business. Add GPT-5.5, Vision Banana, and LPM 1.0 into the mix, and this week has crammed in more new releases than the entire past quarter.
