What's New in Inference Engineering — Philip Kiely, Baseten

Philip Kiely from Baseten describes how inference engineering—the process of running trained AI models—evolves differently for local computers versus large data centers.

A technique called TurboQuant, which reduces the memory footprint of models, went viral in March, but Kiely shows it is slow for data centers even though it works well at home. He presents three main improvements: quantization (making models smaller by using fewer bits per number), cache optimization (storing and compressing computation data smartly), and speculation (small helper models quickly guess the next tokens, which data centers now train specifically for). The biggest trend is that these techniques are trained as their own process rather than applied afterward.

More to read

Lumaril, which develops AI software for optimizing industrial production by connecting to existing control systems, raised a $66M Series A at a $400M valuation

Techmeme · 3 h ago

DeepSeek open sources Super-Agent as a research prototype, a proactive coding agent it says can monitor group chats for coding issues and suggest solutions

Techmeme · 4 h ago

Insurance software maker Guidewire agrees to acquire Hyperexponential, which makes AI-powered insurance underwriting and pricing software, for $1.7B in cash

Techmeme · 4 h ago

One word for AI agents in 2027

Nate Herk · 8 h ago