Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs
Yuval Belfer from AI21 Labs shows there is no universal correct size for splitting text when indexing data for AI search.
One example: a question about Jerry's favorite church in Seinfeld transcripts is found best with small text chunks, while a question about his nemesis requires larger context. His solution is to index the same data six times at different sizes, search all of them simultaneously, and then rank the retrieved documents together instead of choosing one fixed size. The method costs two to five times more memory but delivers 20 to 40 percent better search results.
Vibekollen prepared this summary with AI from the original publication. The content belongs to AI Engineer.
More to read
Lumaril, which develops AI software for optimizing industrial production by connecting to existing control systems, raised a $66M Series A at a $400M valuation
Techmeme 4 h ago
DeepSeek open sources Super-Agent as a research prototype, a proactive coding agent it says can monitor group chats for coding issues and suggest solutions
Techmeme 5 h ago
Insurance software maker Guidewire agrees to acquire Hyperexponential, which makes AI-powered insurance underwriting and pricing software, for $1.7B in cash
Techmeme 5 h ago
One word for AI agents in 2027
Nate Herk 9 h ago