Topic: decoding

A new method called Recursive Sparse Attention allows for efficient processing of up to one million tokens, significantly reducing computational costs compared to dense attention techniques, thus enabling better handling of longer documents in machine learning models. This breakthrough is crucial as it addresses the scarcity of long training data and improves the scalability of language models.

arxiv.org↗

A new study by TraceSpec examines speculative decoding in large language models (LLMs) to better understand and improve how LLM agents make decisions during task execution, emphasizing the need for more detailed supervision.

arxiv.org↗

Polymathic AI and Princeton researchers have developed FlashMap, a speculative decoding drafter that can draft 16 tokens in one pass, achieving up to 6.14 times faster lossless speedup on Qwen3-8B, significantly enhancing LLM decoding efficiency.

A new method called SPECMEM aims to accelerate tool-calling agents by using speculative decoding and live memory management, reducing latency without altering accuracy. This advancement is crucial as it addresses the growing issue of deployment latency in tool-calling agents, making their use more efficient and practical.

arxiv.org↗