Topic: speculative

Ollama released version v0.32.6, which includes improvements to Qwen3.5's performance on Apple GPUs by automatically using the model's MTP head for speculative decoding and updates to `/v1/chat/completions` streaming to match OpenAI's wire format, enhancing compatibility.

A new method called "Predictive Speculative KV Replication" has been developed to improve the efficiency of Large Language Model (LLM) inference, particularly in handling bursty workloads. This technique aims to reduce latency and resource usage by preemptively replicating key-value pairs, making it crucial for enhancing real-time applications that rely on LLMs.

jwlabs.vercel.app↗