Topic: distribute

OpenAI transformed Habitat, initially a Python library, into a global storage platform to handle the massive demand from over 1 billion ChatGPT users, processing up to 22 million requests per second. This scaling is crucial for maintaining user experience and supporting the application's rapid growth.

openai.com↗

A new paper proposes X-CoSD, a communication-efficient method for collaborative speculative decoding that involves an on-device small language model generating candidates while a server large language model verifies them, aiming to improve distributed inference processes in large language models. This approach is crucial as it could enhance the efficiency and scalability of AI applications across devices.

arxiv.org↗

The ggml/llama.cpp project updated its codebase to convert the FILL operation into a 2D distribution of workgroups in Vulkan, addressing an issue with Intel GPUs on Qwen 3.8, ensuring compatibility and performance optimization. This update is crucial for maintaining functionality across different hardware architectures.