Topic: distributed

OpenAI transformed Habitat, initially a Python library, into a global storage platform to handle the massive demand from over 1 billion ChatGPT users, processing up to 22 million requests per second. This scaling is crucial for maintaining user experience and supporting the application's rapid growth.

openai.com↗

A new paper proposes X-CoSD, a communication-efficient method for collaborative speculative decoding that involves an on-device small language model generating candidates while a server large language model verifies them, aiming to improve distributed inference processes in large language models. This approach is crucial as it could enhance the efficiency and scalability of AI applications across devices.

arxiv.org↗