Microsoft has announced the launch of “Mu,” a new compact on-device language model specifically engineered to operate entirely on the Neural Processing Units (NPUs) found in its latest Copilot+ PCs. This development marks a significant step towards enabling powerful AI functionalities directly on personal computers without constant reliance on cloud connectivity.
Mu, a 330 million parameter model, was rigorously trained using Azure’s powerful A100 GPUs and refined through a multi-phase fine-tuning process. Its innovative design leverages a Transformer encoder-decoder architecture, which distinctly separates input and output tokens. This architectural choice allows for significantly faster and more efficient inference, a critical factor for on-device performance.
In terms of performance, Mu delivers impressive speeds, achieving over 100 tokens per second. Notably, it reduces first-token latency by 47% compared to similarly sized decoder-only models. On the new Surface Laptop 7, Mu is capable of generating over 200 tokens per second, demonstrating its real-time processing capabilities.
One of Mu’s initial practical applications is its integration as an AI agent within the Windows Settings app. This allows users to intuitively adjust system settings using natural language queries, transforming how individuals interact with their operating system.
To optimize the model for device-specific requirements, Microsoft implemented post-training quantization, converting model weights to lower-precision formats. These crucial optimizations were a collaborative effort, involving close partnerships with leading hardware manufacturers such as AMD, Intel, and Qualcomm.
The development of Mu builds upon Microsoft’s previous work with its Phi and Phi Silica models. It is anticipated that Mu will serve as a foundational element for a new generation of AI agents and enhanced intelligent features across future Windows devices, further cementing the role of local AI in personal computing.
Also read:Â Viksit Workforce for a Viksit Bharat
Do Follow: The Mainstream formerly known as CIO News LinkedIn Account | The Mainstream formerly known as CIO News Facebook | The Mainstream formerly known as CIO News Youtube | The Mainstream formerly known as CIO News Twitter |The Mainstream formerly known as CIO News Whatsapp Channel | The Mainstream formerly known as CIO News Instagram
About us:
The Mainstream formerly known as CIO News is a premier platform dedicated to delivering latest news, updates, and insights from the tech industry. With its strong foundation of intellectual property and thought leadership, the platform is well-positioned to stay ahead of the curve and lead conversations about how technology shapes our world. From its early days as CIO News to its rebranding as The Mainstream on November 28, 2024, it has been expanding its global reach, targeting key markets in the Middle East & Africa, ASEAN, the USA, and the UK. The Mainstream is a vision to put technology at the center of every conversation, inspiring professionals and organizations to embrace the future of tech.
