DeepSeek V4: AI Model's Trillion-Parameter Breakthrough

Summary

Discover how DeepSeek, a remarkably resource-constrained team, has developed the DeepSeek V4 AI model, boasting 1.6 trillion parameters and a 1 million token context window, rivaling top closed-source AI labs. This achievement is attributed to innovative solutions like hybrid attention, which compresses and selectively uses past information to overcome the computational and memory bottlenecks of long contexts. Furthermore, DeepSeek V4 employs manifold constrained hyperconnections to prevent signal explosions in its massive neural network and a custom optimizer, Muon, for faster, more stable training. The model's efficiency is further enhanced by optimized data transfer choreography within data centers and a curriculum learning approach for training data. Remarkably, DeepSeek V4 has achieved a perfect score on the Putnam 2025 math competition and matches or exceeds the performance of leading models like GPT-4 on various benchmarks, all while being open-sourced.