ERNIE-5.5-Flash-Base Unveiled

🌐 @missedminutes 🕒

Key points

• 280B MoE with only 14B active per token, pretrained from scratch on 33.5T tokens
• Pure base model with no mid- or post-training, enabling domain-specific adaptation
• Only 6 of 43 layers keep token-level access to full context; other 37 squeeze every 128 tokens into a single cached entry, keeping KV cache small while supporting up to 1M tokens for fine-tuning and inference
• Multi-Router MoE: multiple routers select experts together without increasing activated expert compute or communication costs
• ECHO: multi-token prediction runs through the model's own final layer, eliminating need for separate MTP module
• Early evals show strong results: MMLU-Pro 78.4 vs DeepSeek-V4-Flash-Base's 69.2; HumanEval+ 82.3 vs 69.5; LiveCodeBench 40.1 vs 31.9

First posted on the Missed Minutes channel.