High Bandwidth Flash: A New Memory for AI Data Centers and Edge Computing

Artificial intelligence is on a relentless march across the computing landscape. While about one in seven data centers today is equipped to host AI workloads, that’s expected to approach 70% by 2030.¹ AI is migrating from hyperscale to enterprise data centers and out to the network perimeter, where edge AI applications are projected to generate nearly $66.5 billion by the end of the decade.² The fuel for the new computing era is data — staggeringly large volumes that must be fed at high speed to demanding and rapidly scaling AI computing infrastructure.

These vast content repositories overwhelm conventional storage structures and bring an inherent architectural weakness into sharp relief. Data center memory (DRAM and specialized high-bandwidth memory known as HBM) is increasingly struggling to keep pace with the growing demands of large AI models in terms of density, storage capacity and scalability.

At the same time, hyperscale computing manufacturers are contending with rising DRAM and HBM production costs, design complexity, and energy consumption. The challenge is even more daunting in enterprise data centers and edge AI applications, where a proportionately smaller physical footprint renders them ill-equipped to absorb rising memory costs and power use.

Another pressing issue is introduced by AI inference, which is now the dominant AI workload and has different data-management requirements than AI training. Inference stores large — and growing — AI models, and HBM and DRAM-based memory have shown they lack the capacity and cost scalability to keep up with these new demands. Given these distinctly different memory characteristics, an opportunity exists for a memory technology optimized specifically for AI inference.

Why DRAM and HBM Underserve AI Inference Workloads

To understand why DRAM and HBM alone are suboptimal for long-term AI deployment, consider the following drawbacks.³ These began as small fissures, but if uncorrected, they will expand over time to undermine the foundation of next-generation AI-centric storage.

• Density penalties: DRAM capacity scaling has stalled while the need for higher capacity to address AI inference is growing.³
• Mismatched for AI inference: The advantage of DRAM’s low latency and random-access characteristics aren’t relevant for AI inference, where access patterns are deterministic and more latency-tolerant thanks to techniques like data prefetching.³

Attributes of an Optimized AI inference Memory Architecture

These fault lines run beneath a $120 billion DRAM industry⁴ that’s eager to retain its hold on the data center, given that spending by hyperscale providers on AI infrastructure could reach $6.7 trillion by the end of the decade.⁵

What if it’s time to make a clean break and design a new memory from the ground up that meets the needs of the application, rather than the other way around? An AI-tuned storage-class memory would have the following attributes:

• Larger and scalable memory capacity provisioned for inference workloads
• Higher memory density (GB/mm²)
• High bandwidth to meet the requirements of AI inference      
• Lower system-level power consumption
• Cost-effective metrics ($/TB)

High Bandwidth Flash Takes Aim at the AI Data Center

High Bandwidth Flash (HBF) is a disruptive new memory architecture, purpose-built to drive the next generation of AI computing. HBF meets the capacity, energy, throughput, and scalability requirements of advanced computing and data-intensive applications.

Compared to HBM, HBF provides higher capacity and memory density with comparable bandwidth that better aligns with AI inference trends (Fig. 1). As a persistent storage medium, HBF also retains data when power is lost and it’s thermally stable, supporting high operating temperatures.⁶