What changed

Marvell Technology Inc. and Penguin Solutions Inc. are exploring photonic memory pools as a solution to memory bottlenecks in AI data centers. The approach involves moving high-bandwidth memory (HBM) and DDR5 memory into a separate box connected via Marvell’s Photonic Fabric, allowing multiple servers across racks to share memory resources. This setup enables shared memory pools that can reduce data movement and improve efficiency.

What this means in practice

Penguin’s MemoryAI Photonic Appliance holds 16 terabytes of HBM and DDR5 memory, accessible by servers across multiple racks. Andy Mills, vice president of advanced product development at Penguin Solutions, explained that this setup is necessary to handle large token lengths and prevent LLMs from losing context during long conversations. The use of optics also changes the energy math, with Marvell’s senior vice president Preet Virk noting that shared pools could reduce energy consumption for data movement by up to a quarter, measured in picojoules per bit.

Limitations and availability

The solution is still in development and not yet widely available. The approach requires photonic fabric infrastructure and is designed for large-scale data centers with distributed AI workloads. While the technology is being tested, the supplied announcement does not specify exact deployment timelines or hardware requirements beyond the use of photonic fabric and shared memory pools. The practical implementation of shared pools is expected to address memory bottlenecks as inference scales across more racks and sites.

Source facts for Photonics and shared memory pools aim to reduce AI data center bottlenecks: Memory bottlenecks are emerging as one of the costliest problems in AI infrastructure as inference overtakes training. GPUs sit idle waiting on data, while high-bandwidth memory, or HBM, stays bolted to processors that cannot share it. Photonics is shaping up as one way out.
Original AINewsTrend explainer based on the cited source. This graphic summarizes source statements; it is not an independent benchmark.

Sources and further reading

  1. siliconangle: AI’s hidden energy tax: More than 80% of scale-up traffic is just memory moving around