What changed

AWS has improved its embedding throughput by 50% through optimizations on Neuron hardware, specifically AWS Inferentia2 and Trainium2 chips. This enhancement allows for more efficient indexing of large volumes of documents, supporting advanced semantic search and data retrieval capabilities.

What this means in practice

The optimization was driven by the need to re-embed a corpus of 30 billion high-priority documents, which temporarily doubled the embedding load. This prompted a re-evaluation of hardware choices and led to a series of optimizations that significantly improved throughput. The improvements were achieved through weight layout optimization, autocast, and custom kernel development, resulting in a 47% improvement in performance and cost per 1 million documents embedded. These optimizations are now part of an agentic development workflow that allows for continuous improvement and validation of the embedding process.

Limitations and availability

The optimizations are currently available on AWS Inferentia2 and Trainium2 hardware, with the initial implementation running on a single NeuronCore. The performance gains are specific to these hardware instances and may vary based on the region and availability of spot and on-demand instances. Exa has optimized the setup for the US West (Oregon) Region (us-west-2), but further tuning across other regions is needed to ensure consistent performance and availability.

Source facts for AWS boosts embedding throughput by 50% using Neuron hardware: The keystone technology behind this is an in-house vector database. It must ingest as much high-quality web data as possible and embed all of it. At the scale of tens or hundreds of billions of documents, finding the hardware that delivers the best performance per dollar isn’t a tuning detail.
Original AINewsTrend explainer based on the cited source. This graphic summarizes source statements; it is not an independent benchmark.

Sources and further reading

  1. awsml: Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips