What changed
AWS has improved its embedding throughput by 50% through optimizations on Neuron hardware, specifically AWS Inferentia2 and Trainium2 chips. This enhancement allows for more efficient indexing of large volumes of documents, supporting advanced semantic search and data retrieval capabilities.
What this means in practice
The optimization was driven by the need to re-embed a corpus of 30 billion high-priority documents, which temporarily doubled the embedding load. This prompted a re-evaluation of hardware choices and led to a series of optimizations that significantly improved throughput. The improvements were achieved through weight layout optimization, autocast, and custom kernel development, resulting in a 47% improvement in performance and cost per 1 million documents embedded. These optimizations are now part of an agentic development workflow that allows for continuous improvement and validation of the embedding process.
Limitations and availability
The optimizations are currently available on AWS Inferentia2 and Trainium2 hardware, with the initial implementation running on a single NeuronCore. The performance gains are specific to these hardware instances and may vary based on the region and availability of spot and on-demand instances. Exa has optimized the setup for the US West (Oregon) Region (us-west-2), but further tuning across other regions is needed to ensure consistent performance and availability.
