Tutorial
Exa cuts embedding cost per document by 47% on AWS Inferentia2
Exa described how it optimized embedding inference for its web index on AWS Neuron hardware, reaching a 47% improvement in performance and cost per million documents embedded versus its initial Inferentia2 baseline. The gains came from weight layout changes, autocast, NKI kernels including a custom fused SwiGLU MLP kernel, and model-specific attention rewrites that cut padding and HBM round trips. Exa reports roughly 1.5x throughput improvement and says the same process carries over to Trainium2. Agents built the optimizations in a Flyte-based harness gated on 0.999 cosine similarity correctness checks.
ExaAWS Neuronvector search infrastructure