Intel Arc Pro B70 users can now achieve double the KV-cache capacity through FP8 quantization, significantly improving inference capabilities for long-context workloads. The technology enables a deterministic 2.0× increase in token capacity across all tested models, from 1B to 72B parameters, while maintaining accuracy for most models. This enhancement directly translates into higher concurrency, with raw 4K session capacity increasing by about 2.0× at the same max_model_len. For example, Qwen2.5-14B-Instruct (TP=2) scales from 48.52 to 97.05 sessions, and Mistral-Small-24B-Instruct-2501 (TP=2) from 43.27 to 86.53 sessions, making FP8 KV a strong enabler for capacity-constrained deployments. Throughput improvements are most notable in long-context workloads (16K–32K), where serving becomes KV-bandwidth bound. The peak gain reaches +42.3% for Qwen2.5-14B-Instruct (TP=1, 16K), with strong gains for Qwen3-8B and Llama-3.1-8B, and material improvements for 70B-class TP=4 models at 32K.

FP8 KV-cache quantization in vLLM directly addresses the memory bottleneck by storing KV tensors in FP8 rather than BF16 using --kv-cache-dtype fp8. Because FP8 uses half the storage per KV element, it doubles effective KV-token capacity within the same physical KV-cache memory allocation. This requires no model retraining, no architecture change, and no additional GPU memory. A representative Intel XPU launch configuration is: VLLM_TARGET_DEVICE=xpu

VLLM_MLA_DISABLE=1

VLLM_USE_V1=1

VLLM_ENGINE_READY_TIMEOUT_S=600

HF_HOME=/llm/models

vllm serve Qwen/Qwen2.5-14B-Instruct

--tensor-parallel-size 2

--max-model-len 4096

--gpu-memory-utilization 0.9

--quantization fp8

--kv-cache-dtype fp8

--enforce-eager

--block-size 64

--port 8000

Source: huggingface