Hugging Face released ShadowPEFT on September 15, 2026, saying it provides a novel approach to parameter-efficient fine-tuning by introducing a stateful shadow model. It is the company's first major update to its PEFT library since the release of LoRA in 2024.
Hugging Face reported that ShadowPEFT outperformed LoRA and DoRA in experiments, with performance improving as the parameter budget increased, while LoRA remained flat and DoRA degraded. That compares with previous results where LoRA's performance was limited by its rank and number of adapted matrices.
ShadowPEFT is built on the PEFT library and targets efficient fine-tuning of large language models. Availability begins with the release of the library, initially for developers and researchers working on transformer-based models.
"ShadowPEFT introduces a compact and stateful shadow model with a persistent hidden state: s(0)→s(1)→s(2)→⋯→s(L)", said Zongxi Li, a researcher at Hugging Face. The shadow state carries task-specific information at each depth, enabling cross-layer coordination and a bidirectional information flow.
The announcement follows the release of LoRA and DoRA, which have been widely adopted for parameter-efficient fine-tuning. Hugging Face said ShadowPEFT expands PEFT beyond lightweight parameter injection toward reusable, cross-scale adaptation dynamics.
Hugging Face did not say how the shadow model will scale to larger models, and it raised the open question of how much capacity the task-specific computation should have. The company said it will continue to explore the implications of stateful adaptation in future updates.
Source: huggingface
