AMD has introduced Low Kruskal-Rank Adaptation (LoKRA), a new parameter-efficient fine-tuning (PEFT) method that enhances the performance of large language models (LLMs) during adaptation. The technique replaces the standard matrix rank used in Low-Rank Adaptation (LoRA) with Kruskal rank, which better captures redundancy and duplicated directions in parameter updates. According to AMD, LoKRA improves average accuracy by 2.4%-5.0% across various base models, outperforming LoRA and other baselines.
The method also introduces LoKRA+, an enhanced variant that provides a tighter theoretical lower bound on Kruskal rank and stronger empirical results. These improvements are achieved by optimizing the Krusk,al rank of the update matrix and using the Khatri-Rao product to replace matrix multiplication. The results were tested on multiple LLMs, including LLaMA-7B/13B, LLaMA2-7B, LLaMA3-8B, and Qwen3-8B, with performance measured on commonsense reasoning datasets.
The experiments were conducted on a single AMD Instinct MI300 Accelerator with ROCm. "We argue that the Kruskal rank offers a more informative criterion for characterizing update diversity," said AMD. "This leads to better generalization and stability in model adaptation." The paper is accepted by ICML 2026, and the code is publicly available on GitHub. AMD also invites users to explore the AMD Developer Cloud, featuring accelerators designed for AI workflows.
The company plans to continue exploring the combination of Kruskal rank and LoRA in future research. "We are committed to advancing LLM pruning research and deployment," AMD added. "You can dive deeper into the methodology and extensive benchmarks in our paper, and access our implementation on GitHub." The paper is available at the provided link.
Source: amd
