LoRAFusion: Energy-Efficient Multi-Task Adaptation Framework for ReRAM Accelerators

Resistive RAM (ReRAM) crossbars effectively accelerate machine learning inference, but high reprogramming energy limits their on-device adaptability. While Parameter-Efficient Fine-Tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) reduce training overhead, modifying a ReRAM crossbar for LoRA remains costly, and this reprogramming energy compounds significantly when supporting multi-task device adaptation.

 

Researchers at Arizona State University have developed a parameter-efficient multi-task fine-tuning method, LoRAFusion, for large language models deployed on ReRAM-based crossbar hardware accelerators. LoRAFusion enhances on-device adaptation for ReRAM-based accelerators by leveraging a fusion of pre-trained low-rank adaptation (LoRA) modules to drastically reduce reprogramming energy and trainable parameters. It introduces layer-wise fusion coefficients and a learnable magnitude vector to improve accuracy and learning capacity, while mapping frozen parameters to ReRAM and trainable ones to SRAM for optimized hardware efficiency. Demonstrated on the BigBench Hard dataset with the Google FLAN-T5 model, LoRAFusion achieves comparable performance to full fine-tuning at a fraction of energy and parameter cost.

 

LoRAFusion is a novel framework that enables efficient multi-task adaptation of large language models on ReRAM crossbar accelerators by fusing pre-trained LoRA modules with minimal energy and parameter overhead.

 

For more information about this opportunity, please see