AI and the World's Hard Problems · Theoretical computer science · arXiv · 2026-10-21
OPLoRA: Orthogonal Projection LoRA Prevents Catastrophic Forgetting during Parameter-Efficient Fine-Tuning
Low-Rank Adaptation (LoRA) enables efficient fine-tuning of large language models but suffers from catastrophic forgetting when learned updates interfere with the dominant singular directions encoding pre-trained knowledge. The proposed OPLoRA decomposes frozen weights via SVD and constrains LoRA updates to lie entirely within the orthogonal complement of the top-k singular subspace using double-sided orthogonal projections, proving that this construction exactly preserves the top-k singular triples. It also introduces ρ_k, a metric for quantifying subspace interference, and experiments on commonsense reasoning, mathematics, and code generation show significantly reduced forgetting while maintaining competitive task-specific performance on LLaMA-2 7B and Qwen2.5 7B.